DEV Community: Rahad Bhuiya

Building aegis-filter: Why Standard Kalman Filters Fail in Adversarial Environments (and How We Replaced Them in Pure Rust)

Rahad Bhuiya — Fri, 09 Oct 2026 05:43:28 +0000

When building telemetry filters for autonomous drones or robotics, the textbook recommendation is always the same: “Just slap a Kalman Filter on it.”

In theory, the linear Kalman filter is mathematically optimal under Gaussian white noise. In the real world, however, physical sensors do not live in textbook conditions:

Adversarial GPS/Baro Spoofing: A rogue transmitter or RF jammer injects sudden, massive delta jumps (e.g., +150 meters in a single millisecond).
Numerical Covariance Degeneracy: Floating-point round-off errors over millions of iterations cause the error covariance matrix $P$ to lose its positive-definiteness ($P \not\succ 0$), producing negative variances and exploding state estimates.
Severe Non-Linearities: Radar range calculations ($r = \sqrt{x^2 + y^2}$) and orientation quaternions break linear assumptions; Extended Kalman Filters (EKF) require calculating brittle analytical Jacobian matrices that drift rapidly.

To solve these mission-critical vulnerabilities, we built and open-sourced aegis-filter under Exploidus—a cyber-resilient, #![no_std], hard real-time estimation engine written from scratch in pure Rust.

Here is the engineering breakdown of how it works and what the simulation numbers prove.

1. The Core Architecture

aegis-filter is engineered with three core defenses:

A. Joseph-Stabilized Covariance Update

Standard Kalman filters update covariance using $P_k = (I - K_k H_k) P_{k|k-1}$. In finite-precision IEEE 754 floating-point arithmetic, subtraction can yield asymmetric, non-positive-definite matrices.

aegis-filter enforces the Joseph Stabilized Form:
$$P_k = (I - K_k H_k) P_{k|k-1} (I - K_k H_k)^T + K_k R_k K_k^T$$

Because both terms are quadratic forms ($A P A^T + B R B^T$), positive-definiteness is mathematically guaranteed across infinite operational lifetimes.

B. Chi-Square / 3-Sigma Outlier Rejection Gating

Before any raw measurement is allowed to alter the state vector, the innovation $\tilde{y}k = z_k - H_k \hat{x}{k|k-1}$ is evaluated against the innovation covariance $S_k = H_k P_{k|k-1} H_k^T + R_k$:
$$\text{Reject if } |\tilde{y}_k| > \gamma \sqrt{S_k}$$

If an adversary transmits a spoofed altitude or an IMU glitches, the reading is gated at line-rate. The filter falls back safely to its physical state propagation without corrupting the state.

C. Unscented Kalman Filtering (UKF) & $H_\infty$ Robust Minimax

UKF (Merwe Scaled Sigma Points): Replaces brittle Jacobian derivations by propagating $2n + 1$ deterministic sigma points through non-linear observation closures.
$H_\infty$ Minimax: Optimizes state estimation against the worst-case disturbance rather than assuming well-behaved Gaussian noise.

2. Flight Telemetry & Spoofing Resilience Simulation

To prove real-world resilience, we simulated a quadcopter ascending at $1.5\text{ m/s}$ subjected to atmospheric turbulence, normal sensor noise, and deliberate adversarial spoofing:

Step 12: Instantaneous $+150\text{m}$ GPS spoofing attack.
Step 22: Severe $-80\text{m}$ sensor hardware dropout glitch.

Here is the telemetry log produced by running cargo run --example flight_telemetry:

===============================================================================
  AEGIS-FILTER :: FLIGHT TELEMETRY & ADVERSARIAL RESILIENCE SIMULATION
  Developed under Exploidus (https://github.com/Exploidus)
===============================================================================

Step   | True Alt (m) | Sensor Raw (m) | Aegis Est (m)  | Abs Error    | Status    
-------+--------------+----------------+----------------+--------------+-----------
1      | 1.50         | 1.79           | 1.75           | 0.254        | VALID     
2      | 3.00         | 3.22           | 3.24           | 0.238        | VALID     
3      | 4.50         | 3.46           | 4.31           | 0.187        | VALID     
4      | 6.00         | 6.74           | 6.06           | 0.057        | VALID     
5      | 7.50         | 7.67           | 7.58           | 0.081        | VALID     
6      | 9.00         | 8.90           | 9.04           | 0.045        | VALID     
7      | 10.50        | 9.68           | 10.39          | 0.113        | VALID     
8      | 12.00        | 12.98          | 12.08          | 0.075        | VALID     
9      | 13.50        | 13.41          | 13.55          | 0.047        | VALID     
10     | 15.00        | 14.69          | 14.99          | 0.009        | VALID     
11     | 16.50        | 16.04          | 16.42          | 0.080        | VALID     
12     | 18.00        | 168.00         | 17.92          | 0.080        | SPOOF REJECT
13     | 19.50        | 19.08          | 19.36          | 0.138        | VALID     
14     | 21.00        | 20.65          | 20.83          | 0.172        | VALID     
15     | 22.50        | 22.43          | 22.34          | 0.156        | VALID     
16     | 24.00        | 24.93          | 24.01          | 0.012        | VALID     
17     | 25.50        | 24.79          | 25.40          | 0.099        | VALID     
18     | 27.00        | 26.78          | 26.88          | 0.116        | VALID     
19     | 28.50        | 28.76          | 28.44          | 0.060        | VALID     
20     | 30.00        | 30.64          | 30.04          | 0.044        | VALID     
21     | 31.50        | 30.61          | 31.41          | 0.095        | VALID     
22     | 33.00        | -47.00         | 32.91          | 0.095        | SPOOF REJECT
23     | 34.50        | 34.96          | 34.50          | 0.003        | VALID     
24     | 36.00        | 36.25          | 36.04          | 0.036        | VALID     
25     | 37.50        | 36.60          | 37.39          | 0.110        | VALID     
26     | 39.00        | 39.34          | 38.96          | 0.041        | VALID     
27     | 40.50        | 40.99          | 40.54          | 0.039        | VALID     
28     | 42.00        | 41.84          | 42.01          | 0.009        | VALID     
29     | 43.50        | 42.78          | 43.40          | 0.100        | VALID     
30     | 45.00        | 45.60          | 45.00          | 0.003        | VALID     

-------------------------------------------------------------------------------
  SIMULATION SUMMARY:
  - Total Steps Processed      : 30
  - Valid Readings Fused       : 28
  - Adversarial Spikes Blocked : 2 (100% Gated)
  - Final Variance Confidence  : 0.1185
===============================================================================

Key Takeaways from the Data:

Millimeter Tracking Precision: Under normal noisy conditions (Step 10, Step 30), estimation error dropped to 0.003–0.009 meters (3–9 mm).
Flawless Spoof Rejection: When the raw altitude leaped to $168\text{m}$ at Step 12, the filter isolated the anomaly, kept the aircraft altitude at $17.92\text{m}$ (error: $0.080\text{m}$), and prevented a fatal motor cut.

3. Quick Start (Rust)

Add aegis-filter to your Cargo.toml:

[dependencies]
aegis-filter = "0.1.0"

For bare-metal Cortex-M/RISC-V embedded targets:

[dependencies]
aegis-filter = { version = "0.1.0", default-features = false }

1D Outlier-Gated Fusion Example:

use aegis_filter::{ScalarKalmanFilter, FilterError};

fn main() -> Result<(), FilterError> {
    // Initial state: 0.0m, Initial variance: 5.0, Process noise: 0.02, Measurement noise: 0.8
    let mut filter = ScalarKalmanFilter::new(0.0, 5.0, 0.02, 0.8)?;

    let sensor_stream = [1.5, 3.1, 4.4, 168.0 /* Spoofed spike */, 6.0];

    for &raw_reading in &sensor_stream {
        filter.predict(1.5)?; // Predict forward by 1.5m climb

        match filter.update(raw_reading, Some(3.0)) { // 3-sigma gate
            Ok(estimated_alt) => println!("Estimated Altitude: {:.2}m", estimated_alt),
            Err(FilterError::MeasurementGated) => {
                println!("Spoofed sensor reading ({:.1}m) rejected! Holding state.", raw_reading);
            }
            Err(e) => return Err(e),
        }
    }

    Ok(())
}

4. Open Source Links

The engine is open-source under dual MIT/Apache-2.0 licenses:

Crates.io: crates.io/crates/aegis-filter
GitHub Repository: Exploidus/aegis-filter
Documentation: docs.rs/aegis-filter

Feedback, issues, and PRs from control engineers and systems developers are warmly welcome!

Building a Line-Rate Linux Kernel Firewall in Pure Rust with Aya and eBPF/XDP

Rahad Bhuiya — Sat, 03 Oct 2026 13:27:10 +0000

Volumetric Distributed Denial of Service (DDoS) and transport-layer flooding attacks pose an existential challenge to modern cloud infrastructure, edge gateways, and container workloads.

Conventional Linux packet filtering utilities such as Netfilter (iptables and nftables) evaluate incoming frames only after the kernel has allocated a socket buffer (sk_buff) descriptor and scheduled softirq processing through the core TCP/IP stack. Under multi-gigabit line-rate flood conditions (millions of packets per second), the operating system spends the vast majority of its CPU cycles simply allocating and freeing sk_buff memory metadata, triggering kernel starvation and packet drop bottlenecks even before firewall rules are evaluated.

To address this fundamental OS-level bottleneck, I built xdp-guard: a production-grade, memory-safe in-kernel defense engine and per-IP rate limiter implemented strictly in Pure Rust using the Aya eBPF framework.

Why eBPF / XDP at the Driver Layer?

The eXpress Data Path (XDP) provides a programmable, bare-metal packet processing execution hook directly inside the Network Interface Card (NIC) driver subsystem before any memory descriptor allocation occurs.

Conventional Path:
[NIC RX] -> [IRQ / SoftIRQ] -> [Allocate sk_buff] -> [Netfilter/iptables] -> [Socket / App]
                                      ^
                           Severe CPU Allocation Overhead

xdp-guard Path:
[NIC RX] -> [eBPF / XDP Driver Hook] ---> (Match Blocklist) ---> [ XDP_DROP ] (Near-Zero CPU)
                             |
                      (Match Allowlist)
                             |
                             v
                   [ Pass to OS Stack ]

When hostile or unapproved packets are identified, returning XDP_DROP causes the network driver to immediately recycle the packet ring buffer without allocating a single byte of kernel socket memory.

Architectural Highlights

1. Zero C Dependencies (Pure Rust)

Traditional eBPF tools rely heavily on C toolchains (clang, llvm, libbpf). xdp-guard is built entirely in Rust from the user-space management CLI down to the no_std kernel eBPF bytecode using Aya.

2. Strict 64-Bit Memory Boundary Alignments

To ensure safe zero-copy communication across the kernel-userspace boundary, shared structures use explicit padding fields to guarantee 8-byte alignment:

#[repr(C)]
#[derive(Clone, Copy, Debug)]
pub struct RuleValue {
    pub action: u32,       // 0 = Pass, 1 = Drop
    pub _pad: u32,         // Explicit 4-byte padding for 8-byte boundary
    pub created_at: u64,   // Monotonic timestamp
    pub ttl_secs: u64,     // Time-to-live expiration
}

The presence of _pad: u32 is critical: it prevents architecture-specific compiler padding discrepancies and satisfies the Linux Kernel BPF Verifier.

3. Dynamic BPF Virtual Filesystem (bpffs) Synchronization

Firewall rules (blocklist and allowlist) are dynamically synchronized using pinned BPF hash maps located in /sys/fs/bpf/xdp_guard. Rules can be inserted or deleted instantaneously from the CLI without tearing down network interfaces or interrupting traffic flows.

4. In-Kernel Token-Bucket Rate Limiter

In addition to static blocklists, xdp-guard embeds a stateful token-bucket rate limiter directly in kernel space. Monotonic arrival intervals ($\Delta t = t_{now} - t_{last}$) are computed using nanosecond bpf_ktime_get_ns() clocks, discarding over-limit burst floods at line-rate.

Empirical Verification: Real Linux Kernel Telemetry

To validate xdp-guard under realistic operating conditions, we set up an automated Linux network namespace testbed over virtual ethernet (veth) pairs (xdp_client at 10.10.0.2 and xdp_server at 10.10.0.1).

Telemetry streamed directly from the kernel STATS_MAP array demonstrated instantaneous 100.00% packet mitigation upon inserting an attacker IP into the kernel blocklist, followed by zero-loss recovery:

TIMESTAMP            PASSED (PKTS)   DROPPED (PKTS)  DROP RATIO     
-----------------------------------------------------------------
09:56:56 | Pass: 1        (30.2 KB) | Drop: 0        (0 B)   | Ratio: 0.00%
[!] Inserting IP 10.10.0.2 into BLOCKLIST_MAP... [SUCCESS]
09:56:57 | Pass: 0        (30.2 KB) | Drop: 1        (98 B)  | Ratio: 100.00%
09:56:58 | Pass: 0        (30.2 KB) | Drop: 1        (196 B) | Ratio: 100.00%
09:56:59 | Pass: 0        (30.2 KB) | Drop: 1        (294 B) | Ratio: 100.00%
...
09:57:55 | Pass: 0        (30.2 KB) | Drop: 1        (5.7 KB)| Ratio: 100.00%
[!] Removing IP 10.10.0.2 from BLOCKLIST_MAP... [SUCCESS]
09:57:56 | Pass: 1        (30.3 KB) | Drop: 0        (5.7 KB)| Ratio: 0.00%
09:57:57 | Pass: 1        (30.4 KB) | Drop: 0        (5.7 KB)| Ratio: 0.00%

Every single drop occurred directly within the NIC driver hook, requiring zero kernel TCP/IP stack overhead.

Source Code & Research Paper

The complete project, documentation, and formal research paper are fully open-source:

GitHub Repository: https://github.com/rahadbhuiya/xdp-guard
Citable Zenodo DOI: 10.5281/zenodo.23120995
Automated CI: Continuous testing across Rust Stable and Nightly LLVM targets via GitHub Actions.

If you are working on Linux kernel systems, eBPF, or Rust networking, I would love to hear your thoughts, feedback, or suggestions!

Writing a Programming Language & Native x86-64 Compiler from Scratch in C: The Story of Yolish

Rahad Bhuiya — Sun, 27 Sep 2026 03:56:05 +0000

Every programmer has, at some point, looked at their favorite programming language and thought:

“Why was it designed this way? How hard can it actually be to build one myself?”

Most developers who explore language design stop at simple arithmetic evaluators or toy interpreters. But when I was building Exploidus OS—a custom x86-64 reactive capability operating system—I ran into a fundamental architectural barrier:

Standard systems languages (like C) have zero concept of operating-system capability tokens. In C, if you have a pointer, you can dereference it. If you have an address space, any line of code can call any syscall unless restricted by complex external sandboxes.

I wanted a modern, expressive language that felt as ergonomic as Python or Rust, but was deeply capability-aware at the syntax level—able to compile directly to bare-metal x86-64 binaries without pulling in gigabytes of LLVM dependencies.

That ambition gave birth to Yolish—a fast, expressive, capability-aware programming language with a tree-walking interpreter, a bytecode virtual machine, and a direct x86-64 native compiler.

Here is how I designed Yolish, wrote its compiler from scratch in pure C, and what I learned building a complete language ecosystem.

1. Language Design: Ergonomics Meets Systems Control

When designing Yolish's syntax (.y), I wanted code to read cleanly without syntactic noise, while enforcing defensive systems programming invariants.

Immutability by Default

Variables in Yolish default to immutable constants using let, while mutable state must be explicitly declared using var:

let name  = "Yolish"   -- immutable constant
var count = 0          -- mutable state
count = count + 1

Pattern Matching with Ranges & Guards

Instead of archaic C-style switch statements with fallthrough bugs, Yolish features expressive pattern matching:

let grade = match score {
    90..100      => "A"
    80..90       => "B"
    n if n >= 60 => "C"
    _            => "F"
}

Structs and `impl` Blocks

Yolish rejects complex class hierarchies in favor of data-first structs and explicit method implementations:

struct Circle { radius }

impl Circle {
    fn area(self) {
        return y.math.pi * self.radius * self.radius
    }
}

let c = Circle { radius: 5 }
y.println(c.area())

2. Bridging Code to the Kernel: First-Class Capabilities

Because Yolish is the official userspace language for Exploidus OS, security cannot be an afterthought.

In Yolish, dangerous operations (network access, raw disk writes, peripheral I/O) are gated by declarative capability annotations:

@cap(net.read, fs.write)
fn sync_data(url, filepath) {
    let payload = y.net.get(url)
    y.fs.write(filepath, payload)
}

When compiled for Exploidus OS, the Yolish compiler inspects these annotations and validates them against the process's cryptographic capability table. If an unprivileged function attempts to make an unauthorized syscall, the operation is blocked before execution even reaches kernel space.

3. The Tri-Engine Architecture: Interpreter, Bytecode VM, and Native Compiler

Building a language engine involves tough tradeoffs. Tree-walking interpreters are easy to write but slow. Bytecode VMs are fast and portable. Native compilers deliver maximum speed but are deeply architecture-specific.

Instead of choosing one, Yolish implements all three:

Source Code (.y)
       |
    [Lexer] -> Tokens
       |
   [Parser] -> Abstract Syntax Tree (AST)
       |
       +---> Tree-Walking Interpreter  (`ys file.y` - Instant startup)
       |
       +---> Bytecode Compiler & VM    (`ys vm file.y` - High speed execution)
       |
       +---> Direct x86-64 Machine Code (`ys -c file.y` - Standalone binaries)

The Interactive Interpreter (ys file.y): Evaluates the AST directly. Perfect for quick scripts, one-liners, and the interactive REPL.
The Bytecode VM (ys vm file.y): Compiles the AST into an optimized instruction stream executed by a custom register/stack-based virtual machine.
The Native x86-64 Compiler (ys -c file.y): Generates real machine code instructions (MOV, ADD, CALL, RET, SSE2 float operations) directly into executable binary containers.

4. No LLVM, No Bloat: Direct Multi-Platform Binary Emission

Most modern compiler projects delegate code generation to LLVM. While LLVM is powerful, it is also a massive, multi-gigabyte dependency with slow compilation times.

I wanted the Yolish toolchain (ys) to be a single, self-contained C binary with zero external dependencies. You can clone the repo and compile the entire language toolchain with a single make command using standard gcc or clang in seconds.

To achieve this without LLVM, Yolish writes binary headers and machine code directly:

Linux ELF64: Directly formats ELF headers, program headers (PT_LOAD), .text sections, and Linux syscall dispatchers.
Windows PE32+: Generates valid Portable Executable (PE32+) headers, image import directories, and embeds Windows application icons.
macOS Mach-O: Generates 64-bit Mach-O load commands (LC_SEGMENT_64).

You can cross-compile for any major operating system with a single CLI flag:

ys -c app.y --target linux    # Emits static Linux ELF64 binary
ys -c app.y --target windows  # Emits Windows PE32+ .exe
ys -c app.y --target macos    # Emits macOS Mach-O binary

5. Runtime Architecture: Mark-and-Sweep Garbage Collection

A systems language with dynamic strings and closures needs safe, automatic memory reclamation.

Yolish implements a custom Mark-and-Sweep Garbage Collector (GC) written from scratch in C:

Root Scanning: Traverses the active call frame stack, global environment variables, and the VM operand stack to identify all actively reachable pointers.
Mark Phase: Recursively traverses heap-allocated objects (strings, dynamic arrays, closures, struct instances), setting the GC mark bit.
Sweep Phase: Scans the global heap allocation pool. Any block whose mark bit is zero was unreachable; it is freed back to the system memory pool, while surviving objects have their mark bits cleared for the next cycle.

Developers have full programmatic introspection into memory usage via gc.stats() and can trigger explicit collection via gc.collect().

6. A Tooling-First Philosophy

A great programming language is more than just a syntax parser; it is an entire developer experience.

Too many new languages fail because they require third-party formatters, linter plugins, and package wrappers before you can write a serious project.

From day one, Yolish was designed with a tooling-first philosophy:

Built-in Test Runner (ys test file.y): Write native test blocks alongside code and execute them without third-party frameworks.
Static Checker (ys check file.y): Performs static semantic analysis, flags undefined variables, and detects type mismatches before runtime.
Code Formatter (ys fmt file.y): Enforces an opinionated, canonical code style across codebases.
Interactive REPL (ys): Instant feedback loop with color-highlighted evaluation.

7. What Writing a Compiler in C Teaches You

Writing a programming language and compiler from scratch in pure C is one of the most humbling and rewarding journeys in software engineering:

Memory Discipline: Managing AST nodes, symbol tables, and lexical scopes in C forces you to understand every single byte you allocate. Memory leaks in a compiler slow down compilation; memory corruption in code generation crashes the output binary.
The Magic of Abstraction: When you write the code that translates a simple if (x > 10) into a CMP and JLE assembly jump, you demystify how computers actually think. You realize that high-level abstractions are not magic; they are just structured transformations of intent.
Simplicity Over Complexity: Features are easy to add to a grammar; keeping the language coherent, orthogonal, and bug-free across interpretation and native compilation is where the real engineering happens.

Conclusion & Open Source

Yolish started as a specialized language project for Exploidus OS, but has evolved into a versatile, cross-platform programming environment with its own native compiler, VM, and standard library.

The entire project is 100% open-source under the MIT license:

GitHub Repository: https://github.com/rahadbhuiya/Yolish
Supported Platforms: Linux, Windows, macOS, and Exploidus OS.

Download the binary, try out the REPL, write a script, and inspect the C compiler source!

Have you ever thought about designing a programming language? What is the one feature you wish your daily language had? Let's discuss in the comments below!

Why Fail2ban Isn't Enough: Building CNSL, a Self-Hosted SIEM for Linux & Kubernetes

Rahad Bhuiya — Sat, 26 Sep 2026 04:35:44 +0000

If you manage Linux servers, your security stack probably looks something like this:

Fail2ban scanning /var/log/auth.log for failed SSH passwords.
A reverse proxy or web server (NGINX / Caddy) writing HTTP access logs.
A database (PostgreSQL / MySQL) logging authentication errors.
A firewall (iptables / UFW) enforcing static port rules.

On paper, this sounds like a layered defense. In practice, it has a glaring, dangerous architectural flaw: these tools operate in complete isolation.

Each tool watches a single log file and counts failures independently. And modern attackers know this.

An attacker does not hammer your SSH port with 100 rapid-fire passwords anymore—that would trip Fail2ban within seconds. Instead, they scan your web endpoints for exposed .env files, probe your database port for default passwords, and attempt just two SSH logins with leaked administrator credentials.

To your web server, it was just a minor 404 error. To your database, it was a routine connection failure. To Fail2ban, two failed SSH logins are well below the threshold of 5.

Every single tool saw a harmless blip. In reality, your server was undergoing a coordinated, multi-stage breach.

To eliminate this architectural blind spot, I built CNSL (Correlated Network Security Layer)—a self-hosted, lightweight SIEM for Linux and Kubernetes that correlates attacks spanning web, SSH, database, and cloud logs simultaneously, stopping breaches before they complete.

Here is the story of why I built it, how its cross-vector correlation engine works, and what it takes to design a modern defensive security layer from scratch.

1. The Core Philosophy: Cross-Vector Correlation

The foundational principle behind CNSL is simple: No security event should ever be evaluated in a vacuum.

When an IP address or actor touches your infrastructure, CNSL tracks their activity across every listening service simultaneously. Instead of waiting for a single service threshold to fire, CNSL computes a dynamic threat score across all vectors:

Web scan (directory traversal)  from 45.33.32.1  --+
SSH brute attempt               from 45.33.32.1  --+---> HIGH Alert & Auto-Block
DB authentication failure       from 45.33.32.1  --+

When CNSL observes an actor combining reconnaissance with authentication probing across disparate services, the threat trajectory escalates immediately.

The attacker is automatically banned at the firewall/iptables level or quarantined across the entire cluster—before they ever guess a password or exploit an unpatched service.

2. Kill-Chain Progression & Predictive Blocking

Traditional intrusion prevention systems (IPS) are reactive: they wait until a threshold is crossed before taking action.

CNSL integrates Cyber Kill-Chain Tracking modeled after the MITRE ATT&CK framework:

Reconnaissance: Port probing, vulnerability scanning, spidering.
Weaponization & Delivery: Testing malicious payloads, path traversal.
Exploitation & Initial Access: Credential stuffing, brute-forcing, privilege escalation attempts.
Lateral Movement: Probing internal service meshes, database exfiltration.

Predictive Trajectory Blocking

Instead of waiting for an attacker to complete the exploitation phase, CNSL features opt-in Predictive Blocking.

By analyzing the velocity and breadth of an attack vector, the correlation engine calculates the statistical probability of an impending breach. If an IP rapidly transitions from passive web reconnaissance to active database auth probing, CNSL severs the connection proactively—reacting to the attack's trajectory before any single rule's threshold trips.

3. Beyond IP Addresses: Attacker Fingerprinting & Campaign Graphs

Sophisticated threat actors do not use a single IP address. They utilize residential proxy networks, VPNs, and Tor exit nodes, rotating their source IP address between requests to evade IP-based rate limiters.

If your security system only tracks IP addresses, you are playing an endless game of whack-a-mole.

To counter IP rotation, CNSL incorporates Attacker Fingerprinting and Graph Correlation:

Behavioral Fingerprinting: Tracks TLS cipher suites, TCP window signatures, request ordering, and user-agent entropy.
Graph-Based Campaign Correlation: Connects seemingly disparate IP addresses attacking your servers into a unified attack campaign graph.
When Actor X rotates from IP_A to IP_B, CNSL recognizes the behavioral fingerprint, associates the new IP with the existing threat entity, and immediately inherits the accumulated threat score.

4. Threat Intelligence Federation: Built-in STIX 2.1 and TAXII 2.1

Security should not be isolated to a single machine. When one node in your cluster discovers an aggressive threat actor, every other node should immediately know about it.

Rather than relying on proprietary, vendor-locked sync protocols, CNSL natively implements global open threat intelligence standards:

STIX 2.1 Export: Converts detected threat actors, campaigns, and indicators of compromise (IOCs) into standardized STIX JSON objects.
Built-in TAXII 2.1 Server: Runs a native TAXII endpoint directly inside CNSL. Other servers in your infrastructure (or peer organizations) can subscribe to your TAXII collections to ingest real-time threat intelligence.
Multi-Node Federation: Allows multi-region VPS deployments or edge servers to pool threat intelligence without routing all traffic through a centralized, vulnerable bottleneck.

5. From Single VPS to Kubernetes DaemonSet

CNSL was designed to be lightweight enough to run on a $5/month VPS, yet cloud-native enough to protect modern containerized environments.

Running on a Single Linux Server

CNSL can be installed directly with Python and pip:

pip install cnsl[full]
sudo python -m cnsl --dashboard --no-tcpdump

By default, CNSL boots into safe dry-run mode—it correlates events, populates the live dashboard, and logs detected campaigns without modifying firewall rules until you explicitly pass --execute.

The built-in web dashboard (running on port 8765) gives operators real-time visibility into active threats, geographic origins, attack kill-chain phases, and blocked IPs.

Running on Kubernetes

In modern Kubernetes clusters, attackers frequently compromise one container and attempt lateral movement toward neighbor pods or database secrets.

CNSL provides an official Helm chart designed to run as a DaemonSet:

helm install cnsl ./helm/cnsl --namespace cnsl --create-namespace

One CNSL agent runs per Kubernetes node, monitoring container socket events, host logs, and ingress traffic, feeding telemetry into a unified multi-node Hub view.

6. What I Learned Building a SIEM from Scratch

Building a real-time correlation engine in Python and systems-level Linux tooling provided some invaluable engineering insights:

Log parsing is a performance minefield: When a high-traffic server generates 10,000 log lines per second, regex matching can easily peg your CPU at 100%. Implementing ring buffers, compiled zero-allocation parsers, and asynchronous pipeline queues is mandatory to keep memory and CPU footprints negligible.
False positives destroy credibility: If a security tool accidentally bans an administrator or a legitimate customer, operators will turn it off immediately. Multi-vector correlation actually reduces false positives because a user making a typo on a web form is never banned—only entities combining suspicious behavior across multiple services are flagged.
Open standards win every time: Implementing STIX/TAXII and integrations with Wazuh/OSSEC meant CNSL could plug seamlessly into existing SOC workflows without requiring operators to reinvent their operational playbooks.

Conclusion: Take Back Control of Your Server Security

Enterprise SIEMs (like Splunk or Datadog) cost tens of thousands of dollars per year and consume massive amounts of memory. On the other end of the spectrum, single-log watchers like Fail2ban are too blind to stop multi-vector attacks.

CNSL was built to bridge this gap: an open-source, self-hosted, correlated security engine that gives developers, self-hosters, and DevOps engineers enterprise-grade situational awareness with zero vendor lock-in.

The entire project is open-source under the MIT license:

GitHub Repository: https://github.com/rahadbhuiya/cnsl
PyPI Package: https://pypi.org/project/cnsl

Clone it, test it in dry-run mode on your servers, and let me know your thoughts!

How do you currently monitor and correlate security events across your Linux servers? Have you ever caught an attacker probing across multiple ports? Let's discuss in the comments below!

Building an x86-64 Operating System from Scratch in C: The Exploidus Story

Rahad Bhuiya — Thu, 24 Sep 2026 13:36:21 +0000

Most software engineers spend their careers working high up the abstraction ladder: web frameworks, containers, virtual machines, and cloud APIs. We treat the operating system as an invisible, infallible foundation. We assume malloc() will magically allocate RAM, write() will reliably flush bytes to disk, and the CPU will seamlessly multiplex threads.

A few years ago, I decided to pull back the curtain and ask a fundamental question:

What actually happens if you strip away Linux, Windows, libc, and every single runtime library, and sit directly on the bare metal of an x86-64 processor with nothing but C and assembly?

That question led to Exploidus—a custom x86-64 reactive capability operating system kernel built completely from scratch.

Today, Exploidus boots via Multiboot2/GRUB, runs in full 64-bit long mode, implements 4-level paging with NX enforcement, features a custom journaling filesystem (ExFS), runs an in-kernel TCP/IP stack with real Gigabit Ethernet drivers, enforces BLAKE3 cryptographic capability tokens, supports 82 POSIX-style syscalls, and even powers a double-buffered userspace window compositor.

Here is the story of how I designed and built it, the architectural hurdles I faced, and what bare-metal systems programming teaches you about software engineering.

The Starting Line: From Real Mode to 64-Bit Long Mode When an x86-64 processor powers on, it does not wake up as a modern 64-bit machine. It awakens in 16-bit Real Mode, essentially pretending to be an Intel 8086 from 1978 with 1 MB of addressable memory.

To get Exploidus running, the bootloader transition had to be orchestrated meticulously:

Multiboot2 Compliance: The kernel binary begins with a Multiboot2 header so GRUB2 can discover and load our ELF64 image into memory.
Transitioning Modes: Moving through 32-bit Protected Mode, setting up the Global Descriptor Table (GDT), enabling Physical Address Extension (PAE), and finally setting the Long Mode Enable (LME) bit in the EFER model-specific register.
Entering Long Mode: Once paging is turned on and a 64-bit GDT code descriptor is loaded, a far jump lands execution into pure 64-bit Long Mode.
At this stage, you have no printf, no standard library, and no memory allocator. If a pointer dereferences wrong, the CPU triple-faults and the machine instantly reboots in a split second.

The first milestone of any OS developer is the VGA text-mode driver at physical address 0xB8000. Writing characters directly into video memory bytes and watching the word Exploidus Kernel Initializing... render on a blank screen is a sensation every programmer should experience at least once.

Memory Architecture: 4-Level Paging and Colored Zones Virtual memory is the bedrock of process isolation. Without it, any runaway pointer can overwrite the kernel or another process's state.

In Exploidus, memory management is divided into two core layers:

4-Level x86-64 Paging
Exploidus uses the standard x86-64 page-table hierarchy:

PML4 (Page Map Level 4)
PDPT (Page Directory Pointer Table)
PD (Page Directory)
PT (Page Table)
Every page entry manages a 4 KB physical frame. To enforce strict security:

NX (No-Execute) Bit Enforcement: All data, heap, and stack pages are flagged with the XD/NX bit (bit 63). Code cannot execute from the stack or heap, neutralizing classic buffer overflow shellcode attacks at the hardware level.
User/Supervisor Protection: Kernel space is mapped into the higher half and strictly protected by disabling the User bit (U/S = 0), preventing Ring 3 userspace applications from reading or writing kernel data structures.
Colored Physical Frame Allocation
Instead of a naive linear allocator or a slow buddy system, Exploidus introduces a Colored Zone Physical Memory Manager:

GREEN ZONE: Unrestricted, verified general-purpose memory for userspace processes.
YELLOW ZONE: DMA-capable and peripheral-bounded memory mapped for hardware device drivers (e.g. NIC ring buffers, USB host controllers).
RED ZONE: Cryptographically isolated, non-swappable physical frames reserved strictly for kernel capability tokens, master page tables, and audit logs.

Capability Security & Crash Isolation Most commodity operating systems rely on ambient authority (e.g. standard UID/GID or Unix permissions). If a process runs as root, it can execute anything.

Exploidus explores a Reactive Capability-Based Security Model:

Resources (file descriptors, sockets, physical devices, IPC channels) are gated behind unforgeable capability tokens.
Hardware-Seeded BLAKE3 Tokens: Tokens are generated by hashing resource pointers and generation counters using the BLAKE3 cryptographic algorithm, seeded with random entropy directly from the CPU's RDRAND instruction.
Crash Isolation: If a userspace process suffers a segmentation fault, divides by zero, or triggers an invalid opcode, the kernel does not panic. The CPU interrupt handler traps the exception, destroys the faulting process's capability table, reclaims its address space, and terminates only that process. The shell and the rest of the OS continue running uninterrupted.

Designing a Custom Filesystem: ExFS You cannot have a functional operating system without persistent storage. While many hobby kernels stop at read-only tar ramdisks or FAT12, Exploidus features its own custom filesystem: ExFS.

Key Architectural Elements of ExFS:
Generic Block Device Layer: An abstraction layer (blockdev.c) decouples storage hardware from the filesystem. Whether a volume is an IDE/ATA hard drive (ata0) or a USB thumb drive (usb0), ExFS interacts through unified sector read/write interfaces.
Metadata Journaling: ExFS uses a write-ahead metadata journal to ensure crash consistency. Directory modifications, inode allocations, and block allocations are recorded in a circular journal before being committed to disk structures. If a sudden power loss occurs, replay recovery restores volume consistency upon reboot.
Data Provenance: Inodes in ExFS carry immutable provenance records, tracking the creator capability and timestamp of creation to provide built-in forensic audit trails.

Drivers from the Metal: USB and Gigabit Ethernet Writing device drivers without third-party libraries or Linux helper APIs is where hardware reality hits you hardest. Data sheets for real silicon can be hundreds of pages long and filled with subtle timing quirks.

UHCI USB Subsystem
Exploidus includes a custom UHCI (Universal Host Controller Interface) USB driver:

Probes PCI configuration space to discover USB host controllers.
Configures physical memory transfer descriptors (TDs) and queue heads (QHs) in DMA-accessible memory.
Implements USB device enumeration, Control, Interrupt, and Bulk endpoints.
Supports USB Mass Storage (Bulk-Only Transport + SCSI command wrappers), allowing developers to plug in a USB flash drive, mount it into the VFS, and read files via the shell.
Gigabit Networking & In-Kernel TCP/IP Stack
Exploidus features an in-kernel network stack communicating through an Intel 82540EM (e1000) Gigabit Ethernet controller:

Layer 2: Ethernet frame parsing and ARP table resolution with cache expiration.
Layer 3: IPv4 parsing, checksum calculation, and IP fragment reassembly.
Layer 4: ICMP (ping echo reply/request), UDP, and full TCP state machine (three-way handshake SYN -> SYN-ACK -> ACK, sequence/acknowledgment tracking, window scaling, and packet retransmission).
Because the TCP/IP stack runs natively inside the kernel, userspace processes can issue POSIX-like socket(), connect(), send(), and recv() syscalls to communicate over the real local network.

From the exploish Shell to a GUI Compositor An OS needs an interface to be useful. Exploidus provides two layers:

The exploish Interactive Shell
exploish runs as a Ring 3 userspace process, executing commands via system calls:

ps: Inspects real kernel process tables, priorities, and execution states.
audit: Reads cryptographic audit log entries directly from the kernel.
mount / ls / cat: Traverses and manipulates the ExFS filesystem.
The alien Window Compositor
Moving beyond text mode, Exploidus includes a userspace graphical compositor called alien:

Utilizes VESA / Linear Framebuffer modes discovered via Multiboot.
Single-Syscall Window Blitting (SYS_FB_BLIT): Minimizes user-to-kernel context switching during rendering.
Double-Buffering & Dirty-Region Redraws: Windows maintain their own back-buffers in userspace; the compositor only copies modified bounding boxes to the screen buffer, providing flicker-free rendering with real-time frame pacing.

What Building an OS Taught Me Writing an operating system from scratch over months and years completely reshapes how you look at software engineering:

Hardware is not clean; it is quirky: Silicon behaves in strange ways. Concurrency is not just threads racing; it is hardware interrupts firing in the middle of a critical pointer assignment. You develop an immense respect for atomic operations, IRQ-safe spinlocks, and memory barriers.
Abstractions have a real cost: When you have to write your own kmalloc(), page fault handler, and scheduling timer, you realize that memory allocations and context switches are not free. High-level frameworks often hide enormous amounts of churn.
Debugging without tools builds discipline: When you don't have GDB or stack traces, and an invalid memory write freezes the CPU instantly, you learn to reason through your code with mathematical precision. You read every line, analyze register states, and build defensive invariants.
Conclusion & Open Source
Exploidus started as an ambitious experiment in Bangladesh to explore what it takes to build an operating system from zero without relying on existing Linux codebases. Today, it stands as a fully documented, open-source x86-64 capability kernel with a rich ecosystem of drivers, filesystems, and networking.

If you are interested in OS development, systems programming, or seeing how an x86-64 kernel works under the hood, the entire codebase is open-source:

GitHub Repository: https://github.com/rahadbhuiya/Exploidus
Feel free to star the repo, clone it, run it in QEMU, or read through the kernel source!

Have you ever experimented with operating system development or bare-metal programming? What was the hardest low-level bug you ever encountered? Let's discuss in the comments below!

What if servers balanced load like Emperor Penguins? Meet HuddleCluster

Rahad Bhuiya — Fri, 04 Sep 2026 14:10:38 +0000

In Antarctic blizzards where temperatures drop below -40°C and winds scream past 200 km/h, Emperor Penguins survive without any central coordinator.

Their survival relies on a single distributed rule:

If you are freezing on the outer edge, push inward toward the core.
If you are getting too warm at the center, drift outward to cool down.

The colony continuously self-organizes, maintaining dynamic equilibrium and thermal fairness automatically.

As a systems engineer, I realized: Why aren't our server clusters doing the same thing?

The Problem With Traditional Load Balancers

Traditional load balancers (like NGINX, HAProxy, or standard round-robin proxies) rely on static thresholds, rigid timeout intervals, and binary health checks.

A server is either marked 100% "healthy" or abruptly declared "dead".

In unpredictable production environments (traffic spikes, noisy neighbors, or thermal throttling), this binary model frequently causes cascading cluster blackouts:

A node slows down under sudden traffic and its latency spikes.
Standard schedulers keep hammering it until a hard timeout threshold trips.
The load balancer abruptly severs the node completely from the cluster.
The remaining servers suddenly absorb 100% of the redirected traffic.
This sudden stampede overwhelms the healthy nodes, pushing them into saturation and triggering a domino-effect crash across the entire fleet.

How HuddleCluster Solves This

I built HuddleCluster (v4.15.0) to replace rigid binary thresholds with continuous, bio-inspired thermal attenuation.

Instead of a flat pool, servers self-organize into concentric rings:

Inner Core Ring: Actively serves critical, high-throughput requests.
Relative Anomaly Eviction: Rather than comparing metrics against arbitrary fixed numbers, nodes are continuously evaluated using moving-window Z-scores across the fleet.
Adaptive Cooling Ring: When a node experiences elevated latency, error rates, or compute pressure, it smoothly drifts to an outer ring. Its traffic weight is reduced, shedding load while allowing in-flight connections to drain cleanly.
Autonomous Convergence: Once the node's metrics cool down and normalize, it rotates back into the active core automatically—without human intervention or manual triage.

Production-Grade Multi-Node Fleet
Beyond single-instance routing, HuddleCluster provides an enterprise-ready distributed control plane:

Master-Agent Architecture: Built on FastAPI with a lightweight CLI tool (huddle-cluster).
High Availability (HA): Raft-simplified leader election, state persistence across restarts, and write forwarding.
Canary & Rolling Updates: Weight-based traffic splitting with automated health gates.
Kubernetes Native: Native Kubernetes Service Discovery and official Helm deployment charts.
Security: Fine-grained RBAC scopes and Mutual TLS (mTLS) node identity verification.
Observability: Structured JSON logging, distributed trace IDs, and Prometheus metrics.

Interactive Live Simulation & Source Code
I built an interactive HTML5 simulation where you can watch the penguin ring rotation algorithm live in your browser:

Live Interactive Website: https://rahadbhuiya.github.io/HuddleCluster/
GitHub Repository (MIT Licensed): https://github.com/rahadbhuiya/HuddleCluster
PyPI: pip install huddle-cluster
Preprint / Paper: Zenodo DOI: 10.5281/zenodo.20348019

What Do You Think?
I would love to hear feedback from backend engineers, SREs, and distributed systems enthusiasts:

How does your team currently prevent cascading microservice timeouts during traffic spikes?
What edge cases would you like to see benchmarked in future releases?
If you find this bio-inspired approach interesting, check out the repository, star it on GitHub, and let me know your thoughts in the comments below!

Quick Example in Python

HuddleCluster has a zero-dependency Python core and can be deployed in just a few lines of code:


python
import requests
from huddle_cluster import create_cluster

# Initialize the self-organizing pool
cluster = create_cluster([
    ("srv-01", "10.0.0.1", 8080),
    ("srv-02", "10.0.0.2", 8080),
    ("srv-03", "10.0.0.3", 8080),
])
cluster.start()

# Route requests with automatic thermal awareness
with cluster.get_server_context() as server:
    response = requests.get(f"http://{server.host}:{server.port}/api/data")
    print(f"Processed by node: {server.name}")

# Inspect self-healing status
print(cluster.health_report())
# Output: {"fairness_score": 0.94, "rotation_count": 12, "cluster_health": "healthy"}

cluster.stop()

Turing's Escape

Rahad Bhuiya — Fri, 19 Jun 2026 15:54:52 +0000

This is a submission for the June Solstice Game Jam

What I Built

Turing's Escape is a single-file browser puzzle game set in a fictional 1954 cryptography station, built around one idea: a literal Turing Test. The player is a new recruit who has to clear three tests before going home — break a Caesar-shifted intercept, talk to two telegraph operators and work out which one is human, then decode a punched-tape Morse message to open the vault. The middle test is the whole point: one operator's replies are scripted, the other's come live from Google's Gemini API, re-randomized every playthrough, and the player has to guess which is which with no hint beyond the conversation itself.
It ties into the jam themes on three levels at once rather than one per box: the "longest day" setting (a desk-lamp lit case file on the solstice night, light pooling against the dark edges of the page), the Alan Turing tribute (explained fully below), and a closing dedication that quietly lets Pride Month and Turing's own history sit in the same place, since June is both.
Play it: https://rahadbhuiya.itch.io/turings-escape
Built with: HTML, CSS, vanilla JavaScript, Google Gemini API — no backend, no build step, no framework.

Code

github: https://github.com/rahadbhuiya/turings-escape

How I Built It

Most jam entries treat their bonus categories as separate boxes to check: a Turing reference bolted onto otherwise generic gameplay, an AI API called once for a line of flavor text nobody reads twice. The decision that shaped everything else here was refusing to do that — building one mechanic that was simultaneously the theme, the tribute, and the AI integration, so pulling out any single piece would break the game rather than just shrink a credits screen. The other honest constraint was a five-day window and a zero-dollar API budget, which ruled out a backend, paid models, and any art we didn't have time to make.
What that left is a single static HTML file: no server, no database, no build step. The state machine, all three puzzle modules, and a small WebAudio engine that synthesizes every sound effect on the fly (there are no audio asset files in the project at all) run entirely client-side in vanilla JavaScript.

The only network call the game ever makes is the one outbound request to Gemini, for the live half of the Turing Test puzzle — everything else, including the entire "human" side of that same conversation, is local and offline by design. That made the game trivial to deploy as one file and just as trivial to demo with no internet connection at all, if a key isn't on hand.

Prize Category

Best Ode to Alan Turing — Turing isn't a skin on top of a generic escape room here; he's the reason the central mechanic exists. His 1950 paper, "Computing Machinery and Intelligence," asked whether a human judge could reliably tell a machine from a person through conversation alone — that question is the entire second test in this game. We chose not to put Turing on stage as a speaking character; he never appears, and nothing invents a quote in his name. Instead the closing screen dedicates the game to him as history rather than fiction: breaking Enigma, the 1950 paper, his 1952 prosecution for being gay under then-current UK law, and his death two years later at forty-one. It's also why the visuals are paper, typewriter type, and rubber stamps instead of a green-phosphor hacker terminal — interactive CRT screens barely existed yet in 1954, so the case-file look is the one actually true to his era, not just the one that reads as "computer-y" now.

Best Google AI Usage — One of the two operator lines, randomized to Line A or Line B every playthrough, is answered live by Google's Gemini API (gemini-2.5-flash) rather than pre-written. Every player message goes out with a system instruction that puts the model in character as a 1954 telegraph operator and tells it never to reveal it's an AI; whatever comes back streams straight into the chat log, unedited. Flash and Flash-Lite are the tiers Google currently leaves open for free-tier use, with Pro held back for paid accounts, which made Flash the only realistic choice on a zero-budget build. Because a live demo in front of judges is the worst place for a network call to fail, every Gemini request is wrapped end to end — a missing key, a failed fetch, a rate limit, or an empty response all fall through silently to a pool of scripted lines tuned to feel just slightly too smooth and agreeable, the uncanny-valley opposite of the deliberately imperfect lines written for the human side. The AI is real and load-bearing for the puzzle, but never a single point of failure for the submission itself.

A load balancer inspired by how Emperor Penguins survive Antarctic winters

Rahad Bhuiya — Mon, 15 Jun 2026 15:32:22 +0000

Why I modeled a load balancer after Emperor Penguin huddles

A few months ago I was reading about how emperor penguins survive Antarctic winters. Temperature drops to -40°C, wind hits 120km/h, and somehow these birds make it through. Not because they're individually tough. Because they rotate.

Cold penguins on the outside push inward. Warm ones from the center move out to rest. Nobody coordinates this. No penguin is in charge. It emerges from one simple rule: if you're cold, push in. If you're warm, you'll get pushed out eventually.

I couldn't stop thinking about this.

I was working on a service mesh at the time and dealing with the usual problem — one slow server quietly dragging down the whole cluster. Round robin doesn't care. Least connections helps but not always. Weighted approaches need manual tuning that goes stale immediately.

The penguin thing kept nagging at me. What if servers had a "temperature"? What if hot servers rotated out to rest?

That's HuddleCluster.

The basic structure

Two rings:

Inner ring (deque): Active servers. Requests go to them round-robin. Simple, fair, zero overhead for normal traffic.

Outer ring (min-heap): Resting servers. Keyed by temperature — coolest server sits at the top, ready to rotate back in first.

When a server in the inner ring runs hot past a threshold, it moves out. When an outer ring server cools down, it comes back in.

That's the entire rotation logic. About 50 lines of Python.

What is "temperature"?

This took me a while to get right.

My first attempt was just raw latency. That was bad. A server handling one slow database query looks terrible even when it's completely healthy. I needed something more composed.

Current formula:

pythontemperature = EMA(
0.7 * relative_latency_anomaly +
0.1 * cpu_score +
0.1 * memory_score +
0.1 * (error_rate + connection_score)
)

Three decisions here worth explaining.

EMA over simple moving average

EMA weights recent measurements more heavily. If a server just had a bad spike but recovered, EMA reflects that recovery faster than a window average would. The formula:

EMA_t = α * current_value + (1 - α) * EMA_{t-1}

Higher α means faster reaction but more noise sensitivity. Lower means smoother but slower detection. I tuned α empirically — the benchmark section covers what I observed.

Relative latency anomaly, not absolute latency

This is the part I'm most happy with.

Instead of flagging a server when latency crosses some hardcoded threshold like "300ms is bad," I compare each server to the cluster median:

pythonrelative_anomaly = (server_latency - cluster_median) / cluster_median

Why does this matter? If your whole cluster is running at 200ms — maybe it's a heavy batch job period, maybe your database is under load — that's just the current normal. A 200ms server shouldn't be punished when everyone is at 200ms. But a 400ms server in a 200ms cluster? That's a real anomaly.

No manual threshold needed. The system self-calibrates to whatever your current traffic looks like. 60ms is fine if the cluster median is 60ms. The scoring is scale-invariant.

70/30 split between latency and system metrics

Latency is the user-visible signal. CPU, memory, and connections are leading indicators — they can catch problems before latency visibly degrades. I weighted latency higher because that's ultimately what matters, but the other signals pull their weight.

Data structure choices

Inner ring as deque: Round-robin is just rotating a deque. Append to right, pop from left. O(1) for both. I tried a list first — the index tracking got messy and error-prone whenever servers got removed mid-rotation. Deque was cleaner and the right tool.

Outer ring as min-heap: I want the coolest resting server to come back in first. Min-heap gives me that in O(log n) for insertion and extraction. I briefly considered just sorting the outer ring on every update — fine at small n, but min-heap felt more principled and honest about the intent.

The whole thing is about 700 lines of Python with zero external dependencies. I deliberately avoided pulling in anything external. I wanted this to be droppable into any project without a dependency audit conversation.

What the benchmarks showed

I tested with 6 FastAPI servers on loopback (all on the same machine). That's a significant caveat — I'll address it directly in the limitations section.

Normal load: HuddleCluster performs comparably to round-robin and least-connections. No meaningful difference. That's expected. When nothing is degraded, rotation rarely triggers and the deque just does round-robin like anything else.

Server failure simulation: I introduced artificial 5-second delays on one server mid-test. This is where the gap appeared.

AlgorithmP95 LatencyRound Robin5,026msLeast Connections4,891msHuddleCluster85.6ms

Round-robin kept routing 1-in-6 requests to the slow server throughout the test. HuddleCluster evicted it after approximately 3 request cycles — detection converges in about 36 cluster requests on average.

Inner ring fairness: Gini coefficient was 0.00 in every test scenario. The deque distributes perfectly evenly among active servers.

Routing overhead: 10.7μs per request average. Acceptable for the use case.

Where it breaks

I think this section matters as much as the benchmark numbers.

Loopback is not production. Every benchmark I ran is on a single machine. WAN introduces higher base latency, more jitter, and failure modes I haven't tested. The EMA sensitivity I tuned for loopback may need adjustment for real network conditions — high per-server jitter could cause false evictions if α is too aggressive. This is the most honest gap in the current work.

The k ≥ n/2 problem. Relative scoring works well when a minority of servers degrade. If half or more of your cluster slows down simultaneously — shared database contention, a network event, a traffic spike hitting everyone — the cluster median shifts up and no individual server looks anomalous. The algorithm goes blind. I document this in the paper but haven't solved it yet.

No cross-host state. HuddleCluster runs per-process. Multiple load balancer instances don't share rotation state. There's a gossip protocol stub in v1.3.0 but it's not complete.

Detection speed

One thing that surprised me during benchmarking: how fast eviction actually happens in practice.

A 3× slower server gets rotated out in roughly 3 request cycles. For most traffic volumes that's well under a second. I expected slower convergence — the EMA smoothing should delay reaction — but the relative scoring amplifies the signal enough that threshold crossing happens fast.

What's next

WAN benchmarks are the obvious priority. I want to understand how the EMA α needs to change under real network jitter before claiming this is production-ready.

The multi-server simultaneous degradation case also needs empirical testing. I have theoretical analysis of why relative scoring breaks at k ≥ n/2, but I want to measure the actual degradation curve — at what point does detection start failing?

Distributed state via gossip is in progress.

GitHub: github.com/rahadbhuiya/HuddleCluster

Paper: doi.org/10.5281/zenodo.20348019

If you've worked with adaptive load balancing in production — especially anything with relative or percentile-based scoring — I'd be curious what threshold strategies held up under WAN jitter.