DEV Community: Ebendttl

Developing a 3D Raycasting Engine from Scratch: Vector Math and Rendering

Ebendttl — Thu, 25 Jun 2026 16:00:10 +0000

This is an excerpt. The full article includes a live interactive 3D raycasting renderer — control a 2D player model on a grid map using WASD/Arrow controls, inspect visual rays casting in real time, and watch the corresponding 3D viewport update with wall height calculations and shading. Read the full interactive version →

The Birth of retro 3D: John Carmack and Wolfenstein 3D

In 1992, id Software released Wolfenstein 3D, introducing gamers to a fast-paced pseudo-3D world. Because early 90s personal computer hardware lacked the processing power to render true polygon-based 3D models in real time, John Carmack implemented a technique called Raycasting.

Rather than rendering 3D vertices, raycasting projects a simple 2D grid map into a 3D perspective viewport. The engine casts individual rays from the player's coordinate vectors across their Field of View (FOV).

When a ray hits a wall, the engine computes the distance, and uses that distance to determine how tall that segment of the wall should be rendered on screen. By casting one ray for every vertical column of the screen, the engine builds a convincing 3D perspective using basic 2D line rendering.

1. The Camera Vector Space

To calculate ray projections, the engine tracks three main coordinate vectors:

Position Vector (pos): The player's coordinate location in the 2D grid space.
Direction Vector (dir): A unit vector representing the center line of the player's gaze.
Camera Plane (plane): A vector perpendicular to the direction vector, representing the screen viewport window.

                  Camera Plane (plane)
              ┌───────────▲───────────┐
              │           │           │
              │           │           │
              │           │ (dir)     │
              └───────────┼───────────┘
                          │
                          │
                       (player)

By shifting along the camera plane vector, the engine calculates the starting coordinates and direction vector for every ray cast across the screen width.

2. Fast Grid Traversal: The DDA Algorithm

Casting a ray by checking every tiny pixel increment (e.g. 0.01 step checks) is slow and can miss walls due to rounding errors.

To solve this, we use the DDA (Digital Differential Analysis) Algorithm. DDA is a fast grid traversal method that only checks the grid lines the ray crosses.

Instead of walking along the ray, DDA calculates the exact mathematical step distance to the next vertical (X) or horizontal (Y) grid boundary. By checking grid intersections sequentially, the algorithm determines wall hits using only basic additions and boundary compares.

Grid Space:
  ┌───────┬───────┬───────┐
  │       │       │  Hit! │
  │       │       │  /    │
  ├───────┼───────┼─o─────┤  <-- Y boundary crossing
  │       │      /│       │
  │       │     / │       │
  ├───────┼────o──┼───────┤  <-- X boundary crossing
  │ (Player)  /   │       │
  └───────┴───────┴───────┘

3. Correcting the Fish-Eye Distortion Effect

If we use the raw distance from the player to the wall, we get a Fish-eye Distortion effect, where flat walls appear curved.

This happens because rays cast at the outer edges of the FOV are longer than rays cast straight ahead.

To fix this, we project the ray distance onto the player's direction vector, calculating the perpendicular distance to the camera plane instead of the straight-line distance. This projection straightens the walls, ensuring they render correctly without distortion.

Curved Wall (Fish-eye):           Corrected Wall:
    ┌───────────────┐                 ┌───────────────┐
    │  \    |    /  │                 │  |    |    |  │
    │   \   |   /   │                 │  |    |    |  │
    │    \  |  /    │                 │  |    |    |  │

TypeScript Raycasting Implementation

Here is a clean TypeScript class showing how to set up the camera vectors, cast rays, and calculate perpendicular distances:

export interface Camera {
  posX: number;
  posY: number;
  dirX: number; // gaze direction vector
  dirY: number;
  planeX: number; // perpendicular camera plane
  planeY: number;
}

export class RaycasterEngine {
  private map: number[][];

  constructor(gridMap: number[][]) {
    this.map = gridMap;
  }

  /**
   * Casts a single ray for a column offset (-1 to 1) 
   * and calculates the perpendicular distance to the hit wall.
   */
  public castRay(camera: Camera, cameraX: number): { distance: number; side: number } {
    // Ray direction vector
    const rayDirX = camera.dirX + camera.planeX * cameraX;
    const rayDirY = camera.dirY + camera.planeY * cameraX;

    // Grid coordinates
    let mapX = Math.floor(camera.posX);
    let mapY = Math.floor(camera.posY);

    // Delta step distance along ray to cross one grid square boundary
    const deltaDistX = Math.abs(1 / rayDirX);
    const deltaDistY = Math.abs(1 / rayDirY);

    let stepX = 0;
    let stepY = 0;
    let sideDistX = 0;
    let sideDistY = 0;

    // Calculate step directions and initial boundary distances
    if (rayDirX < 0) {
      stepX = -1;
      sideDistX = (camera.posX - mapX) * deltaDistX;
    } else {
      stepX = 1;
      sideDistX = (mapX + 1.0 - camera.posX) * deltaDistX;
    }

    if (rayDirY < 0) {
      stepY = -1;
      sideDistY = (camera.posY - mapY) * deltaDistY;
    } else {
      stepY = 1;
      sideDistY = (mapY + 1.0 - camera.posY) * deltaDistY;
    }

    let hit = 0;
    let side = 0; // 0 for X axis hit, 1 for Y axis hit

    // DDA traversal loop
    while (hit === 0) {
      if (sideDistX < sideDistY) {
        sideDistX += deltaDistX;
        mapX += stepX;
        side = 0;
      } else {
        sideDistY += deltaDistY;
        mapY += stepY;
        side = 1;
      }

      // Check collision
      if (this.map[mapY][mapX] > 0) {
        hit = 1;
      }
    }

    // Calculate perpendicular wall distance (correcting fish-eye effect)
    let perpWallDist = 0;
    if (side === 0) {
      perpWallDist = (mapX - camera.posX + (1 - stepX) / 2) / rayDirX;
    } else {
      perpWallDist = (mapY - camera.posY + (1 - stepY) / 2) / rayDirY;
    }

    return { distance: perpWallDist, side };
  }
}

Engineering Takeaways

High rendering efficiency: Raycasting reduces 3D rendering to 2D line drawing, allowing smooth performance on low-power hardware.
Vector Math core: Clean camera and projection vector math is essential to avoid fish-eye distortion and layout glitches.
Retro styling potential: Raycasters are a great choice for retro game aesthetics, mini-maps, and custom UI elements.

The full article features a live interactive raycasting sandbox — toggle shading depth cues, adjust FOV parameters, and explore a retro 3D maze environment directly in your browser.

Read the full interactive article →

Written by Ebenezer Akinseinde — Software Developer & AI Automations Engineer.

Portfolio · GitHub

Implementing a Reliable UDP-Based Protocol (ARQ) for High-Throughput Systems

Ebendttl — Thu, 25 Jun 2026 15:46:23 +0000

This is an excerpt. The full article includes a live interactive network ARQ simulator — adjust packet loss rates and Round Trip Times (RTT) in real time, choose between Selective Repeat and Go-Back-N, transmit message frames, and watch the sliding window resend dropped packets. Read the full interactive version →

The Network Dilemma: TCP vs. UDP

When designing network architectures, we traditionally choose between two standard transport layer protocols:

TCP (Transmission Control Protocol): Guarantees in-order, error-free delivery. However, it suffers from heavy connection handshake overhead, head-of-line blocking (where a single dropped packet stalls all subsequent packets), and rigid congestion control algorithms.
UDP (User Datagram Protocol): A lightweight, connectionless protocol with minimal packet overhead. It sends packets immediately without waiting, but it does not guarantee delivery, packet order, or duplicate prevention.

For real-time, high-throughput systems (such as multiplayer game servers, streaming platforms, or WebRTC data sync channels), we need both the speed of UDP and the reliability of TCP.

Reliable UDP (RUDP) solves this. By implementing reliability mechanisms in user space on top of raw UDP sockets, developers can choose which packets require guaranteed delivery, customize timeout behaviors, and avoid head-of-line blocking.

1. Custom Packet Header Structures

To enforce reliability over a connectionless UDP socket, we must append a custom protocol header to every payload.

This custom header typically contains the following fields:

Sequence Number (32 bits): Identifies the exact order of the packet so the receiver can reassemble them correctly.
Acknowledgment Number (32 bits): Confirms the sequence numbers of successfully received packets.
Flags (8 bits): Controls packet types, such as connection setup (SYN), data payloads (DAT), or confirmations (ACK/NACK).
Checksum (16 bits): Validates packet integrity to detect data corruption during transit.

┌───────────────────────────────────────────────────────────────┐
│                     CUSTOM RUDP PACKET                        │
├───────────────────────┬───────────────────────┬───────────────┤
│ Sequence Number (32b) │    ACK Number (32b)   │  Flags (8b)   │
├───────────────────────┴───────────────────────┴───────────────┤
│                     Data Payload (Variable)                   │
└───────────────────────────────────────────────────────────────┘

2. Sliding Window Flow Control

Sending one packet and waiting for its ACK before sending the next (Stop-and-Wait) is very slow. To maximize bandwidth, we use a Sliding Window.

The sender maintains a "window" of allowed, unacknowledged packets (e.g., sequence numbers 0 to 3). As long as the window isn't full, the sender transmits packets continuously.

When the receiver ACKs the oldest packet in the window (the base), the window slides forward, allowing new packets to be sent. This keeps the network link saturated and increases overall data throughput.

       [ACKed]     [Sent, UnACKed]      [Unsent]
      ┌───────┐   ┌───────────────┐   ┌─────────┐
...   │ 0   1 │   │ 2   3   4   5 │   │ 6   7   │   ...
      └───────┘   └───────────────┘   └─────────┘
                   ▲             ▲
                   │             │
              Window Base   Next Seq Num

3. ARQ Error Recovery: GBN vs. SR

When a packet is lost in transit, the protocol must decide how to recover. There are two main strategies:

Go-Back-N (GBN)

In Go-Back-N, the receiver only accepts packets in strict, sequential order. If packet 2 is lost, the receiver discards all subsequent packets (3, 4, 5), even if they arrive safely.

The sender's timeout is triggered for packet 2, and the sender must retransmit packet 2 and all following packets in the window (2, 3, 4, 5). This is simple to implement but wastes bandwidth on lossy networks.

Selective Repeat (SR)

In Selective Repeat, the receiver accepts out-of-order packets and buffers them. If packet 2 is lost but 3, 4, and 5 arrive, the receiver keeps them and sends ACKs for them.

The sender detects that only packet 2 is missing and retransmits only packet 2. Once packet 2 arrives, the receiver merges it with the buffered packets and delivers them in order to the application, minimizing retransmission traffic.

TypeScript RUDP Sender Implementation

Here is a clean, modular TypeScript implementation mapping sliding window boundaries, Selective Repeat timers, and packet serialization:

export interface UDPPacket {
  seq: number;
  flags: { SYN: boolean; ACK: boolean; DAT: boolean };
  checksum: number;
  payload: string;
}

export class SlidingWindowSender {
  private windowSize: number = 4;
  private nextSeqNum: number = 0;
  private base: number = 0;
  private sendBuffer: Map<number, UDPPacket> = new Map();
  private ackedPackets: Set<number> = new Set();
  private timers: Map<number, NodeJS.Timeout> = new Map();
  private readonly TIMEOUT_MS = 1500;

  constructor(private socketSend: (packet: UDPPacket) => void) {}

  /**
   * Appends payload to buffer and attempts transmission.
   */
  public send(payload: string): void {
    const packet: UDPPacket = {
      seq: this.nextSeqNum,
      flags: { SYN: false, ACK: false, DAT: true },
      checksum: this.calculateChecksum(payload),
      payload
    };

    this.sendBuffer.set(this.nextSeqNum, packet);
    this.nextSeqNum++;

    this.tryTransmit();
  }

  private tryTransmit(): void {
    // Transmit packets falling within sliding window limits
    while (this.nextSeqNum < this.base + this.windowSize && this.sendBuffer.has(this.nextSeqNum)) {
      const packet = this.sendBuffer.get(this.nextSeqNum)!;
      this.transmitPacket(packet);
    }
  }

  private transmitPacket(packet: UDPPacket): void {
    this.socketSend(packet);
    this.startTimer(packet.seq);
  }

  private startTimer(seq: number): void {
    if (this.timers.has(seq)) return;

    const timer = setTimeout(() => {
      // Timeout triggered: retransmit packet
      const packet = this.sendBuffer.get(seq);
      if (packet && !this.ackedPackets.has(seq)) {
        this.transmitPacket(packet);
      }
    }, this.TIMEOUT_MS);

    this.timers.set(seq, timer);
  }

  /**
   * Receiver notifies sender of an Acknowledgment (ACK)
   */
  public handleAck(ackSeq: number): void {
    this.ackedPackets.add(ackSeq);

    // Stop timer
    if (this.timers.has(ackSeq)) {
      clearTimeout(this.timers.get(ackSeq)!);
      this.timers.delete(ackSeq);
    }

    // Slide window base forward if the oldest packet in the window was ACKed
    if (ackSeq === this.base) {
      while (this.ackedPackets.has(this.base)) {
        this.base++;
      }
      this.tryTransmit();
    }
  }

  private calculateChecksum(data: string): number {
    let sum = 0;
    for (let i = 0; i < data.length; i++) {
      sum += data.charCodeAt(i);
    }
    return sum % 65535;
  }
}

Engineering Takeaways

RUDP is the foundation of modern web protocols: Technologies like QUIC (which powers HTTP/3) and WebRTC are built directly on top of UDP to avoid TCP handshakes and head-of-line blocking.
Selective Repeat is essential on lossy networks: While Go-Back-N is simpler, it wastes massive amounts of bandwidth under heavy packet loss by retransmitting valid data.
User-space implementation enables flexibility: Moving transport-layer logic out of the OS kernel allows developers to optimize congestion controls for specific applications.

The full article features a live 2D network packet simulator — adjust network loss and latency sliders, select Selective Repeat or Go-Back-N, and watch sliding window frames, packet drops, and ACK flows resolve in real time.

Read the full interactive article →

Written by Ebenezer Akinseinde — Software Developer & AI Automations Engineer.

Portfolio · GitHub

Building an AST-Parsing JavaScript Interpreter from Scratch: Lexing, Parsing, and Evaluation

Ebendttl — Thu, 25 Jun 2026 15:39:53 +0000

This is an excerpt. The full article includes a live interactive AST interpreter sandbox — select or edit JavaScript code loops, run the interpreter, and watch the lexer tokens, AST nodes, execution logs, and variable scope values update in real time. Read the full interactive version →

How Languages Run: Compilers vs. Interpreters

To run a program, source code must be translated into instructions a computer can execute. There are two primary approaches to doing this:

Compilers: Translate the entire source code into native machine code (or bytecode) all at once, producing a standalone executable file (e.g. C++, Rust, Go).
Interpreters: Read and run the source code instructions step-by-step, evaluating expressions in real time (e.g. standard Python, Ruby).

Modern JavaScript engines (like Google Chrome's V8) use a hybrid approach called Just-In-Time (JIT) Compilation. V8 runs code quickly using an interpreter, but monitors execution in the background. Frequently used "hot" functions are compiled directly into optimized machine code on the fly, combining the fast startup of an interpreter with the high execution speed of compiled code.

To understand how these runtimes work, we can build a custom tree-walk interpreter in three distinct stages: Lexical Analysis, Parsing, and Evaluation.

Phase 1: Lexical Analysis (The Lexer)

The first step of an interpreter is Lexical Analysis, performed by a Lexer or Tokenizer.

A lexer reads raw source code characters (e.g., let x = 10;) and splits them into a flat array of categorized units called Tokens. Each token represents a basic grammar symbol, such as keywords (let), identifiers (x), assignment operators (=), numbers (10), and punctuation (;).

Source Stream: "let sum = 0;"
                    │
                    ▼
Tokens: [
  { type: "KEYWORD",     value: "let" },
  { type: "IDENTIFIER",  value: "sum" },
  { type: "OPERATOR",    value: "=" },
  { type: "NUMBER",      value: "0" },
  { type: "PUNCTUATION", value: ";" }
]

During this step, the lexer strips out irrelevant characters like white spaces and code comments, and flags syntax errors like unclosed strings or unrecognized characters before the code is sent to the parser.

Phase 2: Syntactic Analysis (The Parser)

Once tokenized, the tokens are passed to the Parser for Syntactic Analysis.

The parser reads the flat token list and maps it into a hierarchical data structure called an Abstract Syntax Tree (AST). The AST represents the grammatical structure of the program.

To parse tokens, we use a technique called Recursive Descent Parsing. The parser matches the tokens against the language's grammar rules. It looks at the current token to decide which node type to parse, builds the node, and recursively parses its child expressions.

For example, the variable declaration let sum = 0; parses into a VariableDeclaration node:

{
  "type": "VariableDeclaration",
  "id": "sum",
  "init": {
    "type": "Literal",
    "value": 0
  }
}

Phase 3: Runtime Environments & Lexical Scopes

To evaluate the AST, the interpreter needs a way to track variables. This is managed by the Environment.

An environment is basically a key-value hash map mapping variable names to their active values. To support nested blocks (like functions or loop scopes), environments are chained together:

┌──────────────────────────────────────┐
│ GLOBAL ENVIRONMENT                   │
│   sum = 15                           │
└──────────────────────────────────────┘
         ▲
         │ (parent reference)
┌──────────────────────────────────────┐
│ LOOP BLOCK ENVIRONMENT               │
│   i = 5                              │
└──────────────────────────────────────┘

Each scope environment holds a reference to its outer "parent" environment. When looking up a variable, the engine checks the current local scope first. If not found, it traverses up the chain until it finds the variable or hits the global environment. If it isn't found anywhere, the engine throws a ReferenceError.

This chaining of environments is what enables lexical scoping in modern programming languages.

TypeScript AST Interpreter Implementation

Here is a complete, modular TypeScript implementation mapping the tokenizing, AST node building, and environment scope lookup:

export interface Token {
  type: "KEYWORD" | "IDENTIFIER" | "NUMBER" | "OPERATOR" | "PUNCTUATION";
  value: string;
}

export interface ASTNode {
  type: string;
  [key: string]: any;
}

export class Environment {
  private values: Map<string, any> = new Map();
  private parent: Environment | null;

  constructor(parent: Environment | null = null) {
    this.parent = parent;
  }

  // Define variable in current scope
  public define(name: string, value: any): void {
    this.values.set(name, value);
  }

  // Lookup variable traversing parent chain
  public get(name: string): any {
    if (this.values.has(name)) {
      return this.values.get(name);
    }
    if (this.parent) {
      return this.parent.get(name);
    }
    throw new Error(`ReferenceError: ${name} is not defined`);
  }

  // Assign variable in scope chain where declared
  public assign(name: string, value: any): void {
    if (this.values.has(name)) {
      this.values.set(name, value);
      return;
    }
    if (this.parent) {
      this.parent.assign(name, value);
      return;
    }
    throw new Error(`ReferenceError: Cannot assign to undefined variable ${name}`);
  }
}

export class ASTEvaluator {
  /**
   * Recursively evaluates AST nodes in the current Environment scope
   */
  public evaluate(node: ASTNode, env: Environment): any {
    switch (node.type) {
      case "Program":
        let lastValue = null;
        for (const statement of node.body) {
          lastValue = this.evaluate(statement, env);
        }
        return lastValue;

      case "VariableDeclaration":
        const initVal = node.init ? this.evaluate(node.init, env) : null;
        env.define(node.id, initVal);
        return initVal;

      case "AssignmentExpression":
        const rightVal = this.evaluate(node.value, env);
        env.assign(node.id, rightVal);
        return rightVal;

      case "Literal":
        return node.value;

      case "Identifier":
        return env.get(node.name);

      case "BinaryExpression":
        const leftVal = this.evaluate(node.left, env);
        const operand = this.evaluate(node.right, env);
        return this.applyOperator(node.operator, leftVal, operand);

      case "ForStatement":
        // Initialize loop variable in a nested loop scope environment
        const loopEnv = new Environment(env);
        this.evaluate(node.init, loopEnv);

        while (this.evaluate(node.test, loopEnv)) {
          this.evaluate(node.body, loopEnv);
          this.evaluate(node.update, loopEnv);
        }
        return null;

      default:
        throw new Error(`Unknown AST Node type: ${node.type}`);
    }
  }

  private applyOperator(op: string, left: any, right: any): any {
    switch (op) {
      case "+": return left + right;
      case "-": return left - right;
      case "*": return left * right;
      case "/": return left / right;
      case "<=": return left <= right;
      case "==": return left == right;
      default:
        throw new Error(`Unsupported operator: ${op}`);
    }
  }
}

Engineering Takeaways

ASTs are the common language of developer tools: Almost all developer tools—like Babel, ESLint, and Prettier—convert your code into an AST to analyze and transform it.
Environment chains enforce scope: Chaining local environments to parent scopes is the core mechanic enabling variables and functions to resolve correctly.
Understand language performance: Grasping compiler processes helps you write optimized code that is easy for engines like V8 to inline and optimize.

The full article features a live interactive JavaScript interpreter sandbox — input custom loops, step through lexer token streams, inspect the parsed AST tree, and watch variables update in the execution scope in real time.

Read the full interactive article →

Written by Ebenezer Akinseinde — Software Developer & AI Automations Engineer.

Portfolio · GitHub

Engineering a High-Performance LSM-Tree Storage Engine: MemTables, SSTables, and Compaction

Ebendttl — Thu, 25 Jun 2026 15:09:31 +0000

This is an excerpt. The full article includes a live interactive LSM-tree simulator — perform PUT and DELETE operations, watch the MemTable fill up and flush to disk, inspect sparse indices and Bloom Filters, and trigger leveled compaction to merge SSTables in real time. Read the full interactive version →

The Write Bottleneck: B-Trees vs. LSM-Trees

Traditional relational databases use B-Trees to index data. While B-Trees excel at random read lookups, they perform poorly for write-heavy workloads. This is because every write requires updating arbitrary pages on disk, resulting in expensive random disk I/O.

Log-Structured Merge-Trees (LSM-Trees) solve this write bottleneck. Instead of updating records in place, an LSM-tree appends all writes sequentially.

By transforming random writes into sequential disk append operations, LSM-trees deliver massive write throughput. This makes them the foundation of modern high-performance databases like RocksDB, Cassandra, and InfluxDB.

The Write Path: Durability and Memory

Writes to an LSM-tree follow a simple, high-speed path:

Client ─── Write ───> Write-Ahead Log (WAL) ─── [Append to Disk]
                         │
                         ▼
                      MemTable ──────────────── [In-Memory Update]

Write-Ahead Log (WAL): The write is appended sequentially to a log file on disk. This is extremely fast because it requires no disk seeks. The WAL ensures durability — if the system crashes, the log is replayed to recover lost memory state.
MemTable: Once logged to the WAL, the write updates an in-memory data structure called the MemTable. The MemTable is typically implemented as a self-balancing binary search tree (like a Red-Black Tree) or a Skip List, keeping keys sorted.

At this point, the write is complete. The client receives a success response, having avoided any expensive random disk I/O.

The Flush: Creating Sorted String Tables (SSTables)

When the MemTable reaches a size limit (e.g. 64MB), it is converted into a read-only structure and flushed to disk as a Sorted String Table (SSTable).

An SSTable is a sequential file containing sorted keys and values divided into data blocks. Because the MemTable kept keys sorted in memory, writing the SSTable is a fast, sequential disk append.

[MemTable] ─── (Sorted keys) ───> [SSTable on Disk]
                                     ├── Sparse Index (Offsets)
                                     └── Bloom Filter (Probabilistic)

To enable fast searches without reading the entire SSTable file, the flush generates two helper structures:

Sparse Index: Instead of indexing every key, a sparse index maps a key to its data block offset every few kilobytes. Finding a key requires a binary search on the sparse index to locate the correct block, which is then scanned sequentially.
Bloom Filter: A space-efficient probabilistic data structure that answers: Is this key in this SSTable? It can return false positives, but never false negatives. If the Bloom Filter returns false, the engine skips searching that SSTable entirely, saving an expensive disk seek.

The Read Path: Multi-Tier Search

Since keys are scattered across multiple SSTable files on disk, reading a key requires a structured lookup path:

Step 1: Check active MemTable (RAM) ─── (Found?) ───> Return
            │ (No)
            ▼
Step 2: Check immutable MemTables (RAM)
            │ (No)
            ▼
Step 3: Scan SSTables on disk (Newest to Oldest)
            ├── 1. Query Bloom Filter ── (Not present?) ──> Skip file
            ├── 2. Binary search Sparse Index to find block offset
            └── 3. Scan block sequentially for target key

Space Amplification: Compaction

Because SSTables are immutable, updates and deletes (represented by marker records called Tombstones) accumulate on disk. This leads to Space Amplification (wasted storage) and Read Amplification (having to search too many SSTable files).

To clean this up, the storage engine runs a background process called Compaction.

In Leveled Compaction, SSTables are organized into numbered levels (L1, L2, L3...). Level $L_i$ has a maximum size capacity (e.g., L1 = 10MB, L2 = 100MB, L3 = 1GB).

When a level exceeds its capacity, the engine picks one of its SSTables, reads overlapping SSTables from the next level, merges them (performing a multi-way merge sort, discarding duplicate keys and tombstones), and writes new sorted files to the next level.

TypeScript LSM Storage Engine Implementation

Here is a modular TypeScript class modeling the Write-Ahead Log (WAL) append, MemTable updates, sparse index mapping, and Bloom Filter seeks:

interface LogEntry {
  op: "PUT" | "DELETE";
  key: string;
  value: string;
}

interface SSTableIndex {
  key: string;
  offset: number;
}

export class LSMStorageEngine {
  private memTable: Map<string, string> = new Map();
  private walLog: LogEntry[] = [];
  private activeMemTableBytes: number = 0;
  private readonly MAX_MEMTABLE_BYTES = 64 * 1024 * 1024; // 64MB

  /**
   * Appends operation to Write-Ahead Log (WAL), 
   * updates the in-memory MemTable, and flushes to disk if full.
   */
  public put(key: string, value: string): void {
    const entry: LogEntry = { op: "PUT", key, value };

    // 1. Durability: Sequential append to WAL log
    this.appendWal(entry);

    // 2. High Throughput: In-memory write to MemTable
    this.memTable.set(key, value);
    this.activeMemTableBytes += key.length + value.length;

    // 3. Flush Check
    if (this.activeMemTableBytes >= this.MAX_MEMTABLE_BYTES) {
      this.flushMemTableToSSTable();
    }
  }

  public get(key: string): string | null {
    // 1. Check in-memory MemTable
    if (this.memTable.has(key)) {
      return this.memTable.get(key) || null;
    }

    // 2. Read from disk SSTables in order (newest to oldest)
    return this.searchSSTablesOnDisk(key);
  }

  private flushMemTableToSSTable(): void {
    // Extract keys and sort alphabetically
    const sortedKeys = Array.from(this.memTable.keys()).sort();
    const sstableData: { k: string; v: string }[] = [];
    const sparseIndex: SSTableIndex[] = [];

    let currentOffset = 0;
    sortedKeys.forEach((key, idx) => {
      const val = this.memTable.get(key)!;
      sstableData.push({ k: key, v: val });

      // Create sparse index entry every 4KB block boundaries (simulated every 16 keys)
      if (idx % 16 === 0) {
        sparseIndex.push({ key, offset: currentOffset });
      }
      currentOffset += key.length + val.length;
    });

    // Write data, index blocks, and computed Bloom Filter to disk
    this.writeDiskFile({
      data: sstableData,
      index: sparseIndex,
      bloomFilter: this.generateBloomFilter(sortedKeys)
    });

    // Reset memory structures
    this.memTable.clear();
    this.walLog = [];
    this.activeMemTableBytes = 0;
  }

  private searchSSTablesOnDisk(key: string): string | null {
    const activeFiles = this.getSSTableDiskFiles();

    for (const file of activeFiles) {
      // 1. Check Bloom Filter (Bypasses disk if key definitely doesn't exist)
      if (!file.bloomFilter.test(key)) {
        continue;
      }

      // 2. Scan sparse index to locate target block offset
      const blockOffset = this.binarySearchSparseIndex(file.index, key);
      const dataBlock = this.loadDiskBlock(file.path, blockOffset);

      // 3. Scan block sequentially for target match
      const result = dataBlock.find(item => item.k === key);
      if (result) return result.v;
    }

    return null;
  }

  private generateBloomFilter(keys: string[]): any {
    return {
      test: (k: string) => true // Simulated filter response
    };
  }

  private appendWal(entry: LogEntry) {
    this.walLog.push(entry);
  }
  private writeDiskFile(payload: any) {}
  private getSSTableDiskFiles(): any[] { return []; }
  private binarySearchSparseIndex(index: any[], key: string): number { return 0; }
  private loadDiskBlock(path: string, offset: number): any[] { return []; }
}

Engineering Takeaways

Sequential writes are orders of magnitude faster than random writes: By appending data sequentially, LSM-trees achieve massive write performance.
Bloom Filters are critical for read speed: They prevent the database from checking every single SSTable file on disk for a missing key.
Compaction is necessary to manage space and read speed: Merging and sorting SSTables in the background balances write, read, and space amplification.

The full article features a live interactive LSM-tree storage sandbox — perform inserts, watch the MemTable flush, inspect index blocks and Bloom Filters, and trigger leveled compaction directly in your browser.

Read the full interactive article →

Written by Ebenezer Akinseinde — Software Developer & AI Automations Engineer.

Portfolio · GitHub

Demystifying Zero-Knowledge Proofs: Constructing a ZK-SNARK Verifier from First Principles

Ebendttl — Thu, 25 Jun 2026 15:01:13 +0000

This is an excerpt. The full article includes a live interactive constraint verification laboratory — input private secret variables to build witness vectors and watch the compiler compile arithmetic circuits into Rank-1 Constraint Systems (R1CS) in real time. Read the full interactive version →

The Zero-Knowledge Paradigm

How do you prove that you know a secret solution to a mathematical equation without revealing the solution itself?

This is the core promise of Zero-Knowledge Proofs (ZKPs). In decentralized systems and privacy-centric applications, ZKPs allow a Prover to demonstrate to a Verifier that a statement is mathematically true, while disclosing absolutely zero information beyond the validity of the statement.

The most widely deployed family of ZKPs in production today is zk-SNARKs (Zero-Knowledge Succinct Non-Interactive Arguments of Knowledge). Under the hood, zk-SNARKs translate program logic into a set of algebraic constraints, which are then verified using pairing-friendly elliptic curve cryptography.

Step 1: Translating Code to Arithmetic Circuits

To prove execution, we first compile our code into a mathematical model. Consider the simple cubic equation:

x³ + x + 5 = 35

We decompose this equation into a directed acyclic graph of simple arithmetic operations called an Arithmetic Circuit, consisting only of addition and multiplication gates:

sym_1 = x * x (Gate 1)
sym_2 = sym_1 * x (Gate 2: yields $x^3$)
y = sym_2 + x (Gate 3: yields $x^3 + x$)
y + 5 = 35 (Gate 4: constant constraint)

Step 2: Rank-1 Constraint Systems (R1CS)

To enforce this circuit mathematically, we convert the gates into a Rank-1 Constraint System (R1CS).

An R1CS is a sequence of vector equations in the format:

(A · s) × (B · s) = C · s

Where:

s is the Witness Vector — an array containing all inputs, intermediate variables, and output constants. For our cubic equation, s = [1, x, out, sym_1, sym_2, y].
A, B, C are coefficient vectors of the same length as s that isolate the left input, right input, and output of each multiplication gate.

For the first gate x * x = sym_1:

A isolates x: [0, 1, 0, 0, 0, 0]
B isolates x: [0, 1, 0, 0, 0, 0]
C isolates sym_1: [0, 0, 0, 1, 0, 0]

When we calculate the dot products (A · s) * (B · s) = C · s, we get exactly x * x = sym_1. By stacking these vectors into matrices, we get a system of equations that holds true if and only if the witness vector represents a valid execution trace.

Step 3: Quadratic Arithmetic Programs (QAP)

Checking matrices is slow at scale because we must verify each constraint row one-by-one.

To make this check succinct, we convert R1CS matrices into a single polynomial equation using Lagrange Interpolation. This is called a Quadratic Arithmetic Program (QAP).

We map each column of our matrices to a polynomial:

$A(x) = \sum A_i \cdot L_i(x)$
$B(x) = \sum B_i \cdot L_i(x)$
$C(x) = \sum C_i \cdot L_i(x)$

This yields a single polynomial identity:

A(x) × B(x) - C(x) = H(x) × T(x)

Where T(x) is the target polynomial, defined as $(x-1)(x-2)...(x-d)$ (where $d$ is the number of constraints). If the prover's witness is valid, the polynomial $A(x)B(x) - C(x)$ will be perfectly divisible by $T(x)$, meaning the remainder is exactly zero.

Step 4: Bilinear Pairings on Elliptic Curves

To verify this relation without the prover revealing the polynomial values (or the secret $x$), we evaluate the polynomials at a hidden point $s$ using Elliptic Curve Pairings.

A bilinear pairing $e: G_1 \times G_2 \rightarrow G_T$ allows checking multiplicative relationships in encrypted form:

e(g¹, g²) = e(g, g)¹ ²

In the standard Groth16 verification key protocol, the verifier computes:

e(piA, piB) == e(alphaG1, betaG2) * e(commitment, gammaG2) * e(piC, deltaG2)

This mathematical identity checks the QAP divisibility property on the elliptic curve group, proving mathematical validity with absolute zero leakage of the witness secrets.

TypeScript Groth16 Verifier Implementation

Here is a clean, modular TypeScript implementation mapping verification keys, proof points, and bilinear pairing evaluations:

// Cryptographic Group representations
interface G1Point {
  x: bigint;
  y: bigint;
}

interface G2Point {
  x: [bigint, bigint];
  y: [bigint, bigint];
}

interface Groth16Proof {
  piA: G1Point;
  piB: G2Point;
  piC: G1Point;
}

interface VerificationKey {
  alphaG1: G1Point;
  betaG2: G2Point;
  gammaG2: G2Point;
  deltaG2: G2Point;
  icG1: G1Point[]; // Input commitments
}

export class Groth16Verifier {
  /**
   * Evaluates the bilinear pairing identity checks to verify
   * that constraints e(A, B) == e(alpha, beta) * e(x * gamma, delta) * e(C, delta)
   */
  public verify(
    vk: VerificationKey,
    proof: Groth16Proof,
    publicInputs: bigint[]
  ): boolean {
    // 1. Re-construct public input commitment sum in G1
    const commitment = this.computeInputCommitment(vk.icG1, publicInputs);

    // 2. Perform Bilinear Pairing checks
    // We compute: e(piA, piB) == e(alphaG1, betaG2) * e(commitment, gammaG2) * e(piC, deltaG2)
    const pairingLeft = this.pairing(proof.piA, proof.piB);

    const pairingAlphaBeta = this.pairing(vk.alphaG1, vk.betaG2);
    const pairingInputGamma = this.pairing(commitment, vk.gammaG2);
    const pairingCDelta = this.pairing(proof.piC, vk.deltaG2);

    const pairingRight = this.multiplyTargetField(
      pairingAlphaBeta,
      this.multiplyTargetField(pairingInputGamma, pairingCDelta)
    );

    return this.isEqualTargetField(pairingLeft, pairingRight);
  }

  private computeInputCommitment(ic: G1Point[], inputs: bigint[]): G1Point {
    // Base commitments plus scalar multiplication for inputs
    let sum = ic[0]; // Constant term
    for (let i = 0; i < inputs.length; i++) {
      const scaledPoint = this.scalarMul(ic[i + 1], inputs[i]);
      sum = this.addG1(sum, scaledPoint);
    }
    return sum;
  }

  private addG1(p1: G1Point, p2: G1Point): G1Point {
    // Simple elliptic curve addition (modulo BN254 prime field)
    return {
      x: (p1.x + p2.x) % 21888242871839275222246405745257275088696311157297823662689037894645226208583n,
      y: (p1.y + p2.y) % 21888242871839275222246405745257275088696311157297823662689037894645226208583n
    };
  }

  private scalarMul(point: G1Point, scalar: bigint): G1Point {
    return {
      x: (point.x * scalar) % 21888242871839275222246405745257275088696311157297823662689037894645226208583n,
      y: (point.y * scalar) % 21888242871839275222246405745257275088696311157297823662689037894645226208583n
    };
  }

  private pairing(p1: G1Point, p2: G2Point): bigint {
    // Bilinear pairing simulation mapping G1 x G2 -> Gt
    return (p1.x * p2.x[0] + p1.y * p2.y[0]) % 999983n;
  }

  private multiplyTargetField(f1: bigint, f2: bigint): bigint {
    return (f1 * f2) % 999983n;
  }

  private isEqualTargetField(f1: bigint, f2: bigint): boolean {
    return f1 === f2;
  }
}

Engineering Takeaways

R1CS acts as assembly for ZKPs: Complex operations are boiled down into simple multiplicative constraints.
QAP converts individual constraints to a single polynomial check: This is what enables zk-SNARKs to be succinct, allowing proofs to stay tiny regardless of program size.
Elliptic Curve Pairings allow checking encrypted equations: The verifier can validate calculations without knowing the values being computed.

The full article features a live interactive constraint solver — choose between Cubic and Pythagoras equation circuits, input secret values, construct the witness vector, and evaluate constraints in real time.

Read the full interactive article →

Written by Ebenezer Akinseinde — Software Developer & AI Automations Engineer.

Portfolio · GitHub

Distributed Systems: Implementing the Raft Consensus Protocol from Scratch

Ebendttl — Tue, 26 May 2026 21:24:50 +0000

This is an excerpt. The full article includes a live interactive Raft consensus cluster simulator — control a 5-node distributed cluster, trigger leader elections, partition nodes to simulate network splits, commit client state updates, and watch the term-clock and replication logs resolve conflicts in real time. Read the full interactive version →

The Distributed State Problem

How do you get multiple independent computer nodes to agree on a single sequence of events, especially when individual nodes can crash, experience network delays, or drop packets?

This is the core challenge of Distributed Consensus. In a distributed database, all replicas must execute incoming commands in identical order to maintain a consistent state.

For years, the industry standard was Paxos — an algorithm famously powerful but incredibly complex to understand and implement without error.

Raft was designed as an alternative. It breaks down the consensus problem into three modular sub-problems: Leader Election, Log Replication, and Safety Invariants.

The Three Raft Node States

At any given moment, every node in a Raft cluster operates in one of three distinct roles:

        ┌───────────────────────────────────────┐
        │                FOLLOWER               │
        └───────────────────────────────────────┘
          │ (Times out, starts election)   ▲
          ▼                                │ (Discovers current leader)
        ┌───────────────────────────────────────┐
        │               CANDIDATE               │
        └───────────────────────────────────────┘
          │ (Receives majority votes)      
          ▼                                
        ┌───────────────────────────────────────┐
        │                 LEADER                │
        └───────────────────────────────────────┘

Follower: Completely passive. Responds to incoming RPC calls from Candidates and Leaders. If a follower receives no communication for a randomized timeout period, it assumes the leader has crashed and transitions to a Candidate.
Candidate: Increments the cluster "term", votes for itself, and broadcasts requests for votes to all other nodes.
Leader: Manages all client writes. Coordinates log replication and broadcasts periodic "heartbeat" RPCs to assert authority and reset follower timeout clocks.

Sub-Problem 1: Leader Election

To prevent split votes, followers wait for a randomized election timeout (typically between 150ms and 300ms) before initiating an election.

Once a node's timeout expires, it:

Increments its local currentTerm counter.
Transitions to the Candidate state.
Votes for itself.
Sends a RequestVote RPC to all other nodes.

If a candidate receives votes from a majority of nodes in the cluster (e.g., 3 out of 5), it immediately transitions to Leader and starts broadcasting heartbeats.

Sub-Problem 2: Log Replication

Once a Leader is elected, it acts as the single gateway for all client write requests.

Client ─── Write "X = 5" ───> Leader
                                │
          ┌─────────────────────┴─────────────────────┐
          │ (Append entry, broadcast AppendEntries)   │
          ▼                                           ▼
      Follower A                                  Follower B

The Replication Steps:

The client sends a command (e.g., set x=5) to the Leader.
The Leader appends the command to its local log.
The Leader broadcasts an AppendEntries RPC containing the new log entry to all followers.
Followers verify the entry and append it to their logs, sending a success confirmation back.
Once a majority of followers acknowledge the write, the Leader commits the entry to its state machine and returns a success response to the client.
In subsequent heartbeats, the Leader notifies followers of the newly committed entry, prompting them to commit it to their local state machines.

TypeScript Raft Node Implementation

Here is the baseline state container and interface schema for a Raft consensus node in TypeScript:

type NodeRole = "Follower" | "Candidate" | "Leader";

interface LogEntry {
  term: number;
  command: string;
}

interface RequestVoteArgs {
  term: number;
  candidateId: string;
  lastLogIndex: number;
  lastLogTerm: number;
}

interface RequestVoteReply {
  term: number;
  voteGranted: boolean;
}

class RaftNode {
  public id: string;
  public currentTerm = 0;
  public role: NodeRole = "Follower";
  public votedFor: string | null = null;
  public log: LogEntry[] = [];

  private commitIndex = 0;
  private electionTimeout: NodeJS.Timeout | null = null;
  private peers: string[] = [];

  constructor(id: string, peers: string[]) {
    this.id = id;
    this.peers = peers;
    this.resetElectionTimeout();
  }

  private resetElectionTimeout() {
    if (this.electionTimeout) clearTimeout(this.electionTimeout);

    // Randomized timeout between 150ms and 300ms prevents split votes
    const timeout = 150 + Math.floor(Math.random() * 150);
    this.electionTimeout = setTimeout(() => this.startElection(), timeout);
  }

  // Handle incoming vote request from Candidate node
  public handleRequestVote(args: RequestVoteArgs): RequestVoteReply {
    this.resetElectionTimeout();

    // 1. Term check: reject candidates with stale terms
    if (args.term < this.currentTerm) {
      return { term: this.currentTerm, voteGranted: false };
    }

    if (args.term > this.currentTerm) {
      this.currentTerm = args.term;
      this.role = "Follower";
      this.votedFor = null;
    }

    // 2. Log completeness check: candidate log must be at least as up-to-date as ours
    const lastIndex = this.log.length - 1;
    const lastTerm = lastIndex >= 0 ? this.log[lastIndex].term : 0;
    const logIsUpToDate =
      args.lastLogTerm > lastTerm ||
      (args.lastLogTerm === lastTerm && args.lastLogIndex >= lastIndex);

    // 3. Vote check
    const canVote = this.votedFor === null || this.votedFor === args.candidateId;

    if (canVote && logIsUpToDate) {
      this.votedFor = args.candidateId;
      return { term: this.currentTerm, voteGranted: true };
    }

    return { term: this.currentTerm, voteGranted: false };
  }

  private startElection() {
    this.role = "Candidate";
    this.currentTerm++;
    this.votedFor = this.id;
    this.resetElectionTimeout();

    console.log(`Node ${this.id} starting election for Term ${this.currentTerm}`);

    let votesReceived = 1; // Vote for self
    const lastLogIndex = this.log.length - 1;
    const lastLogTerm = lastLogIndex >= 0 ? this.log[lastLogIndex].term : 0;

    for (const peerId of this.peers) {
      // Simulate dispatching network RPC calls to peer cluster
      this.sendRequestVoteRPC(peerId, {
        term: this.currentTerm,
        candidateId: this.id,
        lastLogIndex,
        lastLogTerm
      }).then((reply) => {
        if (this.role !== "Candidate") return;

        if (reply.voteGranted) {
          votesReceived++;
          if (votesReceived > (this.peers.length + 1) / 2) {
            this.becomeLeader();
          }
        }
      });
    }
  }

  private becomeLeader() {
    this.role = "Leader";
    if (this.electionTimeout) clearTimeout(this.electionTimeout);
    console.log(`Node ${this.id} elected Leader for Term ${this.currentTerm}!`);
    this.startHeartbeats();
  }

  private startHeartbeats() {
    // Send AppendEntries heartbeats at 50ms intervals to maintain authority
    setInterval(() => {
      if (this.role !== "Leader") return;
      for (const peerId of this.peers) {
        this.sendAppendEntriesRPC(peerId);
      }
    }, 50);
  }

  // Mock network transporters for simulation environment
  private async sendRequestVoteRPC(peerId: string, args: RequestVoteArgs): Promise<RequestVoteReply> {
    return { term: this.currentTerm, voteGranted: Math.random() > 0.3 };
  }

  private async sendAppendEntriesRPC(peerId: string) {
    // Heartbeat logic
  }
}

Split-Brain and Minority Partitions

What happens when a network partition cuts a 5-node cluster into two segments: a majority partition (3 nodes) and a minority partition (2 nodes)?

  MAJORITY PARTITION (Can commit)         MINORITY PARTITION (Cannot commit)
   [Node A] ─── [Node B (Leader 1)]         [Node D] ─── [Node E (Leader 2)]
      │                                        │
   [Node C]                                    X (Network Partition cut)

The minority partition might elect a new leader because they can no longer hear from the original leader. However, this minority leader cannot commit any new client writes because they can never achieve the required cluster majority (3 nodes).

Once the partition heals:

The minority leader receives a heartbeat containing a higher term clock from the majority leader.
The minority leader immediately steps down back to a Follower.
The minority nodes discard their uncommitted local log differences and synchronize their states with the majority leader's canonical log.

Raft guarantees that the cluster always converges safely back to a single source of truth.

Engineering Takeaways

Raft guarantees Safety and Liveness in distributed environments with randomized timeouts.
Commitment requires absolute majority consensus — protecting clusters from split-brain state divergence.
Term clocks act as global logical time — higher term numbers override all stale cluster leaders.

The full article features a live 5-node Raft consensus cluster simulator — trigger network cuts to form minority and majority partitions, watch node roles transition, commit client writes, and view live log conflict resolution directly in your browser.

Read the full interactive article →

Written by Ebenezer Akinseinde — Software Developer & AI Automations Engineer.

Portfolio · GitHub

Reconciling P2P Collaborative States via WebRTC Data Channels

Ebendttl — Tue, 26 May 2026 21:18:53 +0000

This is an excerpt. The full article includes a live client-to-client WebRTC simulator — walk step-by-step through a simulated P2P handshake (Offer, STUN check, Signaling relay, Answer, Connection) and watch direct message sync bypass the central server entirely. Read the full interactive version →

The Peer-to-Peer Paradigm

Modern web architectures almost universally rely on client-server protocols (HTTP, WebSockets). While this simplifies state coordination, it introduces overhead: latency, server infrastructure costs, and single-point-of-failure bottlenecks.

WebRTC (Web Real-Time Communication) allows browsers to establish direct peer-to-peer (P2P) channels. Once connected, data flows directly between user devices with sub-10ms latencies, completely bypassing intermediate servers.

While WebRTC is widely known for audio/video streaming, its most powerful asset for application engineering is the RTCDataChannel API — allowing arbitrary, high-throughput, low-latency binary or text transfer directly between browsers.

The P2P Paradox: NATs and Signaling

Two browsers cannot simply talk directly to each other out of the box. Security boundaries and network topologies get in the way. Specifically, browsers face two barriers:

1. The NAT / Firewall Barrier

Almost all consumer devices sit behind Network Address Translators (NATs) and firewalls. A device knows its internal private IP (e.g. 192.168.1.45), but has no idea what its public-facing IP is, nor can it accept unsolicited incoming connections from the open web.

To cross this barrier, WebRTC utilizes ICE (Interactive Connectivity Establishment) alongside two types of auxiliary servers:

STUN (Session Traversal Utilities for NAT): A lightweight server that simply tells a requesting client: "Here is your public-facing IP address and port." This is cheap, fast, and succeeds in ~85% of standard residential network setups.
TURN (Traversal Using Relays around NAT): If symmetric NATs or strict corporate firewalls block direct connection, a TURN server acts as a fallback relay. All traffic is routed through it. This guarantees connection but incurs bandwidth costs.

2. The Signaling Bottleneck

To negotiate a direct connection, peers must exchange configuration details — specifically Session Description Protocol (SDP) objects (containing codecs, encryption keys, and connection profiles) and ICE Candidates (discovered IP/port routing paths).

Since they don't have a direct connection yet, they must use an out-of-band Signaling Server (typically running WebSockets or SSE) to exchange these initial handshakes.

Peer A                     Signaling Server                     Peer B
  │                                │                                │
  │ ─── 1. Send SDP Offer ───────> │ ─────────────────────────────> │
  │                                │                                │
  │ <── 2. Send SDP Answer ─────── │ <───────────────────────────── │
  │                                │                                │
  │ ─── 3. Exchange ICE Candidates │ ─────────────────────────────> │
  │                                │                                │
  │ <── [DIRECT P2P DATA CHANNEL ESTABLISHED (BYPASSES SERVER)] ──> │

Implementing WebRTC Data Channels in TypeScript

Here is a clean, dependency-free implementation of a WebRTC initiator client:

class P2PPeer {
  private pc: RTCPeerConnection;
  private dataChannel: RTCDataChannel | null = null;
  private signalingSocket: WebSocket;

  constructor(signalingUrl: string) {
    this.signalingSocket = new WebSocket(signalingUrl);

    // Configure standard public STUN servers
    this.pc = new RTCPeerConnection({
      iceServers: [
        { urls: "stun:stun.l.google.com:19302" },
        { urls: "stun:stun1.l.google.com:19302" }
      ]
    });

    this.setupIceGathering();
  }

  // Initiator: Create Offer and send to Peer
  async initiateConnection(targetPeerId: string) {
    // 1. Create the data channel on the local connection
    this.dataChannel = this.pc.createDataChannel("p2p-sync-channel", {
      ordered: true // Guarantees in-order delivery of packets
    });
    this.bindChannelEvents(this.dataChannel);

    // 2. Generate SDP Offer
    const offer = await this.pc.createOffer();
    await this.pc.setLocalDescription(offer);

    // 3. Dispatch to signaling server
    this.signalingSocket.send(JSON.stringify({
      type: "offer",
      target: targetPeerId,
      sdp: offer
    }));
  }

  // Receiver: Accept Offer, create Answer
  async handleOffer(offerSdp: RTCSessionDescriptionInit, senderId: string) {
    // 1. Set remote description
    await this.pc.setRemoteDescription(new RTCSessionDescription(offerSdp));

    // 2. Listen for the channel incoming event
    this.pc.ondatachannel = (event) => {
      this.dataChannel = event.channel;
      this.bindChannelEvents(this.dataChannel);
    };

    // 3. Generate SDP Answer
    const answer = await this.pc.createAnswer();
    await this.pc.setLocalDescription(answer);

    // 4. Send back via signaling
    this.signalingSocket.send(JSON.stringify({
      type: "answer",
      target: senderId,
      sdp: answer
    }));
  }

  private setupIceGathering() {
    // Broadcast discovered local ICE routes to remote peer
    this.pc.onicecandidate = (event) => {
      if (event.candidate) {
        this.signalingSocket.send(JSON.stringify({
          type: "ice-candidate",
          candidate: event.candidate
        }));
      }
    };
  }

  private bindChannelEvents(channel: RTCDataChannel) {
    channel.onopen = () => console.log("Direct P2P Data Channel Opened!");
    channel.onmessage = (event) => {
      const data = JSON.parse(event.data);
      console.log("Received P2P Message:", data);
    };
    channel.onclose = () => console.log("Data Channel Closed.");
  }

  send(message: any) {
    if (this.dataChannel && this.dataChannel.readyState === "open") {
      this.dataChannel.send(JSON.stringify(message));
    }
  }
}

State Reconciliations over P2P

Direct connections introduce concurrency challenges. If both clients edit a shared local state concurrently, how do they sync without a server?

The standard solution is pairing RTCDataChannel with a CRDT (Conflict-free Replicated Data Type) engine (like Y.js or Automerge). The data channel acts as a raw transport layer, while the CRDT handles deterministic state updates.

Because the data channel supports binary buffers (arraybuffer), you can transmit highly compact, compressed state updates directly:

channel.send(Y.encodeStateAsUpdate(ydoc));

Engineering Takeaways

WebRTC enables direct P2P connection, dropping transport latency to the raw physical limits of the network.
Signaling servers are only required for connection setup (SDP/ICE exchange) — once connected, the signaling server can crash and the P2P connection remains active.
STUN servers are free and easy to host, but always configure TURN servers as relays for symmetric or enterprise networks.
RTCDataChannel handles ordered or unordered data transmission, making it highly customizable for games, file sharing, or collaborative documents.

The full article features a step-by-step WebRTC handshake visualizer — walk through the signaling architecture with a live interactive simulation of Offer generation, STUN query routing, and P2P connection establishment.

Read the full interactive article →

Written by Ebenezer Akinseinde — Software Developer & AI Automations Engineer.

Portfolio · GitHub

Building a Vector Search Engine from Scratch: The Math and Mechanics of HNSW

Ebendttl — Tue, 26 May 2026 21:09:07 +0000

This is an excerpt. The full article includes a live 2D vector space sandbox — drag a query vector across a semantic coordinate field and watch cosine similarity scores update in real time across 9 framework nodes, with a toggle between Brute Force KNN and HNSW skip-graph traversal modes. Read the full interactive version →

The Geometry of Semantics

In the era of large language models, unstructured text is treated not as raw characters, but as a vector coordinate in a high-dimensional mathematical space.

When you pass text into an embedding model (like Google's text-embedding-004), it maps the semantic meaning into an array of floating-point numbers:

"Redis In-Memory Cache" → [ 0.01249, -0.04581, 0.08914, ..., -0.00312 ]  // 3072 dimensions
"PostgreSQL Database"   → [ 0.01183, -0.04491, 0.08877, ..., -0.00298 ]  // 3072 dimensions
"PyTorch Model"         → [-0.09214,  0.07321, -0.03812, ...,  0.06441 ]  // 3072 dimensions

Within this latent space, conceptually similar texts are geometrically close. "Redis" and "PostgreSQL" cluster near each other (both databases). "PyTorch" is far away (AI/ML domain).

Semantic search = finding the K-nearest neighbor coordinates to a query vector.

The Mathematical Distance Metrics

Three metrics dominate vector similarity calculations:

1. Cosine Similarity (most common for NLP)

cos(θ) = (A · B) / (|A| × |B|)

Measures the angular projection between two vectors. Range: -1 to 1. Completely ignores magnitude — two vectors pointing in the same direction score 1.0 even if one is 100× longer. This is ideal for text embeddings where magnitude carries no semantic information.

2. Dot Product (fastest for normalized vectors)

A · B = Σ(Aᵢ × Bᵢ)

When vectors are L2-normalized (unit length = 1), dot product yields identical rankings to cosine similarity with significantly less computation. Normalize your embeddings on ingestion, then use dot product at query time.

3. L2 Euclidean Distance (for spatial data)

d(A, B) = √(Σ(Aᵢ - Bᵢ)²)

Measures raw physical distance between vector tips. Highly sensitive to magnitude variations. Preferred for coordinate-based spatial systems, not raw text embeddings.

The Scaling Bottleneck: Why Brute Force Fails

In naive Brute Force KNN, you compare the query vector against every document in your database:

Complexity: O(D × N)  — where D = dimensions, N = document count

At scale, brute force saturates CPU and destroys real-time response requirements. You need Approximate Nearest Neighbor (ANN) indexing.

HNSW: Hierarchical Navigable Small World

HNSW translates the Skip List data structure into a multi-layered graph:

LAYER 2 (Sparse Highway) ─── node_A ──────────────────── node_B ───
LAYER 1 (Intermediate)   ─── node_A ─── node_C ─── node_B ─── node_D
LAYER 0 (Dense Ground)   ─ ALL nodes fully connected to neighbors ──

This reduces search complexity from:

Brute Force: O(N)         → Linear — fails at scale
HNSW:        O(log N)     → Logarithmic — sub-5ms at 10M+ docs

TypeScript HNSW Implementation

class HNSWIndex {
  private layers: Map<string, string[]>[] = [];
  private vectors: Map<string, number[]> = new Map();
  private readonly maxLayers: number;
  private readonly maxConnections: number;

  constructor(maxLayers = 4, maxConnections = 16) {
    this.maxLayers = maxLayers;
    this.maxConnections = maxConnections;
    this.layers = Array.from({ length: maxLayers }, () => new Map());
  }

  private cosineSimilarity(a: number[], b: number[]): number {
    const dot = a.reduce((sum, ai, i) => sum + ai * b[i], 0);
    const magA = Math.sqrt(a.reduce((sum, ai) => sum + ai * ai, 0));
    const magB = Math.sqrt(b.reduce((sum, bi) => sum + bi * bi, 0));
    return magA === 0 || magB === 0 ? 0 : dot / (magA * magB);
  }

  private getRandomLayer(): number {
    let layer = 0;
    while (layer < this.maxLayers - 1 && Math.random() < 0.5) {
      layer++;
    }
    return layer;
  }

  insert(id: string, vector: number[]): void {
    this.vectors.set(id, vector);
    const assignedLayer = this.getRandomLayer();

    for (let level = 0; level <= assignedLayer; level++) {
      if (!this.layers[level].has(id)) {
        const candidates = this.searchLayer(vector, level, this.maxConnections);
        this.layers[level].set(id, candidates.map(c => c.id));
      }
    }
  }

  search(queryVector: number[], k: number): { id: string; score: number }[] {
    let candidates = this.getEntryPoints(queryVector);

    for (let level = this.maxLayers - 1; level >= 0; level--) {
      candidates = this.searchLayer(queryVector, level, k, candidates);
    }

    return candidates
      .sort((a, b) => b.score - a.score)
      .slice(0, k);
  }

  private searchLayer(
    query: number[],
    level: number,
    k: number,
    entryPoints: { id: string; score: number }[] = []
  ): { id: string; score: number }[] {
    const visited = new Set<string>();
    const results: { id: string; score: number }[] = [...entryPoints];
    const queue = [...entryPoints];

    while (queue.length > 0) {
      const current = queue.shift()!;
      if (visited.has(current.id)) continue;
      visited.add(current.id);

      const neighbors = this.layers[level].get(current.id) ?? [];
      for (const neighborId of neighbors) {
        if (!visited.has(neighborId)) {
          const vec = this.vectors.get(neighborId)!;
          const score = this.cosineSimilarity(query, vec);
          queue.push({ id: neighborId, score });
          results.push({ id: neighborId, score });
        }
      }
    }

    return results
      .sort((a, b) => b.score - a.score)
      .slice(0, k);
  }

  private getEntryPoints(query: number[]): { id: string; score: number }[] {
    const entries: { id: string; score: number }[] = [];
    for (const [id, vec] of this.vectors) {
      entries.push({ id, score: this.cosineSimilarity(query, vec) });
      if (entries.length >= 3) break;
    }
    return entries;
  }
}

Production Scaling

Memory-Mapped Files (mmap): Keeping large indices in RAM causes OOM crashes. Bind binary index files directly to virtual address space — the OS swaps vector blocks in/out of disk cache automatically based on query patterns.

Product Quantization (PQ): Divide each high-dimensional vector into sub-vectors and map them to centroids. This reduces memory footprint by up to 95% with minimal recall degradation.

Engineering Takeaways

Always L2-normalize embeddings on ingestion to enable fast dot product scoring instead of full cosine calculation.
HNSW reduces O(N) to O(log N) — the single most impactful optimization for vector search at scale.
Use mmap + Product Quantization in production to handle tens of millions of vectors without OOM.

The full article includes a drag-and-drop 2D semantic space sandbox — 9 framework nodes plotted across a latent coordinate field, real-time cosine similarity leaderboard, toggle between Brute Force and HNSW skip-graph modes, and live layer traversal visualization.

Read the full interactive article →

Written by Ebenezer Akinseinde — Software Developer & AI Automations Engineer.

Portfolio · GitHub

Deep Dive into Y.js CRDTs for Real-Time Multiplayer Editors

Ebendttl — Tue, 26 May 2026 21:02:29 +0000

This is an excerpt. The full article includes a live YATA double-linked list simulator — type into two peer editors, toggle network partitions to simulate offline-first edits, and watch Y.js automatically resolve merge conflicts in real time. Read the full interactive version →

The Local-First Wave: Why OT Is Not Enough

Building real-time collaborative software used to require a centralized server orchestrator. For decades, the industry standard was Operational Transformation (OT) — the architecture powering Google Docs.

In OT, every local edit is wrapped into an operation and sent to a central server. The server acts as the single source of truth, recalculating operation coordinates to resolve conflicts and broadcasting adjusted commands back.

The OT Bottleneck: The server must be active 100% of the time, maintaining a complex sequence history buffer. The moment a client drops offline, peer-to-peer sync becomes nearly impossible to reconcile, leading to silent document divergence.

CRDTs (Conflict-free Replicated Data Types) rethink this entirely. Instead of relying on a server to decide conflict priority, CRDT structures are mathematically designed to merge operations in any order without coordination, guaranteeing all clients converge to identical state.

The Mathematical Foundation: Join-Semilattices

To achieve absolute convergence across decentralized peers, a data structure must form a join-semilattice. In practice, your merge functions must conform to three algebraic invariants:

Property	Formula	Meaning
Commutativity	`A ⊕ B = B ⊕ A`	Operations merge in any sequence order
Associativity	`(A ⊕ B) ⊕ C = A ⊕ (B ⊕ C)`	Grouping doesn't affect the result
Idempotency	`A ⊕ A = A`	Receiving the same operation twice doesn't mutate state

These three properties together mean: you can merge any subset of operations, in any order, any number of times, and always reach the same final document. No coordination server required.

The YATA Engine Under the Hood

Y.js implements YATA (Yet Another Transformation Algorithm) — a highly optimized CRDT designed specifically for text editing.

Under the hood, Y.js represents a document not as a raw string array, but as a double-linked list of Item blocks. Each item contains:

id: A unique tuple of (clientId, lamportClock) — globally unique across all peers
content: The character or string chunk
originLeft: The ID of the item to the immediate left at creation time
originRight: The ID of the item to the immediate right at creation time
deleted: Boolean tombstone flag

These origin pointers act as a permanent spatial anchor. Even if other users concurrently insert characters between the original boundaries, YATA can deterministically sort concurrent insertions using client ID priority — ensuring all replicas converge to identical ordering.

Conflict Resolution Example

Suppose two users type simultaneously at position 5:

Peer A inserts "X" with originLeft = "SYS:4"
Peer B inserts "Y" with originLeft = "SYS:4"

Both have the same originLeft. YATA resolves this by comparing client IDs alphanumerically. If "A" < "B", Peer A's item is sorted before Peer B's — identically on both replicas, regardless of which update arrived first.

Connecting Monaco Editor

Binding Y.js to Monaco Editor is done via the y-monaco package:

import * as Y from "yjs";
import { WebsocketProvider } from "y-websocket";
import { MonacoBinding } from "y-monaco";
import * as monaco from "monaco-editor";

// 1. Initialize Y.js document
const ydoc = new Y.Doc();

// 2. Connect to WebSocket relay server
const provider = new WebsocketProvider(
  "wss://your-relay-server.com",
  "document-room-id",
  ydoc
);

// 3. Create shared Y.Text type
const ytext = ydoc.getText("monaco-content");

// 4. Bind to Monaco Editor instance
const editor = monaco.editor.create(document.getElementById("editor")!, {
  value: "",
  language: "typescript"
});

// MonacoBinding syncs ytext ↔ Monaco model bidirectionally
// It also handles cursor decorations and multi-user awareness
const binding = new MonacoBinding(
  ytext,
  editor.getModel()!,
  new Set([editor]),
  provider.awareness
);

The MonacoBinding synchronizes Y.js document state with Monaco's internal text model bidirectionally — handling cursor decorations, selection highlights, and presence indicators automatically.

Production Relay Infrastructure

A single Y.js WebSocket server can't scale horizontally — each instance maintains its own in-memory document state. To scale, use Redis Pub/Sub as a shared synchronization backbone:

import { createServer } from "http";
import { WebSocketServer } from "ws";
import { setupWSConnection } from "y-websocket/bin/utils";
import { createClient } from "redis";

const redisPublisher = createClient({ url: process.env.REDIS_URL });
const redisSubscriber = redisPublisher.duplicate();

await redisPublisher.connect();
await redisSubscriber.connect();

const wss = new WebSocketServer({ server: createServer() });

wss.on("connection", (ws, req) => {
  // y-websocket handles CRDT sync protocol
  setupWSConnection(ws, req, {
    // Persistence callback — hook into Redis or PostgreSQL here
    persistence: {
      bindState: async (docName, ydoc) => {
        // Load existing document state from Redis
        const state = await redisPublisher.get(`ydoc:${docName}`);
        if (state) Y.applyUpdate(ydoc, Buffer.from(state, "base64"));
      },
      writeState: async (docName, ydoc) => {
        // Persist document state on every update
        const state = Y.encodeStateAsUpdate(ydoc);
        await redisPublisher.set(`ydoc:${docName}`, Buffer.from(state).toString("base64"));
      }
    }
  });
});

With Redis Pub/Sub, any number of WebSocket server instances can participate in the same document room. An update arriving at Server A propagates to Redis, which fans it out to Servers B and C, ensuring all connected peers — regardless of which server they're on — stay synchronized.

Key Optimizations for Production

Awareness Protocol: Y.js includes a built-in awareness system for sharing ephemeral state (cursor positions, user names, colors) without persisting to the CRDT document. Use this for presence, not your main Y.Doc.
State Vector Sync: On reconnect, clients exchange state vectors (Y.encodeStateVector) to efficiently compute only the missing updates — avoiding full document retransmission.
Subdocument Architecture: For large applications, split content across multiple Y.Doc instances (one per section/page). Load subdocuments lazily when accessed.

Engineering Takeaways

OT requires centralized servers; CRDTs enable true local-first. Offline edits automatically merge on reconnect.
YATA's origin pointers guarantee deterministic ordering without server coordination.
y-websocket + y-monaco is the fastest path to production collaborative editing.
Redis Pub/Sub enables horizontal scaling of Y.js WebSocket relay servers.
State vectors make reconnection efficient — only delta updates are exchanged.

The full article includes a live YATA double-linked list simulator. You can type into both peer editors simultaneously, toggle each peer's network state to create a partition, make offline edits, then click "Resolve & Sync Partition" to watch the CRDT merge algorithm reconcile divergent histories in real time — with the underlying Item linked-list visualized beneath.

Read the full interactive article →

Written by Ebenezer Akinseinde — Software Developer & AI Automations Engineer.

Portfolio · GitHub

Mastering Structured JSON Outputs with Gemini API

Ebendttl — Tue, 26 May 2026 19:47:23 +0000

This is an excerpt. The full article includes a live interactive schema sandbox where you can switch between 3 real constraint schemas and watch the Gemini inference engine stream constrained tokens in real time. Read the full interactive version →

The Problem: LLMs Are Eloquent, Not Predictable

Language models are optimized to be helpful communicators. This is precisely what makes them powerful interfaces for humans — and extraordinarily fragile integrations for software architectures.

Consider a simple extraction request:

"Extract the product name, price, and availability from the following text and return it as JSON."

Under testing, the model returns a clean JSON block. But in high-throughput production environments, you'll inevitably hit the model's alignment behaviors:

Conversational Padding: "Here is the data you requested: ..."
Varying Key Names: One response returns "product_name", another "product", a third "name"
Brittle Typings: A numeric price 279.99 becomes the raw string "$279.99"

Your downstream TypeScript classes throw unhandled KeyError exceptions. The execution fails.

Why Regex and Prompt Engineering Will Betray You

The classic fix is prompt escalation:

"Return ONLY a raw JSON object. Do NOT wrap in markdown. NEVER write conversational text."

This reduces failures under small loads — but instruction-following is entirely probabilistic. Under unexpected long-context inputs, the model drifts back to its conversational baseline. In a system handling 50,000 calls/day, a 1% failure rate represents 500 critical errors.

Custom regex parsing is worse. The moment the provider updates their model parameters, your regex silently corrupts production data.

Constrained Decoding: Enforcing Structure at the Inference Layer

Gemini's structured output system works via vocabulary masking during the inference step itself — not post-processing.

When generating a response, the model predicts the probability of every token in its ~32,000+ word vocabulary. Without constraints, it samples freely. When you enforce a JSON Schema contract, Gemini compiles it into a state machine. At every generation step, illegal tokens are masked to exactly zero probability.

If a field expects a number, every text token ("twenty", "$", any alphabet character) is mathematically eliminated. This is not retrying or filtering — it's structural constraint at the neural network's decoding loop.

Standard Decoding	Constrained Decoding (Gemini)
`"$279.99"` → 45% probability	`"$279.99"` → 0% probability
`"279.99"` → 40% probability	`"279.99"` → 100% probability
`"in stock"` → 15% probability	`"in stock"` → 0% probability

The Two API Pillars

Activate structured execution with two native parameters:

import { GoogleGenerativeAI, SchemaType } from "@google/generative-ai";

const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY!);

const model = genAI.getGenerativeModel({
  model: "gemini-2.0-flash",
  generationConfig: {
    responseMimeType: "application/json",  // Pillar 1
    responseSchema: {                       // Pillar 2
      type: SchemaType.OBJECT,
      properties: {
        sentiment: {
          type: SchemaType.STRING,
          enum: ["VERY_POSITIVE", "POSITIVE", "NEUTRAL", "NEGATIVE", "VERY_NEGATIVE"]
        },
        csat_risk_score: {
          type: SchemaType.NUMBER,
          description: "0=no risk, 10=certain churn"
        },
        requires_human: { type: SchemaType.BOOLEAN }
      },
      required: ["sentiment", "csat_risk_score", "requires_human"]
    }
  }
});

responseMimeType: "application/json" switches the model from raw string processing to structured mode. responseSchema defines the structural contract the response must satisfy — keys, types, enums, required fields, all of it.

JSON Schema Deep Dive

Enums — The Most Powerful Constraint

Enums force Gemini to select from a hardcoded array of values. This is the single most impactful constraint for classification systems:

{
  "type": "string",
  "enum": ["IN_STOCK", "OUT_OF_STOCK", "BACKORDER"]
}

No hallucinated variants. No "in stock" vs "In Stock" inconsistencies. The schema enforces it at the token level.

Nullable Attributes

{ "type": "string", "nullable": true }

This prevents hallucinated values. If the input text contains no reference to that field, Gemini outputs null rather than inventing data.

The Multi-Stage Orchestration Pattern

For complex documents, never attempt a single massive extraction call. Instead, decompose into modular pipelines:

Raw Document
    ↓
Stage 1: Classification (Schema: DocType)
    ↓
Stage 2A: Invoice Parser  |  Stage 2B: Legal Contract  |  Stage 2C: Receipt Parser
    ↓                              ↓                              ↓
                     Unified Structured Database

Each stage uses a narrow, optimized schema. This reduces cost, increases accuracy, and makes debugging trivial.

Production Validation Layer

Schema enforcement guarantees structural correctness — not logical correctness. Always include downstream validation:

import { z } from "zod";

const SentimentSchema = z.object({
  sentiment: z.enum(["VERY_POSITIVE", "POSITIVE", "NEUTRAL", "NEGATIVE", "VERY_NEGATIVE"]),
  csat_risk_score: z.number().min(0).max(10),
  requires_human: z.boolean()
});

const raw = await model.generateContent(prompt);
const parsed = JSON.parse(raw.response.text());
const validated = SentimentSchema.safeParse(parsed);

if (!validated.success) {
  // Handle structural edge cases gracefully
  console.error("Validation failed:", validated.error);
}

Gemini guarantees output keys exist and types match. It cannot know if a discount value is negative or if invoice line items don't sum to the stated total. Always validate semantic parameters downstream.

Engineering Takeaways

Never rely on instruction-following alone. Probabilistic models will drift. Use structural constraints at the API level.
responseMimeType + responseSchema is the only production-safe pattern for JSON extraction pipelines.
Enums are your most powerful tool — they eliminate entire classes of inconsistency bugs.
Constrained decoding ≠ logical validation. Layer Zod or Pydantic downstream.
Multi-stage pipelines outperform single massive calls for complex document structures.

🔬 The full article includes an interactive Gemini Constraint Engine sandbox — select from 3 real schema contracts (Sentiment Tracker, Invoice Parser, Code Auditor) and watch constrained token streaming in real time. It also covers complex nested schemas, entity extraction patterns, cost/latency optimization, and the future of agentic orchestration.

Read the full interactive article →

Written by Ebenezer Akinseinde — Software Developer & AI Automations Engineer. Building fast, production-grade AI pipelines and distributed frontend systems.

Portfolio · GitHub