Building an Append-Only Storage Engine from Scratch in TypeScript

CargoDB ⚡

A lightweight, append-only key-value storage engine and binary blob store built with in-memory offset indexing, isolated bucket partitioning, and automated zero-downtime log compaction.

GitHub Repository: https://github.com/yash-pamireddy/cargo-engine

---

The Core Concept

Most applications default to running heavy database daemons or splitting workloads across multiple cloud vendors—using an external database for JSON records and S3 for media uploads.

CargoDB explores a minimal, high-throughput alternative inspired by log-structured architectures like Bitcask:

• Sequential Disk Writes: Incoming writes and deletions are strictly appended to the end of the log file. This eliminates random disk write overhead and fragmentation.
• O(1) In-Memory Index: Keys are mapped to byte offsets and record lengths directly in RAM, allowing reads to jump straight to the exact position on disk with a single seek.
• Zero-Downtime Compaction: Overwrites and tombstoned deletions are cleaned up during background compaction passes without taking the engine offline.
• Unified Blobs & Records: Store raw media (images, PDFs, archives) alongside structured JSON payloads inside isolated tenant buckets.

---

Architecture Flow

text
               [ Put / Set / Upload / Delete ]
                              │
                              ▼
                 ┌─────────────────────────┐
                 │  Append-Only Disk Log   │ ──> Fast sequential writes
                 └─────────────────────────┘
                              │
                              ▼
                 ┌─────────────────────────┐
                 │ In-Memory Offset Table  │ ──> Maps Key ➔ { offset, length, mime }
                 └─────────────────────────┘
                              │
                              ▼
                      [ Reads: O(1) Seek ]