The latest articles on DEV Community by yash (@yashvardhanreddy_pamiredd). https://dev.to/yashvardhanreddy_pamiredd https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4114597%2F50370562-2b9d-4ac9-a03e-45a4fcbbc990.png https://dev.to/yashvardhanreddy_pamiredd en yash Mon, 07 Sep 2026 21:09:59 +0000 https://dev.to/yashvardhanreddy_pamiredd/building-an-append-only-storage-engine-from-scratch-in-typescript-1ip8 https://dev.to/yashvardhanreddy_pamiredd/building-an-append-only-storage-engine-from-scratch-in-typescript-1ip8 <h1> CargoDB ⚡ </h1> <p>A lightweight, append-only key-value storage engine and binary blob store built with in-memory offset indexing, isolated bucket partitioning, and automated zero-downtime log compaction.</p> <p><strong>GitHub Repository:</strong> <a href="https://github.com/yash-pamireddy/cargo-engine" rel="noopener noreferrer">https://github.com/yash-pamireddy/cargo-engine</a></p> <h3> The Core Concept </h3> <p>Most applications default to running heavy database daemons or splitting workloads across multiple cloud vendors—using an external database for JSON records and S3 for media uploads. </p> <p>CargoDB explores a minimal, high-throughput alternative inspired by log-structured architectures like <strong>Bitcask</strong>:</p> <ul> <li> <strong>Sequential Disk Writes:</strong> Incoming writes and deletions are strictly appended to the end of the log file. This eliminates random disk write overhead and fragmentation.</li> <li> <strong>O(1) In-Memory Index:</strong> Keys are mapped to byte offsets and record lengths directly in RAM, allowing reads to jump straight to the exact position on disk with a single seek.</li> <li> <strong>Zero-Downtime Compaction:</strong> Overwrites and tombstoned deletions are cleaned up during background compaction passes without taking the engine offline.</li> <li> <strong>Unified Blobs &amp; Records:</strong> Store raw media (images, PDFs, archives) alongside structured JSON payloads inside isolated tenant buckets.</li> </ul> <h3> Architecture Flow </h3> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> text [ Put / Set / Upload / Delete ] │ ▼ ┌─────────────────────────┐ │ Append-Only Disk Log │ ──&gt; Fast sequential writes └─────────────────────────┘ │ ▼ ┌─────────────────────────┐ │ In-Memory Offset Table │ ──&gt; Maps Key ➔ { offset, length, mime } └─────────────────────────┘ │ ▼ [ Reads: O(1) Seek ] </code></pre> </div> typescript node database opensource