Kareem

Posted on May 27

How We Built a 3-Tier Node.js Cache That Hits 2.8M ops/sec (And Never Drops a Key on Eviction)

#node #typescript #performance #backend

Caching in Node.js usually forces a frustrating compromise. You either stick with a basic in-memory Map (or LRU-cache) that risks running your process out of memory (OOM) under heavy loads, or you accept the network latency floor and serialization overhead of a standalone Redis/Valkey instance.

When building high-throughput systems, neither option feels entirely right. A localhost Redis round-trip is fast, but it’s still bounded by networking stacks.

We built tricache to eliminate this compromise completely. It’s an open-source, three-tier caching engine for Node.js designed to maximize raw single-threaded throughput while implementing protective guardrails like automated disk spilling, thundering-herd prevention, and an integrated WebAssembly Bloom filter.

The result? 2.81 million read operations per second from a single thread—over 100× faster than a local Redis round-trip—without the risk of unbounded RAM growth.

Here is a look under the hood at how it works and the architectural decisions that made it happen.

The 3-Tier Architecture

To achieve high hit rates without risking OOM crashes, tricache splits data management into three distinct layers:

L1 (Smart Memory Cache): A blazing-fast, process-level memory layer built on top of a highly optimized V8 Map.
L1.5 (Local Disk Spill): When the L1 memory cap (l1MaxBytes) or entry cap is reached, evicted entries aren't simply deleted. They spill over to a local, high-speed NVMe disk directory using optimized msgpackr binary serialization.
L2 (Distributed Backplane): The shared remote tier (Redis or Valkey) used for multi-instance distributed sync.

Core Technical Features & Innovations

1. Advanced Hybrid Eviction via Count-Min Sketch

Standard LRU (Least Recently Used) caching fails catastrophically during sudden scan floods or data bursts, where long-resident, high-value keys get wiped out by single-access keys.

tricache implements a hybrid eviction algorithm that scores keys based on a combination of Priority Score + Access Frequency + Remaining TTL. To track access frequency without introducing a massive memory footprint, we integrated a Count-Min Sketch using a tiny 4 KB Uint16Array (4 rows × 512 columns).

Every time a key is queried or set, it passes through the sketch. When L1 needs to evict data, it uses reservoir sampling to pick an eviction candidate in a single $O(1)$ pass, ensuring a 78% survival rate for high-frequency keys during benchmark flood tests.

2. Inlined WASM Bloom Filter

Before hitting a synchronous Map lookup or checking the filesystem/Redis on a cache miss, queries pass through an ultra-lightweight WASM Bloom filter.

The filter is optimized with $k=7$ hash probes. At a rated capacity of roughly 18,000 entries, it maintains a false-positive rate of just 1%. If the filter says a key does not exist, it is a guaranteed miss, allowing tricache to skip heavy lookups entirely. The 562-byte WASM binary is fully inlined as a Base64 string—meaning zero filesystem access overhead at boot time.

3. Native Thundering-Herd Prevention

When a highly contested key expires under heavy traffic, hundreds of concurrent requests usually hit the underlying database at the exact same moment.

tricache avoids this with a native Inflight Promise Registry. No matter how many concurrent callers request an expired or missing key, the user-supplied fetchFn fires exactly once. All other concurrent requests hook into that single pending promise, protecting your database or upstream APIs from collapsing under sudden load.

4. Enterprise-Grade Resiliency Out of the Box

Stale-While-Revalidate (SWR) & Stale-If-Error: Instantly serves stale data to the client while revalidating the data asynchronously in the background. If the upstream database fails during revalidation, staleIfError kicks in to extend the stale entry's life, keeping your app fully operational during upstream outages.
OOM Guard: A background timer polls heapUsed and heapTotal. If memory utilization breaches a set threshold (e.g., 85%), an emergency eviction routine triggers immediately, clearing out the coldest 20% of L1 memory before the Node.js process crashes.
At-Rest Encryption: Supports automated AES-256-GCM authenticated encryption for L2 values, disk spill files, and cold-start snapshots, complete with zero-downtime key rotation support.

Performance Benchmarks

All metrics were captured executing on a single Node.js thread (with synchronous paths completely un-awaited):

L1 Memory Cache Throughput

Operation	Throughput	Latency	Mechanics
`get` (Hot Hit)	2.81 M/s	~356 ns	Bloom filter → Map lookup → Return
`get` (Cold Miss)	7.14 M/s	~140 ns	Bloom filter gates early return
`delete` (Exact Key)	5.36 M/s	~186 ns	Synchronous Map deletion

End-to-End Service Performance

Operation	Throughput	Latency
`get` (L1 Warm Hit)	2.03 M/s	~491 ns
`get` (SWR Stale Serve)	1.78 M/s	~562 ns
`get` (Miss + `fetchFn`)	13.7 K/s	~73 µs

Quick Start: Getting Started in 30 Seconds

Getting started requires zero initial configuration. Sensible production defaults apply automatically.

npm install tricache
# or pnpm add tricache

import { CacheService } from 'tricache';

// Initialize the process-level singleton
const cache = CacheService.create({
  namespace: 'my-app',
  redisHost: process.env.REDIS_HOST, // Omit in dev to automatically fallback to L1/Disk only
});

// Fetch-or-Get with an automatic database fallback and a 5-minute TTL
const user = await cache.get(
  `user:${userId}`,
  () => db.users.findById(userId),
  300, // TTL in seconds
  { swr: 30 } // Seamless Stale-While-Revalidate for 30 seconds
);

Advanced Group Invalidation with Tags

You can tag items on creation and invalidate entire collections across L1, local disk, and distributed Redis nodes simultaneously:

// Add items to the cache with descriptive tags
await cache.set(`product:1`, productData, 300, undefined, { tags: ['catalog'] });
await cache.set(`product:2`, alternativeData, 300, undefined, { tags: ['catalog'] });

// Atomically evict all catalog entries everywhere
await cache.invalidateTag('catalog');

Open Source & Contributing

tricache is fully open-source, licensed under the MIT license, and built entirely using TypeScript. If you are building high-performance Node.js services or want to dive into the codebase (especially src/smart-memory-cache.ts), check out the repository!

👉 Check out the repo on GitHub: github.com/Kareem411/TriCache

Feedback, bug reports, and performance optimizations via Pull Requests are always welcome! Let me know what caching bottlenecks you're currently facing in your architecture below.

DEV Community