DEV Community

Timevolt
Timevolt

Posted on

Caching Like a Jedi: Strategies Every Backend Developer Must Understand

#systemdesign #architecture #backend #programming

The Quest Begins (The "Why")

I still remember the night our API started buckling under a traffic spike that looked innocent on the dashboard. Users were seeing 502 errors, the support Slack channel lit up like a Christmas tree, and I felt like I was trying to bail out a sinking ship with a teaspoon. We had a monolith that talked directly to the database for every request—reads, writes, even the tiny health‑check pings. The DB was screaming, and our latency graph looked like a roller coaster designed by a sadist.

That’s when I realized we needed a cache not just as an after‑thought, but as a core architectural piece. The question wasn’t “should we cache?” but “how do we cache correctly without turning our system into a house of cards?”

The Revelation (The Insight)

The breakthrough came when I stopped thinking of caching as a simple key‑value store and started seeing it as a state‑machine that sits between the caller and the source of truth. The critical insight: the cache must own the decision of when data is stale, not the caller.

If the caller decides “I’ll ask the DB if the cache is empty,” you inevitably get a cache stampede (also known as the thundering herd problem). Every missed request hits the DB at the same moment, turning a small miss into a massive surge.

Instead, we let the cache itself be responsible for refreshing data. The pattern is called read‑through (or lazy‑loading) combined with a background refresh (sometimes called write‑behind). Here’s why it beats the naive “check‑then‑set” approach:

Approach Stampede risk Complexity Consistency
Check‑then‑set (caller decides) High – many clients miss at once Low Eventually consistent, but can serve stale data for a window
Read‑through + background refresh Low – only one cache node triggers reload Medium – needs a little coordination Strong – cache always holds a fresh copy after the first miss

Think of it like the scene in The Matrix where Neo realizes “there is no spoon.” The spoon (the DB) isn’t the thing you should be bending; you should be bending your perception of where the data lives. The cache becomes the spoon, and you stop trying to fight the DB directly.

Wielding the Power (Code & Examples)

Below is a compact Node.js example that implements a read‑through cache with a background refresher using Redis as the store. The key trick is to use a Redis Lua script that atomically increments a miss counter and, if it’s the first miss, triggers a reload. This prevents the herd from stampeding.

The “before” – naive check‑then‑set 

// ❌ DON'T DO THIS: prone to stampede
async function getUserNaive(userId) {
  const cached = await redis.get(`user:${userId}`);
  if (cached) return JSON.parse(cached);

  const fresh = await db.query('SELECT * FROM users WHERE id = $1', [userId]);
  await redis.setex(`user:${userId}`, 300, JSON.stringify(fresh.rows[0])); // 5‑min TTL
  return fresh.rows[0];
}
Enter fullscreen mode Exit fullscreen mode

When traffic spikes, hundreds of concurrent requests all see an empty cache, hammer the DB, and then all try to set the same key.

The “after” – read‑through with atomic miss handling 

// Lua script: returns the cached value if present.
// On miss, increments a "miss" counter; if the counter becomes 1,
// we signal the caller to reload (by returning null).
const luaScript = `
  local val = redis.call('GET', KEYS[1])
  if val then
    return {1, val}   -- hit
  else
    local miss = redis.call('INCR', KEYS[1] .. ':miss')
    if miss == 1 then
      return {0, nil} -- first miss: caller should reload
    else
      return {0, nil} -- subsequent miss: wait for reload
    end
  end
`;

async function getUser(userId) {
  const key = `user:${userId}`;
  const result = await redis.eval(luaScript, 1, key);
  const [status, value] = result;

  if (status === 1) {               // cache hit
    return JSON.parse(value);
  }

  // cache miss – only the first caller does the reload
  if (status === 0 && value === null) {
    const fresh = await db.query('SELECT * FROM users WHERE id = $1', [userId]);
    await redis.setex(key, 300, JSON.stringify(fresh.rows[0]));
    // reset miss counter so next wave sees a hit
    await redis.del(`${key}:miss`);
    return fresh.rows[0];
  }

  // we raced another reload; spin‑wait briefly then retry
  await new Promise(res => setTimeout(res, 10)); // 10ms backoff
  return getUser(userId); // tail‑recursive, but in practice you'd loop
}
Enter fullscreen mode Exit fullscreen mode

What’s happening?

  1. Atomic check – the Lua script runs inside Redis, guaranteeing that no two clients can both see “miss” and both increment the counter to 1.
  2. First miss wins – only the first caller proceeds to fetch from the DB and repopulate the cache.
  3. Subsequent misses – they either wait a tiny bit and retry, or (in a production version) you could have them block on a Redis Pub/Sub channel until the refresher publishes a “ready” signal.

Trade‑offs

  • Network round‑trip: we still hit Redis on every request, but Redis is in‑memory and far faster than a DB round‑trip.
  • Complexity: a little more code, but the gain in throughput under load is massive.
  • Memory usage: we keep a TTL; stale entries are evicted automatically.

If you prefer a library, ioredis has built‑in getOrSet with a lock option that does essentially the same thing—just remember to scope the lock to the key, not globally.

Why This New Power Matters

Adopting this pattern transformed our service from a fragile, DB‑bound system to one that could sustain 10× the traffic with the same hardware. Latency dropped from ~250 ms to ~30 ms for cached reads, and the DB’s CPU usage fell from a constant 80 % to idle spikes only during writes.

Beyond the numbers, the mental shift was huge: we stopped treating the cache as a “maybe‑useful” side‑car and started seeing it as the primary source of truth for reads, with the DB acting as the durable write‑ahead log. This mindset opens doors to other patterns—write‑behind, multi‑level caches (local LRU + Redis), even geo‑replicated read replicas backed by the same cache layer.

You’ll also find debugging easier. When a request returns stale data, you know exactly where to look: the TTL or the background refresher—not a mysterious “caller decided to skip the cache.”

Your Turn

Here’s a challenge: take a simple endpoint in your own project that currently does a SELECT on every request. Wrap it with the read‑through pattern above (or use your language’s equivalent library). Run a quick load test with something like k6 or hey and watch the difference in error rates and latency.

When you see the cache absorb the traffic like a Jedi deflects blaster bolts, drop a comment with your numbers—or a funny story about the moment the stampede vanished. May the cache be with you! 🚀

Top comments (0)