Caching Like a Pro: Lessons from 'Inception'

The Quest Begins (The "Why")

I still remember the night our API went from “quietly humming” to “screaming like a banshee”. A sudden spike in traffic hit our endpoint that fetched user profiles from a slow downstream service. We had a simple cache‑aside layer: check Redis, if miss, call the service, store the result, return it. Sounds fine, right?

The problem showed up when the TTL expired on a popular key. Suddenly hundreds of requests found the cache empty at the exact same moment, all raced to the backend, and our downstream service buckled under the thundering herd. It felt like we were stuck in a dream within a dream—each request trying to kick‑start the same heavy load, over and over. I spent three hours staring at logs, feeling like I was trapped in an endless hallway, and when I finally saw the pattern, it was like the spinning top finally wobbling.

The Revelation (The Insight)

The “aha!” moment came when I realized we weren’t fighting the cache itself—we were fighting how we handled cache misses. The critical insight was simple: make sure only one thread does the expensive work when a key expires, while everyone else waits for that result. In other words, we needed a single‑flight guard around the cache miss.

Think of it like the totem in Inception: it tells you whether you’re in the real world or a dream. Our totem is a single‑flight provides tells us whether we’re the lucky “dreamer” who gets to rebuild the cache, or just a passive observer waiting for the result.

The trade‑off? A tiny bit of extra complexity and a negligible latency for the first request after expiration (the one that does the reload). But we completely eliminate the stampede, keep downstream load steady, and avoid the need for heavyweight global locks that would serialize all cache accesses.

Wielding the Power (Code & Examples)

The Naïve Version (the trap)

// naiveCache.go
type NaiveCache struct {
    redis *redis.Client
    fetch func(string) ([]byte, error) // call to downstream service
}

func (c *NaiveCache) Get(ctx context.Context, key string) ([]byte, error) {
    data, err := c.redis.Get(key).Bytes()
    if err == nil {
        return data, nil // hit
    }

    // miss – everyone ends up here at the same time
    data, err := c.fetch(key)
    if err != nil {
        return nil, err
    }
    // store with a TTL
    c.redis.Set(key, data, 10*time.Minute)
    return data, nil
}

When the TTL expires, dozens of goroutines hit the fetch line simultaneously. The downstream service sees a spike, latency climbs, and we’re back where we started.

The Single‑Flight Upgrade (the victory)

// singleFlightCache.go
import "golang.org/x/sync/singleflight"

type SmartCache struct {
    redis     *redis.Client
    fetch     func(string) ([]byte, error)
    group     singleflight.Group // <-- our totem
}

func (c *SmartCache) Get(ctx context.Context, key string) ([]byte, error) {
    data, err := c.redis.Get(key).Bytes()
    if err == nil {
        return data, nil // cache hit – fast path
    }

    // cache miss – enter the single‑flight zone
    val, err, _ := c.group.Do(key, func() (interface{}, error) {
        // only one goroutine runs this block for a given key
        data, err := c.fetch(key)
        if err != nil {
            return nil, err
        }
        // store the freshly fetched value
        c.redis.Set(key, data, 10*time.Minute)
        return data, nil
    })
    if err != nil {
        return nil, err
    }
    return val.([]byte), nil
}

What changed?

• We wrapped the expensive load in a singleflight.Group.
• For a given key, the first caller executes the fetch; all others block on the same Do call and receive the result when it’s ready.
• No global lock, no serialized hits—only the miss path is coordinated.

ASCII‑Art Flow

   Request A          Request B          Request C
      |                  |                  |
      v                  v                  v
   Cache Miss?  <---   Cache Miss?  <---   Cache Miss?  <--- (TTL expired)
      |                  |                  |
      |------\   Do(key)  /------\   Do(key)  /------\   Do(key)  /
      |       \_________/        \_________/        \_________/
      |        |                |                |
      |   (only A runs fetch)   |                |
      |        |                |                |
      |   fetch downstream      |                |
      |        |                |                |
      |   store in Redis        |                |
      |        |                |                |
      |<------result------------|<---------------|
      v                  v                  v
   Return data    Return data    Return data

Only Request A actually talks to the downstream service; B and C wake up with the cached value almost instantly.

Common Pitfalls (the traps to avoid)

1. Forgetting to key the group correctly – if you use a static string like "user" instead of the actual user ID, all misses serialize, killing concurrency.
2. Ignoring errors from the fetch – if the downstream fails, you’ll propagate the error to every waiter. Consider a fallback or a short‑circuit retry, but don’t swallow the error silently.
3. Setting an absurdly short TTL – the single‑flight guard helps, but if you expire every second you’ll still pay the reload cost constantly. Choose a TTL that matches your data’s volatility.

Why This New Power Matters

With this pattern in place, our API’s latency chart went from a jagged mountain range to a smooth plateau. The downstream service breathed easy; we stopped getting paged at 2 a.m. because of a cache stampede.

The beauty of single‑flight is that it’s lightweight (just a map of channels under the hood) and language‑agnostic—you can find equivalents in Java (CompletableFuture.supplyAsync with a cache), Python (cachetools with a lock), or even a simple Redis‑based Lua script if you’re stuck in a polyglot environment.

Sure, it adds a tiny bit of code and a micro‑second of latency for the first post‑expiration request, but that’s a bargain compared to the alternative: a thundering herd that can melt your instances, inflate your cloud bill, and erode user trust.

If you’re still using a plain “check‑then‑fetch” cache, give this a try. Wrap your miss logic in a single‑flight guard, watch the request graph flatten, and feel that quiet satisfaction when the system just works—like watching the top finally topple and knowing you’re back in reality.

Your Turn

Take a service that’s been hitting your database or external API hard whenever a popular key expires. Sketch out the naive version, then slip in a singleflight.Group (or your language’s equivalent). Run a load test, compare the error rates, and celebrate when the herd disappears.

Got a story of your own cache conquest? Drop it in the comments—I’d love to hear how you’ve tamed the stampede! Happy caching! 🚀