Day 25/60: Caching with Redis -- Cache-Aside, Stampede, and Stale Reads

60 Day Go Backend Engineering Challenge

Our GET /products/{id} endpoint read Postgres on every hit. During a promo the traffic concentrated on a few hundred hot SKUs, so the same SELECT ran thousands of times a second for identical rows. p99 was 180ms, the database sat near 80% CPU, and the shared connection pool was throttling everything else. A Redis cache-aside layer cut p99 to 9ms and dropped read QPS more than 95%. Then it shipped two classic bugs. This post is both.

Cache-aside in Go

The application owns the cache. Read Redis first; on a hit, return; on a miss, read the database, populate with a TTL, return. With go-redis, a missing key is the sentinel error redis.Nil -- a normal miss, not a failure. Any other error is Redis itself misbehaving, and you fall through to the database rather than failing the request.

type Cache struct {
    rdb *redis.Client
    db  *pgxpool.Pool
}

func (c *Cache) Product(ctx context.Context, id string) (Product, error) {
    key := "product:" + id

    switch val, err := c.rdb.Get(ctx, key).Result(); {
    case err == nil:
        var p Product
        if json.Unmarshal([]byte(val), &p) == nil {
            return p, nil // hit
        }
    case err != redis.Nil:
        log.Printf("cache degraded, serving from db: %v", err)
    }

    p, err := c.fetch(ctx, id) // miss -> source of truth
    if err != nil {
        return Product{}, err
    }
    if b, e := json.Marshal(p); e == nil {
        c.rdb.Set(ctx, key, b, c.ttl())
    }
    return p, nil
}

Choosing a TTL

The TTL is your tolerance for being wrong. It is the longest a value may be stale even if every other safety net fails, and it bounds memory by letting cold keys evict themselves. A product blurb tolerates minutes; a price or stock count wants seconds. Pick the longest staleness the feature survives, not a round number. And jitter it, so keys cached together do not all expire on the same tick:

func (c *Cache) ttl() time.Duration {
    // 10 min base + up to 60s jitter to de-synchronize expiries
    return 10*time.Minute + time.Duration(rand.Intn(60))*time.Second
}

The stampede

The first night, every hot-key expiry spiked p99 back to ~150ms. A hot key takes thousands of reads a second; the instant it expires, every concurrent request misses at once and runs the same query. The cache absorbed the load, then released it all in one synchronized burst -- a stampede, the thundering herd. Caching made it worse by aligning the misses perfectly.

Collapse concurrent misses into one DB call with singleflight. N callers for the same key share one execution; N-1 never touch Postgres.

import "golang.org/x/sync/singleflight"

func (c *Cache) load(ctx context.Context, id string) (Product, error) {
    key := "product:" + id
    v, err, _ := c.sf.Do(key, func() (any, error) {
        p, err := c.fetch(ctx, id)
        if err != nil {
            return Product{}, err
        }
        if b, e := json.Marshal(p); e == nil {
            c.rdb.Set(ctx, key, b, c.ttl())
        }
        return p, nil
    })
    if err != nil {
        return Product{}, err
    }
    return v.(Product), nil
}

Jitter spreads expiries apart; singleflight handles whatever still lands together. Use both.

Invalidation on write

The quieter bug: a price update committed to Postgres but the page served the old value for ten minutes, because cache-aside never invalidates itself. Use delete-on-write -- change the row, then delete the key, in that order:

func (c *Cache) Update(ctx context.Context, p Product) error {
    if err := c.save(ctx, p); err != nil {
        return err // source of truth first
    }
    // Invalidate AFTER the commit. Delete, don't overwrite:
    // concurrent writers can race an overwrite into a wrong value.
    if err := c.rdb.Del(ctx, "product:"+p.ID).Err(); err != nil {
        log.Printf("invalidate product:%s failed: %v", p.ID, err)
    }
    return nil
}

Delete first and a failed write evicts a good value; worse, a concurrent read repopulates the stale row in the gap before your write lands. Commit, then invalidate.

Where it stops

Delete-on-write covers writes your service owns. It does not cover a manual SQL update or a second writer on the same table, and the delete is not transactional with the commit, so a crash in between leaves a stale key until the TTL saves you -- which is why the TTL stays mandatory. Stronger guarantees mean change-data-capture or write-through. For a read-heavy endpoint with one clear owner, cache-aside plus singleflight plus delete-on-write plus a sane TTL is the right amount of machinery.

A cache is a copy, and every copy can stampede and go stale. Plan for both on day one.