There's an old line, usually traced back to Phil Karlton at Netscape"
There are only two hard things in computer science:
cache invalidation and naming things.
More on it in the Side Quest
I'd heard it as a joke for years. Then I spent a few weeks actually living the first half of it, and stopped finding it funny. This is that story — starting with the boring part, which is where it always starts.
At smritea.ai, the backend is predominantly Go, and for a while every section of code that needed to cache something just wrote its own version of the same fifteen lines:
- Check Redis, return data if found.
- If it's a miss, go to the database, find it, write it to redis, return it.
And that just works, but then there are bugs that are hard to track. By the third or fourth service, it was the same bug waiting to happen in slightly different clothes.
Before AI started doing our bidding, this is what used to happen:
- At one place, a developer forgot to set a TTL
- Another didn't handle a cache error as a miss.
- Third one had to fix a quick bug, and had no time to invalidate on write.
- I looked at it, thought that we need better engineer practices, wondered who approved all those PRs, realised it was me. And then left it for a later clean-up.
- Later never came.
Then came AI, and all of this mismatch happens in 10X more places, 10X more often.
None of that is hard to fix individually. What's hard is fixing it consistently across a growing number of services when the fix keeps getting copy-pasted instead of shared.
So I pulled it out into a library — smartcache — and built it the way I'd want to have found it: one generic cache type, a few well-understood failure modes handled once, and a backend you can swap out.
The shape of the cache
The core type is Cache[T], generic over whatever you're caching, sitting on top of a small backend interface:
type CacheStore interface {
// Get returns the raw bytes for key, or ErrStoreMiss if the key is absent.
Get(ctx context.Context, key string) ([]byte, error)
// Set stores val under key with the given ttl. A ttl <= 0 means no expiry.
Set(ctx context.Context, key string, val []byte, ttl time.Duration) error
// Delete removes key. Deleting an absent key is not an error.
Delete(ctx context.Context, key string) error
// Exists reports whether key is present (and not expired).
Exists(ctx context.Context, key string) (bool, error)
}
Cache[T] never sees Redis directly — it talks to CacheStore, so a Redis-backed store and an in-process map-backed store (memstore, mostly for tests) are interchangeable. Reading is read-through:
func (c *Cache[T]) GetByKey(ctx context.Context, key string, loader Loader[T]) (*T, Outcome, error)
Call it with a key and a loader function. On a hit, you get the cached value. On a miss, loader runs, its result gets cached, and you get that back instead. Writing has the mirror shape — PutByKey runs a writer, caches exactly what it returns. The caller never manually juggles "did I remember to cache this."
That much was the easy part. What made it worth writing down is what happened once services actually started using it.
Problem one: everyone misses at the same time
A cached key expires. In the next few milliseconds, twenty requests for that same key show up. All twenty see a miss, because none of them know about the other nineteen. All twenty go to the database for the same row, at the same moment. This is the cache stampede — sometimes called the thundering herd — and it gets worse exactly when you can least afford it: a hot key, under load.
The fix doesn't need anything clever, just coordination between concurrent callers for the same key. Go's singleflight package does exactly that — one call runs, everyone else waiting on the same key gets handed its result instead of starting their own:
var res any
var lerr error
if c.group != nil {
res, lerr, _ = c.group.Do(key, doLoad)
} else {
res, lerr = doLoad()
}
doLoad is the closure that actually calls the loader and populates the cache. group.Do keys on the cache key itself, so twenty concurrent misses on "user:5" collapse into one database call, and nineteen callers get the shared result instead of hitting the database at all. It's on by default; you can turn it off per cache if you have a reason to.
Problem two: everyone expires at the same time
Stampede protection covers concurrent misses on one key. There's a second version of the same failure: a batch of keys all written with the same TTL will all expire at the same moment, and you get a wave of misses across many different keys simultaneously instead of one key. Same underlying cause — synchronised timing — different trigger.
The fix is jitter: shave a small, random amount off each TTL so keys written together don't expire together.
// defaultJitterFraction is the fraction of the base TTL that jitter may shave
// off when a cache does not override it. 0.10 => an effective TTL in
// [0.9*base, base].
const defaultJitterFraction = 0.10
var jitterRand = rand.Float64
// applyJitter returns a downward-jittered TTL: base - jitterRand()*fraction*base.
// It returns base unchanged when base <= 0 (infinite/none) or fraction <= 0
// (jitter disabled). The result is always in [(1-fraction)*base, base] and is
// never negative, since jitterRand() is in [0,1) and fraction is in [0,1).
func applyJitter(base time.Duration, fraction float64) time.Duration {
if base <= 0 || fraction <= 0 {
return base
}
delta := time.Duration(jitterRand() * fraction * float64(base))
return base - delta
}
Every TTL — positive and negative (more on that below) — goes through this before it's used. A 10% jitter on a 10-minute TTL means each key actually expires somewhere between 9 and 10 minutes in. Spread that across a few thousand keys and the expiry wave turns into a trickle.
Problem three: Reduce the cost of saying NO. (Negative caching)
The third one shows up with lookups that can legitimately miss — a user by an ID that doesn't exist, a slug someone mistyped. If a request for a missing key comes in repeatedly, every single one falls through to the database, gets nothing back, and does it again next time. The cache isn't helping at all for that access pattern, because there's never anything to cache — or so it seems until you cache the fact that it's missing, not the missing thing itself.
// negativeMarker is stored in place of a real value to negative-cache "not found".
var negativeMarker = []byte("\x00smartcache\x00negative\x00")
When the loader reports ErrNotFound and a negative TTL is configured, GetByKey writes this marker instead of a real value:
case errors.Is(lerr, ErrNotFound):
if c.opts.NegativeTTL > 0 {
if sErr := c.setValue(ctx, key, negativeMarker, c.negativeTTL()); sErr != nil {
return nil, ErrNotFound
}
}
return nil, ErrNotFound
The next request for that key hits the marker, recognises it, and returns "not found" without touching the database — for a shorter TTL than a real hit, since you'd rather notice a newly-created record sooner than a newly-updated one. It's off by default; you turn it on by setting NegativeTTL, and only where a missing-key flood is actually plausible.
Be careful though, and keep the TTL low. You don't want the cache to stand in the way for too long, in case the DB actually changes
Now that I think of it, what if we remove the negative cache key whenever we write to the key (truly write-through). Too much?
Putting it together
Wrap a Manager around all of this and you get per-cache OTLP metrics for free — hit/miss/load counts and load latency, without every service hand-rolling its own instrumentation. Register a cache once, and stampede protection, jitter, and negative caching are already there; you only turn on what a given cache actually needs.
None of these three problems are unusual — they're the standard failure modes of caching anything at scale, and Redis's own docs on the cache-aside pattern and its cache layer architecture guide cover the same ground. What changed for us wasn't the theory — it was having one place these fixes live, instead of re-deriving them per service.
What it didn't solve
Everything above assumes one key maps to one value. That held until a repository needed to look up a User three different ways — by ID, by email, by slug — and I had to decide what "cache the user" even means when there are three doors into the same room. Caching the value three times means an update has to hit three keys and a bug leaves two of them stale. Caching it once under the ID means the email and slug lookups never get to use the cache at all.
That's where this got genuinely harder, and it's worth being precise about where. None of it is a problem on a single Redis instance — with everything on one node, there's nothing to hotspot in the first place. It turns into one on Redis Cluster: making the value and its aliases update atomically means Redis has to guarantee all of those keys live on the same node, and forcing that onto a busy or large entity puts its entire traffic on one node no matter how big the cluster is. That's where Part 2 picks up, and it took two tries to get the fix right.
And if you want to poke at the real code, smartcache is public.
If you enjoyed this read and want to dive deeper into the trade-off pool, then Part 2 is published as well
───────── ⋆⋅☆⋅⋆ ─────────
Side Quest
The Karlton quote up top is real, but its internet trail is thinner than its fame suggests. I ended up taking a detour while writing this article, because I really had to include it.
The earliest known appearance online is Tim Bray's blog, from December 2005 — Bray says he first heard it from Karlton around 1996–97. Martin Fowler's 2009 bliki entry is what actually popularised it, and Karlton's own son has written about hearing him say it as early as around 1970 at Carnegie Mellon — without being sure it was the first time either.
───────── ⋆⋅☆⋅⋆ ─────────
Thanks for reading this far!
If it helped, I'd love for you to stick around — find me on:
Top comments (0)