How a cache makes apps fast, why stale data happens, and the stampede that takes sites down
A product page took three seconds to load. Then it took three milliseconds.
Same server. Same database. Same code, except for one idea: caching. Almost every fast app you use runs on it.
But the same cache that makes a site a thousand times faster can also take the whole thing down in seconds. This article walks through how caching works, where it goes wrong, and the three-line fix the big companies use. If you'd rather watch, here's the 7-minute animated version:
1. Why the page was slow
Every visit asked the database the same question: what's the price, the stock and the reviews for this product? Thousands of people, the same question, the same answer, computed again every single time.
And asking is expensive. Reading from memory takes around a hundred nanoseconds. A database query over the network takes milliseconds, and a heavy one can take seconds. That's tens of thousands of times slower.
So why ask again when you already know the answer? That's the whole trick: keep the answer somewhere much closer.
2. What a cache is
A cache is just a copy of data, kept closer to where it's needed, so the next read is fast.
You use caches all day without noticing. Your CPU caches memory. Your browser caches images. A CDN caches whole pages in a city near you. And your backend usually has its own cache, often Redis or Memcached, sitting in memory right next to your app.
It's the same idea at every layer: the closer the copy, the faster the answer.
3. Cache-aside: the pattern most apps use
The most common pattern is called cache-aside, and it has three steps:
Check the cache. If the answer is there, that's a hit, and you return it in about a millisecond.
On a miss, go to the database and get the real answer.
Save that answer in the cache with an expiry time (a TTL), so the next person gets a hit.
def get_product(id):
value = cache.get(f"product:{id}")
if value is None: # miss
value = db.query_product(id)
cache.set(f"product:{id}", value, ttl=300)
return value
The first visitor pays the full price, and everyone after them gets it almost for free.
4. The number that decides everything: hit ratio
The hit ratio is the share of requests the cache answers. Say a hit takes 1 ms and a miss takes 100 ms. At a 50% hit ratio, the average request takes about 50 ms. At 99%, it's about 2 ms.
And it's not just speed. Every hit is one less query, so a 99% hit ratio means your database sees only one request in every hundred. That's why a small cache can let a small database serve a huge amount of traffic.
5. Stale data
But the cache is a copy, and copies go out of date. The price of item 42 changes from ₹999 to ₹799 in the database, but the cache still has ₹999. For the next few minutes, customers see the wrong number.
There's a famous line, usually credited to Phil Karlton:
There are only two hard things in Computer Science: cache invalidation and naming things.
Invalidation just means deciding when a cached copy is no longer true, and getting rid of it.
6. The bug most teams ship
When data changes, the instinct is to update the cache, then update the database. But if two updates run at the same time, they can finish in different orders. Now the cache says one price and the database says another, possibly for hours.
The safer default is simple: write to the database first, then delete the cache key. Don't update it, delete it. The next read misses, fetches the fresh value and refills the cache.
def update_price(id, price):
db.update_price(id, price) # 1. the source of truth first
cache.delete(f"product:{id}") # 2. then drop the copy
And always set a TTL as a safety net. Even if an invalidation gets lost, the stale copy dies on its own.
7. When the cache fills up
Caches live in memory, and memory runs out. The most common eviction rule is LRU, least recently used: throw out whatever nobody has asked for in the longest time. Popular items stay, and forgotten ones leave.
8. The stampede
It's the biggest sale of the year. Your homepage is cached, and a hundred thousand people are refreshing it every minute. Then the homepage's cache entry hits its TTL and expires.
In that moment, every request misses at the same time. Thousands of requests all go to the database at once, all asking for the same homepage. The database slows down, slow queries pile up, requests time out and retry, which adds even more load. And because the database is drowning, nobody manages to refill the cache.
That's a cache stampede, also called a thundering herd. The site goes down at its busiest minute.
9. The three-line fix
Let only one request rebuild the value. Everyone else waits for it, or gets the slightly old copy in the meantime. And give your TTLs a little random jitter, so thousands of keys don't expire in the same second.
if cache.lock(key): # one request rebuilds
cache.set(key, db.get(key), ttl + jitter())
else: return cache.stale(key) # everyone else waits or gets stale
One request rebuilds, everyone else waits.
10. The rule that ties it together
Cache data that's read a lot and changes rarely: product pages, profiles, settings, the results of expensive queries.
Give every entry a TTL.
Design for the miss, because misses always happen: on deploys, on restarts, and when a key expires at the worst possible moment.
A cache should make your app faster. It should never be the only thing holding it up.
11. How the biggest sites do it
Facebook runs one of the largest Memcached deployments ever built. In their paper Scaling Memcache at Facebook (NSDI 2013), their engineers describe leases: one client gets a lease to refill a missing key, and everyone else waits briefly or uses a slightly stale value. That's the stampede fix at enormous scale.
CDNs like Cloudflare cache pages in data centres around the world, so most visitors never reach the origin server at all. And your browser respects cache headers like Cache-Control, which is why a site loads faster the second time you visit.
Same idea everywhere: keep a copy close, and protect the source.
12. Caching in one screen
A cache is a copy kept close.
Cache-aside: check, miss, fetch, save.
The hit ratio decides how much you win.
On writes, update the database, then delete the key.
Always set a TTL.
Protect against stampedes with one rebuilder and jittered expiry.
But there's one thing a cache can't fix: a slow miss. If the query behind it takes nine seconds, every miss still takes nine seconds.
▶️ Watch the animated version: Caching Explained (YouTube)
Next up: how one line of SQL turns a nine-second query into two milliseconds. Database Indexes, Saturday.
*If this helped, leave a comment and follow The Byte Mechanics for more visual explainers.
▶️ YouTube: youtube.com/@thebytemechanics, full animated videos
📸 Instagram: instagram.com/thebytemechanics, 60-second Reels
📘 Facebook: facebook.com/thebytemechanics, Reels and updates
𝕏 X: x.com/thebytemechanic, quick takes and diagrams
✍️ Medium: medium.com/@thebytemechanics, the written deep dives
About: The Byte Mechanics turns how software really works into short animated explainers: system design, backend and operating systems, one diagram at a time.
Top comments (0)