Why a 99% Cache Hit Rate Is 10x Faster Than 90% (Not 9%)

I ran the same query 10,000 times: almost 2 seconds of database work for an answer that never changed. The database did not fail. I just kept asking it the same question.

Before fixing it, I wrote down a prediction: a cache should make this about 100 times faster. Keep that number. It is wrong in a way that teaches something.

The whole pattern is a map

Cache-aside is the oldest trick in the book: a map sitting in front of the database. Check the map first. Finding nothing is a miss: run the real query, store the answer on the way out. Next time the map already holds it. That is a hit, and the database never even hears about it.

Map<String, Result> cache = new ConcurrentHashMap<>();

Result read(String key) {
    Result r = cache.get(key);          // 1. check the cache
    if (r != null) return r;            //    hit: the database never hears about it
    r = db.query(key);                  // 2. miss: pay the real price
    cache.put(key, r);                  // 3. store on the way out
    return r;
}

Same 10,000 reads, through the cache this time: 0.6 milliseconds, one database call. That is 3,084x faster. My guess was off by a factor of 30.

The line that matters is the smallest one: the database ran the query once. The cache did not make the query faster. The query still costs what it costs. The cache made it rare. Out of 10,000 reads, 9,999 paid a map lookup (nanoseconds) and one read paid the real price.

One honest note: this map lives inside my process. A cache across the network, like Redis, pays a small toll per hit. The pattern does not change.

The ratio that runs everything

Real systems are not this lucky. Keys change, answers expire, new users arrive. So the number every team watches is the hit rate: the share of reads the cache answers. I forced four hit rates onto the same 10,000 reads. The ratio is simulated; the clock is not.

hit rate	misses (of 10,000)	measured time
0%	10,000	1,993.9 ms
50%	5,000	~1,000 ms
90%	1,000	199.3 ms
99%	100	20.1 ms

Look at the last two rows. 90% to 99% reads like a 9% improvement. It measured 10x. Because a cache does not pay for hits, it pays for misses: at 90%, 1,000 misses are left; at 99, only 100.

That is the whole game. Read the misses, not the hits.

Price one: staleness

The real value keeps changing, and the cache keeps answering from the past. The standard fix is a TTL: keep each answer for 300 ms, then fetch fresh.

I measured that against a value that changes 10 times a second. In one second, the cache served 83 reads. 71 of them were wrong. That is 86% of the answers already out of date, from a cache with nothing broken.

Staleness is not a bug. It is the first price of caching. You do not remove that price; you pick data that can afford it. A profile photo can be a minute old. An account balance cannot. Cache the photo. Never the balance.

Price two: memory

Every cached answer is an object the garbage collector cannot clean, because the map still holds a reference to it. I wrote 60,000 entries into a cache with no limit: 70 MB, all 60,000 still there. Two episodes ago we named this pattern: a memory leak with good intentions.

The fix is a limit. In Java it is one method override:

Map<String, Result> cache = new LinkedHashMap<>(16, 0.75f, true) { // true = access order
    protected boolean removeEldestEntry(Map.Entry<String, Result> eldest) {
        return size() > 1_000;          // keep the 1,000 newest, drop the oldest
    }
};

Same 60,000 writes: 8 MB. The limit is what makes it a cache.

Zoom out

The same pattern is everywhere. Your browser caches. The CDN caches. The database even caches its own disk pages. Every layer is making the same bet: the next read wants the last answer.

One pattern. One ratio. Two prices.

Next time: you update the database, and the cache keeps serving the old answer. Invalidation.

The full episode (embedded at the top) builds the cache live, walks the measured hit-rate ladder, and runs the staleness experiment on screen.

Numbers are from a demo on my machine: the same query run 10,000 times against a real database, timed before and after the cache. The hit-rate ladder forces the ratio (0 / 50 / 90 / 99 percent) but the clock is real. The staleness run is a value changing 10 times a second behind a 300 ms TTL. The memory numbers are heap measurements of the same 60,000 writes with and without an LRU limit. Your database, driver, and network will move the constants; the ratio logic is the part that travels.

What is your cache hit rate in production, and do you actually alert on it?