The Cache Stampede: How One Expired Key Doomed Our Redis Cluster

Using object caching in Redis is extremely important in case there is a need to speed up any application with high load… until one expires a key when traffic is high.

We got caught by a Cache Stampede attack. This was quite a complicated database report dashboard with one very resource-intensive query which took about 3 seconds to execute on custom tables of ours. We had proper caching set up for this query with an expiry time of 1 hour.

And trouble started the second it expired.

In that moment, 150 active users were visiting our dashboard at once, causing the system to receive 150 un-cached database queries, each taking 3 seconds. At once, the database CPU load reached 100% usage, PHP workers got overwhelmed, and the website went down before it could re-cache anything.

Solution:
Now we do not use any expiry times for cache keys. Instead, we now calculate a probabilistic expiry time, apply a Redis Lock, return data to the client from cache while refreshing the cache in a background process.

Let’s not allow the number of concurrent users of the website to determine how many database hits it receives.

Do not let the number of concurrent web users decide how hard your database will be hit.