DEV Community

Cover image for Laravel Redis Cache Stampede, Locks and Memory Limits
devTalk
devTalk

Posted on Originally published at dev-talk.com

Laravel Redis Cache Stampede, Locks and Memory Limits

A cache that works under light traffic can fail under load in three ways: many requests rebuild the same expired value at once, a counter quietly never expires, or Redis fills up and either rejects writes or evicts something you needed. This is Part 2 of the series. Part 1 covers cache-aside, key layout, models and invalidation. Here I reproduce each failure with a real Redis server and then fix it.

Tested on Laravel 13.40.0, PHP 8.3.6 with phpredis 5.3.7, and Redis 7.0.15. Each result below is copied from a script I ran twice with identical output.

Cache stampede: 20 requests, one cold key

When a hot key expires, every request that arrives before the first one finishes rebuilds it. The "slow query" below takes about 0.8 seconds, and I started 20 PHP processes at once against a cold key, counting how many times the closure really ran:

// naive
Cache::remember($key, 60, fn () => slowQuery());
Enter fullscreen mode Exit fullscreen mode
naive: closure ran 20 time(s)
Enter fullscreen mode Exit fullscreen mode

All 20 hit the database. For a query that takes seconds, or a key that thousands of requests want, that is how a cache expiry becomes an outage.

Fix 1: a lock around the rebuild

Laravel's atomic locks (they need the redis, memcached, dynamodb, database, file or array driver, and all servers must use the same central cache server, per the docs) let one process rebuild while the others wait:

use Illuminate\Support\Facades\Cache;

public function dashboardStats(): array
{
    $key = 'stats:dashboard';

    // Fast path: no lock for the common hit.
    $cached = Cache::get($key);
    if ($cached !== null) {
        return $cached;
    }

    // Slow path: one process rebuilds, the rest wait up to 5 seconds.
    return Cache::lock("lock:$key", 10)->block(5, function () use ($key) {
        // Re-check inside the lock: a waiting process finds the value the first one stored.
        return Cache::remember($key, 600, fn () => $this->expensiveQuery());
    });
}
Enter fullscreen mode Exit fullscreen mode
lock: closure ran 1 time(s)
Enter fullscreen mode Exit fullscreen mode

Details that matter:

  • block(5, ...) throws Illuminate\Contracts\Cache\LockTimeoutException if the lock isn't acquired within 5 seconds (documented). Decide what a waiter does then: usually compute anyway or return an error, but not wait forever.
  • The lock's own TTL (10 seconds here) should be longer than the slowest realistic rebuild. If it expires mid-rebuild, a second process can start one. That follows from how expiring locks work; I did not test an expired lock.
  • The re-check inside the lock is what makes the others stop. remember does it for you.
  • By default the lock key is stored through the default Redis connection (database 0), not next to the cache keys. After starting a lock in a test I saw laravel-database-laravel-cache-report in database 0 and nothing in database 1. REDIS_CACHE_LOCK_CONNECTION in config/cache.php controls this.

Fix 2: Cache::flexible (serve stale, refresh in the background)

Cache::flexible takes two numbers: how long the value is fresh, and its total lifetime. Between the two, the stale value is returned immediately and the refresh runs after the response (per the docs, as a deferred function).

$stats = Cache::flexible('stats:dashboard', [60, 600], fn () => $this->expensiveQuery());
Enter fullscreen mode Exit fullscreen mode

I tested it with [2, 6] and a counter for how often the closure ran:

returned compute#1
  runs=1  <- cold miss
returned compute#1
  runs=1  <- fresh hit
returned compute#1
  runs=2  <- stale read, refreshed before the command exits
returned compute#2
  runs=2  <- fresh again
returned compute#3
  runs=3  <- expired, recomputed inline
Enter fullscreen mode Exit fullscreen mode

The stale read returned the old value (compute #1) and then refreshed it, so the next call saw compute #2. After the full 6 seconds, the value was gone and the caller waited for a fresh compute. So the second number is the total lifetime of the entry, not extra time on top of the first. (I ran this as an Artisan command, where the deferred refresh finished before the process exited. In a web request the docs say it runs after the response is sent; I did not test HTTP.)

With 20 processes asking at once while the value was stale, the closure ran once more, not 20 times:

runs after the burst: 2  (1 = the initial compute)
Enter fullscreen mode Exit fullscreen mode

The framework source explains why: the refresh takes a lock and skips the work if another process has already refreshed the value. But flexible does not protect a cold key. With the same 20-process test on an empty cache:

flexible: closure ran 20 time(s)
Enter fullscreen mode Exit fullscreen mode

On a miss, the source computes the value directly without a lock. So the right combination is: use flexible for keys that are already warm, and use the lock pattern (or warm the key from a scheduled command or a deploy step) for the first fill.

Counters: increment() on a missing key never expires

Counters are the other thing people cache in Redis: page views, login attempts, API quotas.

Cache::increment('hits:bad');    // key does not exist yet
Cache::increment('hits:bad');
Enter fullscreen mode Exit fullscreen mode
hits:bad = 2
  TTL=-1  (-1 = never expires)
Enter fullscreen mode Exit fullscreen mode

In Redis, the key was created with no expiry. A "per hour" counter built this way counts forever. The Laravel docs' own example initializes the key first with add, which only writes if the key doesn't exist and is atomic:

Cache::add('hits:good', 0, 60);  // creates it with a 60-second TTL only if missing
Cache::increment('hits:good');
Cache::increment('hits:good');
Enter fullscreen mode Exit fullscreen mode
hits:good = 2
  TTL=60
Enter fullscreen mode Exit fullscreen mode

That gives you a simple fixed-window limiter. Allow 3 requests per minute:

Cache::add('limit:ip', 0, 60);
$count = Cache::increment('limit:ip');

if ($count > 3) {
    // reject
}
Enter fullscreen mode Exit fullscreen mode
request 1 -> count 1 -> allowed
request 2 -> count 2 -> allowed
request 3 -> count 3 -> allowed
request 4 -> count 4 -> blocked
request 5 -> count 5 -> blocked
Enter fullscreen mode Exit fullscreen mode

Two limits to know: a fixed window lets a burst through at the boundary (3 at the end of one minute, 3 at the start of the next), and if the key expires between add and increment, increment recreates it without a TTL. That window is tiny, but I did not try to hit it. For HTTP rate limiting, Laravel's built-in rate limiter is the better tool; this pattern is for your own counters.

Memory: what happens when Redis fills up

Redis does not limit memory unless you tell it to. On a fresh Redis 7.0.15 I read the defaults:

maxmemory         0              (no limit)
maxmemory-policy  noeviction
Enter fullscreen mode Exit fullscreen mode

The Redis docs list the policies. These are the ones that matter for Laravel:

Policy What the Redis docs say
noeviction Nothing is evicted; commands that would add data return an error. Reads still work.
allkeys-lru Evicts the least recently used keys from the whole keyspace.
volatile-lru Evicts the least recently used keys that have a TTL. With no TTL keys, it behaves like noeviction.

I limited maxmemory to 3 MB above current use, then wrote 2,000 values of 4 KB each through Cache::put under each policy. I also stored 100 plain keys without a TTL in database 0 to stand in for data that must not disappear (think queued jobs or sessions; these were raw keys, not real Laravel queue structures).

noeviction: your cache calls start throwing

stored <N>, then RedisException: OOM command not allowed when used memory > 'maxmemory'.
dbsize>0 and <2000: yes; evicted_keys=0
Enter fullscreen mode Exit fullscreen mode

Redis kept what it had and refused new writes. In a Laravel app that means Cache::put throws, and so does anything else that writes to that server. Whether that is acceptable depends on what else lives there.

allkeys-lru: great for a cache, dangerous for everything else

stored all 2000
evicted_keys>0: yes; job keys left in db0: 0 of 100
Enter fullscreen mode Exit fullscreen mode

All 2,000 writes succeeded, and Redis made room by evicting old keys, including all 100 of my no-TTL keys in a different database. Eviction works on the whole server's keyspace, not per database. If your queues, sessions or locks share this Redis instance, allkeys-lru can evict them.

volatile-lru: protects keys without a TTL, until they fill the memory

cache keys with a TTL:
stored all 2000
evicted_keys>0: yes; job keys left in db0: 100 of 100
cache keys with no TTL (Cache::forever):
stored <N>, then RedisException: OOM command not allowed when used memory > 'maxmemory'.
Enter fullscreen mode Exit fullscreen mode

With TTL'd cache writes, Redis evicted only keys that had TTLs and my 100 no-TTL keys survived. But when I wrote the same data with Cache::forever, nothing was evictable and writes failed with the same OOM error. Under volatile-lru, every cache key needs a TTL.

What I'd do

  • Run the cache on its own Redis instance with maxmemory set and allkeys-lru. Keep queues, sessions and anything that must not vanish on another instance (with enough memory and noeviction).
  • Point the cache connection at that instance. In the default config/database.php, the cache connection reads the same REDIS_HOST as default, so add dedicated variables:// config/database.php, in the 'cache' connection 'host' => env('REDIS_CACHE_HOST', env('REDIS_HOST', '127.0.0.1')), 'port' => env('REDIS_CACHE_PORT', env('REDIS_PORT', '6379')),REDIS_CACHE_HOST=10.0.0.5 REDIS_CACHE_PORT=6379 REDIS_CACHE_LOCK_CONNECTION=cache Setting REDIS_CACHE_LOCK_CONNECTION=cache moves the lock keys onto the same instance. I tested this with two local Redis servers: cache keys and the lock key landed on the second one and nothing on the first.
  • Put a TTL on every cache key, even long ones. Avoid Cache::forever and rememberForever unless you are certain the data is small and bounded.
  • Watch used_memory and evicted_keys in INFO. A rising evicted_keys with a falling hit rate means the cache is too small.

Common mistakes

  • Using Cache::remember alone on an expensive, hot key.
  • Assuming Cache::flexible also handles the first fill.
  • Waiting on a lock without handling LockTimeoutException.
  • Calling increment on a key you never created with a TTL.
  • Setting allkeys-lru on a Redis server that also holds queues or sessions.
  • Leaving maxmemory unset and finding out when the host runs out of RAM.

Alternatives

  • Warm the key on a schedule (a scheduled command or a deploy step) so users never see a cold miss. Pair it with a TTL longer than the schedule interval.
  • Add random jitter to TTLs so many keys written together don't expire together. This is a common technique; I did not measure its effect here.
  • Redis Cluster or managed Redis when one node isn't enough. I did not test either.

Related Dev Talk articles

Frequently asked questions

Do I need the lock if I use Cache::flexible?

For the first fill of a key, yes (or warm it ahead of time). Flexible's own refresh is already guarded by an internal lock.

What should the lock TTL be?

Longer than your slowest normal rebuild, short enough that a crashed process doesn't block others for long. As a starting point, a few times the normal rebuild time; measure your own queries.

Can I rely on Redis eviction instead of TTLs?

Not with volatile-* policies, because they only evict keys that have a TTL. With allkeys-* it works, but then the instance should hold nothing you can't lose.

Does the memory test prove how much cache I can hold?

No. It shows how Redis behaves at the limit under three policies. Real capacity depends on your value sizes, key count and Redis overhead, so measure used_memory on your own data.

Sources


Originally published on DEV Talk.

Top comments (0)