Our users were logged out for two weeks before Redis refused to write anything

For about two weeks in May, support received a steady trickle of complaints from people who had been logged out in the middle of using our web app. It was not many, we could not reproduce it, and we put it down to browsers clearing cookies. Then on a Thursday afternoon nobody could log in at all, and our logs were full of OOM command not allowed when used memory > 'maxmemory'.

One Redis cluster held our sessions and our page cache. Both kinds of key had an expiry, so years earlier somebody had chosen the eviction policy volatile-lru: when memory is full, evict the least recently used key among those with an expiry. For that data it was a reasonable choice.

In April a new feature started storing each user's recently viewed items in the same Redis, with no expiry, because the list was meant to last. Those keys could never be evicted. As they grew, the space left for keys that could be evicted shrank, and Redis made room the only way the policy allowed: the page cache first, then sessions, least recently active first. Those were the logouts. On the Thursday the evictable keys ran out, and with nothing left that it was allowed to remove, Redis refused every write. A login is a write.

Our dashboards showed memory at its limit, which it had been for months, because a cache is supposed to be full. Evictions were on a panel nobody watched, and nothing showed what share of keys had no expiry.

We moved the recently viewed lists to Postgres that evening and logins came back. Then we split the cluster. The cache has its own Redis with allkeys-lru, where losing any key is fine. Sessions have their own with noeviction, sized for peak with room to spare and an alert at seventy percent, because a session store that evicts is quietly signing people out. Our Redis client wrapper rejects a write without a TTL unless the call says, by name, that the key is permanent. And we alert on any eviction in the session store, and on the share of keys with an expiry, which INFO keyspace reports for every database.

An eviction policy is a decision about which data you are willing to lose. We made it once for two kinds of data and then let a third move in without asking.

– Sergey Shinder