DEV Community

SoftwareDevs mvpfactory.io
SoftwareDevs mvpfactory.io

Posted on • Originally published at mvpfactory.io

Redis Beyond Caching: Sorted Sets, Streams, and Lua Scripts That Replace Microservices

#programming #webdev

What We Will Build

In this workshop, I will walk you through three Redis patterns that go far beyond GET/SET/EXPIRE. By the end, you will have working examples for a real-time leaderboard with O(log N) updates, an event sourcing pipeline using Redis Streams (no Kafka required), and an atomic Lua rate limiter that eliminates race conditions. I have seen a single well-configured Redis instance absorb the responsibilities of three separate microservices in production. Let me show you how.

Prerequisites

  • A running Redis instance (6.2+ recommended)
  • Basic familiarity with Redis CLI commands
  • Understanding of key-value data patterns

Step 1: Sorted Sets for Real-Time Leaderboards

The ZSET does not get enough credit. Every insert, update, and rank lookup runs at O(log N) against a skip list internally. Here is the minimal setup to get this working.

ZADD leaderboard 1500 "player:42"
ZADD leaderboard 1620 "player:17"
ZINCRBY leaderboard 30 "player:42"
ZREVRANK leaderboard "player:42"    -- returns 0 (top rank)
ZREVRANGE leaderboard 0 9 WITHSCORES -- top 10
Enter fullscreen mode Exit fullscreen mode

At 1 million players, ZREVRANK returns in under 1ms. I have measured consistent sub-millisecond p99 latencies on sorted sets with 5M+ members in production. Compare that to PostgreSQL, where getting a rank means SELECT COUNT(*) WHERE score > x — a full scan or materialized view. Concurrent writers hit row-level locks and potential deadlocks. Redis is single-threaded, so no locks are needed. That is not a benchmark game; it just stays flat.

Step 2: Redis Streams as a Lightweight Kafka Alternative

Redis Streams (XADD, XREAD, XREADGROUP) give you an append-only log with consumer groups, message acknowledgment, and pending entry tracking — without ZooKeeper, JVM tuning, or partition rebalancing.

-- Producer: append event
XADD orders:events * action "placed" order_id "ord-991" total "89.99"

-- Consumer group setup
XGROUP CREATE orders:events fulfillment-svc $ MKSTREAM

-- Consumer: read and acknowledge
XREADGROUP GROUP fulfillment-svc worker-1 COUNT 10 BLOCK 2000 STREAMS orders:events >
XACK orders:events fulfillment-svc 1684012345678-0
Enter fullscreen mode Exit fullscreen mode

For systems processing under 200K events per second — which covers most startups and mid-scale SaaS products — Redis Streams eliminate the entire Kafka operational burden. You get consumer groups, pending entry lists for retry logic (XPENDING), and XCLAIM for rebalancing dead consumers. A complete event sourcing backbone without a single JVM process.

Step 3: Lua Scripting for Atomic Multi-Key Operations

Here is the gotcha that will save you hours. A Lua script executes atomically on the Redis server. No other command runs between your script's operations. This eliminates distributed locks, saga orchestrators, and retry middleware for many common patterns.

Here is a sliding window rate limiter — the pattern I used to replace a dedicated rate-limiting microservice, its API gateway sidecar, its own Redis instance, and its deployment pipeline. Twelve lines of Lua:

-- KEYS[1] = rate limit key
-- ARGV[1] = window (sec), ARGV[2] = max requests, ARGV[3] = now
local key = KEYS[1]
local window = tonumber(ARGV[1])
local max_req = tonumber(ARGV[2])
local now = tonumber(ARGV[3])

redis.call('ZREMRANGEBYSCORE', key, 0, now - window * 1000)
local count = redis.call('ZCARD', key)
if count < max_req then
    redis.call('ZADD', key, now, now .. '-' .. math.random(1000000))
    redis.call('PEXPIRE', key, window * 1000)
    return 1
end
return 0
Enter fullscreen mode Exit fullscreen mode

Without Lua, this pattern requires a distributed lock (Redlock or a separate service) to prevent TOCTOU races between ZCARD and ZADD. With Lua, it is a single atomic call via EVALSHA.

Gotchas

  • Streams are not Kafka. Kafka wins when you need multi-datacenter replication or million-message-per-second partitions. Redis Streams are the 80% solution that saves you from running Kafka when you do not need it.
  • Lua scripts block Redis. Since Redis is single-threaded, a long-running Lua script stalls all other commands. Keep scripts short and deterministic.
  • Sorted sets live in memory. A ZSET with 5M members works great, but plan your memory budget. The docs do not mention this, but member names contribute significantly to memory usage — keep them short.
  • Do not ignore persistence. If you are using Redis as a primary data layer, configure RDB snapshots or AOF. Losing your leaderboard on restart is not a caching miss — it is data loss.

Conclusion

Audit your cache-only Redis usage. If you are only using GET/SET/EXPIRE, you are ignoring 90% of what is available. Sorted sets handle ranking natively. Streams give you consumer groups at a fraction of Kafka's operational cost. Lua scripts eliminate both race conditions and extra services. Redis is not your cache layer — it is a programmable data engine. Let me show you a pattern I use in every project: treat Redis as a first-class data layer, and watch entire services become unnecessary.

Top comments (0)