Redis for Data Engineers: Beyond Caching

Redis for data engineers is a mental-model upgrade, not a new tool to learn — the same in-memory server most teams reach for as a dumb key-value cache is actually a data-structure server whose sorted sets, append-only streams, and probabilistic sketches quietly power leaderboards, rate limiters, deduplicated event ingestion, and billion-row unique-visitor counts inside pipelines you already run. The reason so many engineers stall at GET/SET is that the caching use case is so obvious it hides everything else: the moment you treat a Redis key as a shape — a ranked set ordered by score, a log you can replay with consumer groups, a 12-kilobyte cardinality estimator — the same box that memoizes your query results also becomes the cheapest way to solve an entire class of streaming and analytics problems.

This guide is the walkthrough you wished existed the first time an interviewer asked "you have Redis in the stack already — how would you build a sliding-window rate limiter without hammering Postgres?", or "count the daily unique users across two billion events without storing two billion IDs", or "why would you pick a stream over a list for an ingestion buffer?". It opens the toolbox in layers: the five core value types and the O(1)/O(log N) primitives that make them fast, Redis Streams with consumer groups for durable at-least-once fan-out, sorted sets as the Swiss-army structure behind leaderboards, time-series windows and rate limiting, the HyperLogLog and Bloom filters that trade exactness for constant memory, and finally the persistence, eviction, and atomicity model that decides whether your Redis survives a restart. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the data-structures practice library →, rehearse storage internals on the database practice library →, and sharpen the ingestion axis with the streaming practice library →.

---

On this page

• Redis core data structures — strings, hashes, lists, sets, sorted sets
• Redis Streams — the append-only log with consumer groups
• Sorted sets — leaderboards, time-series windows, rate limiting
• Probabilistic structures — HyperLogLog and Bloom filters
• Persistence, patterns and pitfalls
• Cheat sheet — Redis for data engineers recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Redis core data structures — strings, hashes, lists, sets, sorted sets

The value type you pick is the optimization — five shapes, each with O(1) or O(log N) primitives

The one-sentence invariant: Redis is not a key-value store that happens to have extras — it is a data-structure server where every key holds one of a handful of typed values (string, hash, list, set, sorted set, stream, and a few specialised sketches), and choosing the right shape gives you the exact algorithmic complexity you need for free, because the command set for each type is a curated list of the operations that structure supports cheaply. A counter is a string you INCR; a session is a hash you update field-by-field; a capped event buffer is a list you LTRIM; a de-duplicated membership test is a set you SISMEMBER; a ranked scoreboard is a sorted set you ZADD. The engineer who reaches for GET/SET and serialises a JSON blob into a single string is leaving every one of those primitives on the table.

The five value types and what each is for.

• String. The atom — bytes, an integer, or a float up to 512 MB. But the killer feature is atomic numeric mutation: INCR, INCRBY, DECR, INCRBYFLOAT mutate a counter without a read-modify-write race. Strings back counters, rate-limit tallies, feature flags, and cache entries. SETEX key ttl value writes value + TTL in one command.
• Hash. A field-to-value map under one key — HSET user:42 name Ada plan pro. You read or write individual fields (HGET, HSET, HINCRBY) without deserialising the whole object. This is the correct shape for a "row" — a session, a user profile, a feature-store record — because partial updates are O(1) and memory is far tighter than N separate string keys.
• List. An ordered, doubly-linked sequence — LPUSH/RPUSH at the ends, LPOP/RPOP to drain, LRANGE to read a window, LTRIM to cap length. Lists back simple queues, most-recent-N feeds, and capped log buffers. BLPOP blocks for producer/consumer hand-off.
• Set. An unordered collection of unique members — SADD, SISMEMBER (O(1) membership), SCARD (count), and set algebra (SINTER, SUNION, SDIFF). Sets back de-duplication, tag membership, "has this user seen X", and audience intersections.
• Sorted set (zset). A set where every member carries a floating-point score, kept ordered by score — ZADD, ZRANGE/ZREVRANGE, ZRANK, ZINCRBY, ZRANGEBYSCORE. This is the most powerful data-engineering structure in Redis; section 3 is entirely about it.

The encoding internals interviewers love to probe.

• Small collections are packed. A small hash, list, set, or zset is stored as a listpack (a compact contiguous byte array) — tiny memory, linear scan, but the collection is small so scans are cheap. Above configurable thresholds (hash-max-listpack-entries, zset-max-listpack-entries, etc.) Redis converts to the full structure: a hashtable for hashes/sets, a skiplist + hashtable for zsets.
• Why it matters. The conversion changes both memory footprint and complexity. A 100-field hash in listpack encoding is compact; a 100,000-field hash is a hashtable with O(1) field access but far more overhead. Knowing the threshold explains "why did my memory jump 4× when the collection grew".
• Integer sets. A set of only integers uses the intset encoding — a sorted integer array — until it grows or gains a non-integer member. This is why a set of user IDs is dramatically smaller than a set of UUIDs.

TTL — a first-class property of every key.

• Any key can expire. EXPIRE key seconds, PEXPIRE (ms), EXPIREAT (absolute), and the write-plus-TTL shortcuts SETEX/SET key val EX 60. TTL key reads the remaining life; PERSIST removes it.
• Expiration is lazy + sampled. Redis does not scan for expired keys; it evicts on access (lazy) and via a background sampler that probabilistically clears expired keys. The practical consequence: an expired key stops answering immediately, but its memory is reclaimed slightly later.
• Data-engineering use. TTL is how you build rolling windows, idempotency guards, and self-cleaning caches without a cron job. A rate-limit key that expires after the window, a "processed event ID" set member that expires after the dedup horizon — TTL removes the reconcile job.

What interviewers listen for.

• Do you name the structure, not the command? — "that's a sorted set" beats "I'd use ZADD".
• Do you reach for a hash for object rows instead of a serialised JSON string? — senior signal on memory + partial-update awareness.
• Do you know INCR is atomic and eliminates the read-modify-write race? — required answer for any counter question.
• Do you mention the listpack → hashtable/skiplist conversion when asked about memory? — senior signal.

Worked example — atomic counters with INCR and EXPIRE

Detailed explanation. The most common Redis anti-pattern is GET count, increment in application code, SET count. Under concurrency this loses updates: two workers read 10, both write 11, one increment vanishes. Redis's INCR collapses read-modify-write into a single atomic server-side operation. Pair it with EXPIRE and you have a windowed counter — page views this hour, API calls this minute — that cleans itself up.

• Atomicity. INCR is executed by Redis's single-threaded command loop; no two increments interleave.
• First-write TTL. Set the TTL only on the first increment so the window starts on the first event, not on every event.
• Read. GET returns the current tally; the key vanishes when the window ends.

Question. Build an hourly page-view counter for an article that resets every hour and never loses an increment under concurrent writers.

Input.

Parameter	Value
Key pattern	views:article:{id}:{yyyymmddHH}
Increment op	INCR
Window	1 hour
TTL policy	set on first increment only

Code.

import redis
from datetime import datetime, timezone

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

def record_view(article_id: int) -> int:
    # Bucket key by hour so each hour is a fresh counter
    hour = datetime.now(timezone.utc).strftime("%Y%m%d%H")
    key  = f"views:article:{article_id}:{hour}"

    # INCR is atomic and creates the key at 0 if absent
    count = r.incr(key)

    # Only arm the TTL on the first increment (count == 1),
    # so the window is anchored to the first view of the hour.
    if count == 1:
        r.expire(key, 3600)   # key self-destructs after 1 hour

    return count

def views_this_hour(article_id: int) -> int:
    hour = datetime.now(timezone.utc).strftime("%Y%m%d%H")
    val  = r.get(f"views:article:{article_id}:{hour}")
    return int(val) if val else 0

Step-by-step explanation.

1. The key embeds the hour bucket (...:2026090514), so each hour is a physically separate counter. There is no "reset" step — the new hour simply writes to a new key, and the old key expires on its own.
2. r.incr(key) atomically creates the key as 0 if it does not exist, adds one, and returns the new value — all inside Redis's single-threaded execution, so concurrent callers never lose an increment.
3. The if count == 1 guard arms the TTL exactly once. Calling EXPIRE on every increment would keep pushing the expiry forward (a sliding TTL), which is not what an hourly bucket wants — we want a fixed 1-hour life from the first view.
4. views_this_hour reads the current bucket with GET; a missing key (no views yet, or the hour already expired) returns 0. There is no cleanup job — expiry is handled by Redis.
5. Because the counter lives in Redis, the hot write path never touches the primary database. A nightly job can SCAN the day's buckets and roll them into the warehouse if you need durable history.

Output.

Event	Command	Return	TTL after
1st view, 14:03	INCR views:...:2026090514	1	EXPIRE 3600 armed
2nd view, 14:07	INCR	2	~3360 s remaining
3rd view, 14:59	INCR	3	~40 s remaining
view at 15:00	INCR views:...:2026090515	1	new bucket, TTL armed

Rule of thumb. For any counter, use INCR — never GET + application-side add + SET. Bucket by time in the key name for windowed counts, and arm EXPIRE only on the first increment so the window is anchored, not sliding.

Worked example — a hash-backed session / feature-store row

Detailed explanation. A common mistake is storing a user object as a serialised JSON string: every field read deserialises the whole blob, every field write re-serialises and rewrites it, and two concurrent field updates clobber each other. A Redis hash stores the object as independent fields under one key — read one field, write one field, atomically increment a numeric field — with far tighter memory than N separate string keys.

• Partial reads/writes. HGET, HSET, HDEL, HINCRBY touch single fields in O(1).
• Whole-object read. HGETALL returns every field when you genuinely need the full row.
• Atomic field math. HINCRBY user:42 login_count 1 bumps a counter inside the row without a read.

Question. Model a user session as a hash that stores profile fields, tracks a login counter, and expires 30 minutes after the last activity.

Input.

Field	Type	Example
user_id	string	42
plan	string	pro
login_count	integer	7
last_seen	string (iso)	2026-09-05T14:03:00Z

Code.

import redis
from datetime import datetime, timezone

r = redis.Redis(decode_responses=True)
SESSION_TTL = 1800  # 30 minutes

def start_session(session_id: str, user_id: int, plan: str) -> None:
    key = f"session:{session_id}"
    # HSET writes multiple fields in one round trip
    r.hset(key, mapping={
        "user_id": user_id,
        "plan": plan,
        "login_count": 0,
        "last_seen": datetime.now(timezone.utc).isoformat(),
    })
    r.expire(key, SESSION_TTL)

def touch(session_id: str) -> None:
    key = f"session:{session_id}"
    # Atomic field increment + field update, then slide the TTL forward
    r.hincrby(key, "login_count", 1)
    r.hset(key, "last_seen", datetime.now(timezone.utc).isoformat())
    r.expire(key, SESSION_TTL)   # sliding expiry: 30 min from last touch

def plan_for(session_id: str) -> str | None:
    # Read ONE field without deserialising the whole object
    return r.hget(f"session:{session_id}", "plan")

Step-by-step explanation.

1. HSET ... mapping={...} writes all four fields in a single round trip. Compared to four separate SET session:{id}:plan ... keys, the hash uses one key's overhead and keeps the fields co-located for cheap HGETALL.
2. touch uses HINCRBY to bump login_count atomically — no read of the old value, no lost-update race — and HSET to refresh last_seen. Both mutate single fields; the rest of the row is untouched.
3. The EXPIRE in touch implements a sliding session: every activity resets the 30-minute clock, so an active user never expires and an idle session is reclaimed automatically. This is the opposite policy from the hourly counter, and the choice is deliberate.
4. plan_for reads exactly one field with HGET. A JSON-string design would fetch and parse the entire object to read one attribute — wasteful on a hot path.
5. Because the whole session is one key, it participates cleanly in TTL, DEL, and replication as a unit. There is no risk of the plan key surviving while the login_count key expired — a classic bug when you spread one object across many keys with independent TTLs.

Output.

Step	Command	Effect
login	HSET session:abc ...	row created, 4 fields
activity	HINCRBY session:abc login_count 1	login_count → 1
activity	EXPIRE session:abc 1800	TTL reset to 30 min
read plan	HGET session:abc plan	"pro" (one field)
30 min idle	(lazy expiry)	key removed

Rule of thumb. Store an object as a hash, not a serialised string, whenever you read or write individual fields. Use HINCRBY for in-row counters and reset the TTL on activity for a sliding session, or set it once for a fixed window.

Worked example — a list as a capped "last N events" buffer

Detailed explanation. You often need the most-recent-N of something — last 100 log lines for a job, last 50 actions for a user, a bounded in-memory queue that never grows without limit. A Redis list with LPUSH + LTRIM gives you exactly that: push onto the head, then trim the list to the newest N, in two O(1)/O(N-window) commands.

• Push newest to head. LPUSH key value prepends; the newest item is index 0.
• Cap length. LTRIM key 0 N-1 keeps only indices 0..N-1, discarding the tail.
• Read the window. LRANGE key 0 -1 returns the whole capped buffer, newest first.

Question. Keep the 100 most recent audit events for each pipeline run, discarding older events automatically.

Input.

Parameter	Value
Key	audit:run:{run_id}
Push op	LPUSH (newest first)
Cap	100
Trim op	LTRIM 0 99

Code.

import redis, json
r = redis.Redis(decode_responses=True)

CAP = 100

def record_audit(run_id: str, event: dict) -> None:
    key = f"audit:run:{run_id}"
    # Pipeline: push + trim atomically-ish in one round trip
    pipe = r.pipeline()
    pipe.lpush(key, json.dumps(event))     # newest at head
    pipe.ltrim(key, 0, CAP - 1)            # keep only newest 100
    pipe.expire(key, 86400)                # keep the buffer 1 day
    pipe.execute()

def recent_audits(run_id: str, n: int = 100) -> list[dict]:
    raw = r.lrange(f"audit:run:{run_id}", 0, n - 1)  # newest first
    return [json.loads(x) for x in raw]

Step-by-step explanation.

1. LPUSH prepends the new event, so index 0 is always the most recent. The complementary choice — RPUSH + LTRIM 0 99 — would keep the oldest 100, which is rarely what "recent events" means.
2. LTRIM key 0 99 keeps only the first 100 elements and drops the rest in one command. Without it, the list would grow unbounded — the single most common list-related memory leak.
3. Wrapping LPUSH + LTRIM + EXPIRE in a pipeline sends all three commands in one network round trip. (For strict atomicity you would use MULTI/EXEC or a Lua script — covered in section 5 — but a pipeline is fine here because trim-after-push is self-correcting.)
4. EXPIRE key 86400 bounds the buffer's lifetime to a day, so completed runs eventually vanish even if nobody deletes them.
5. LRANGE key 0 n-1 returns the newest-first window for display. Because the list is capped at 100, reads are cheap and predictable — no scanning a million-element list.

Output.

Operation	List state (head → tail)
push e1	[e1]
push e2	[e2, e1]
push e3..e101	[e101 ... e2, e1] (101 items)
after LTRIM 0 99	[e101 ... e2] (100 items; e1 dropped)

Rule of thumb. For a "last N" buffer, always pair LPUSH with LTRIM 0 N-1 in the same round trip. A list without an LTRIM is an unbounded memory leak waiting to happen.

Data engineering interview question on choosing the right structure

A senior interviewer might ask: "You need a server-side store for web sessions: read and write individual attributes cheaply, atomically bump a per-session request counter, expire a session 30 minutes after the last activity, and keep memory tight across ten million concurrent sessions. Which Redis structure do you pick, how do you key it, and how do you avoid the classic 'partial object' bugs?"

Solution Using a hash-per-session with a sliding TTL and atomic field counters

import redis
from datetime import datetime, timezone

r = redis.Redis(decode_responses=True)
SESSION_TTL = 1800  # 30 min sliding

def create(session_id: str, user_id: int, plan: str) -> None:
    key = f"sess:{session_id}"
    with r.pipeline() as pipe:
        pipe.hset(key, mapping={
            "user_id": user_id,
            "plan": plan,
            "req_count": 0,
            "created_at": datetime.now(timezone.utc).isoformat(),
        })
        pipe.expire(key, SESSION_TTL)
        pipe.execute()

def on_request(session_id: str) -> int:
    key = f"sess:{session_id}"
    with r.pipeline() as pipe:
        pipe.hincrby(key, "req_count", 1)   # atomic in-row counter
        pipe.expire(key, SESSION_TTL)       # slide the window
        count, _ = pipe.execute()
    return count

def read_field(session_id: str, field: str) -> str | None:
    return r.hget(f"sess:{session_id}", field)

Step-by-step trace.

Step	Session state	Reasoning
create("abc", 42, "pro")	{user_id:42, plan:pro, req_count:0} TTL 1800	one key, all fields, tight memory
on_request("abc") @ t=0	req_count → 1, TTL reset to 1800	atomic bump; sliding window
read_field("abc","plan")	returns "pro"	single-field O(1) read
on_request("abc") @ t=1200	req_count → 2, TTL reset	still active; never expires while used
idle 1800 s	key evicted	self-cleaning; no cron

After deployment, each session is exactly one hash key: field reads and writes are O(1), the request counter can never lose an increment because HINCRBY runs server-side, and idle sessions reclaim their own memory via TTL. Ten million sessions of a handful of small fields each stay in listpack encoding until they grow, keeping per-session overhead minimal.

Output:

Metric	Value
Keys per session	1 (hash)
Field read/write	O(1)
Counter safety	atomic (HINCRBY)
Expiry	30 min sliding, self-cleaning
Encoding (small row)	listpack (compact)

Why this works — concept by concept:

• Hash-per-object — one key holds the whole session, so reads and writes target individual fields (HGET/HSET) without deserialising a blob, and the object expires, replicates, and deletes as one unit. This eliminates the "some fields expired, some survived" bug of spreading an object across many keys.
• HINCRBY atomicity — the per-session request counter is mutated server-side in Redis's single-threaded loop, so concurrent requests can never lose an increment the way a read-modify-write in application code would.
• Sliding TTL via EXPIRE on activity — resetting the TTL on every request keeps active sessions alive and lets idle ones self-destruct, replacing a session-reaper cron job with a Redis primitive.
• Listpack encoding for small rows — a session with a few small fields stays in the compact listpack encoding, so ten million sessions cost far less memory than ten million multi-key objects; the structure only converts to a hashtable if a single session grows past the entry threshold.
• Cost — O(1) per field operation and O(1) per session for expiry sampling; memory is O(fields) per session. Compared to a relational session table with row locks and a TTL sweep job, the Redis model is a constant-time hot path with zero cleanup infrastructure.

Data structures
Topic — data-structures
Data-structure selection and complexity problems

Practice →

Database Topic — database Database and key-value store design problems

Practice →

---

2. Redis Streams — the append-only log with consumer groups

redis streams turn Redis into a durable, replayable log with at-least-once consumer groups — the ingestion buffer you already have

The mental model in one line: a Redis Stream is an append-only, ID-ordered log (like a single-partition Kafka topic) living inside Redis — producers XADD entries that get a monotonic ms-seq ID, consumers read ranges with XRANGE/XREAD, and consumer groups (XREADGROUP + XACK + XPENDING + XCLAIM) give you at-least-once delivery with per-consumer load balancing and failure recovery, so a stream is the right shape whenever you need a buffer that survives restarts, supports replay, and fans work out to a pool of workers. Unlike pub/sub (fire-and-forget, no history) and unlike a plain list (no acknowledgement, no groups), a stream remembers what has and has not been processed.

The stream primitives.

• XADD stream * field value ... appends an entry; the * tells Redis to assign the ID <millisecondsTime>-<sequence>, which is monotonically increasing and globally ordered. You can pass an explicit ID for idempotent producers.
• XLEN / XRANGE / XREVRANGE. XLEN is O(1) length; XRANGE stream - + reads all entries in ID order; ranges accept (id for exclusive bounds and COUNT for paging.
• XREAD BLOCK ms STREAMS stream $. A blocking tail read — $ means "only entries added after I started blocking". This is the simple single-consumer tail; it does not track acknowledgements.
• Trimming. XADD stream MAXLEN ~ 100000 * ... caps the stream to ~100k entries (the ~ allows efficient approximate trimming); XTRIM trims after the fact. Without trimming a stream grows forever.

Consumer groups — the reliability layer.

• XGROUP CREATE stream grp $ creates a consumer group starting at the current end ($) or from the beginning (0). The group tracks a last-delivered ID and a Pending Entries List (PEL) per consumer.
• XREADGROUP GROUP grp consumer COUNT n STREAMS stream > delivers new (>) entries to this consumer and records them as pending (delivered-but-not-acked). Two consumers in the same group split the entries — load balancing for free.
• XACK stream grp id marks an entry done and removes it from the PEL. An entry read but never acked stays pending forever — that is the at-least-once guarantee.
• XPENDING / XCLAIM / XAUTOCLAIM. XPENDING inspects stuck entries (which consumer, how long idle); XCLAIM/XAUTOCLAIM reassign entries idle longer than a threshold to a healthy consumer — this is how you recover work from a crashed worker.

Stream vs pub/sub vs list — the three-way interview question.

• Pub/sub (PUBLISH/SUBSCRIBE). Fire-and-forget broadcast. Zero history — a subscriber that was offline misses everything. Use it for live fan-out notifications where losing a message on disconnect is acceptable (cache invalidation broadcasts, live dashboards).
• List (LPUSH/BRPOP). A durable queue, but no consumer groups and no acknowledgement — once BRPOP pops an entry, if the worker crashes before finishing, the entry is gone. Fine for best-effort work queues where a retry mechanism lives elsewhere.
• Stream. Durable, ordered, replayable, with groups and acknowledgement. The right choice when you need "process every event at least once, recover from worker crashes, and be able to replay history". The cost is more moving parts (PEL management, trimming).

Common interview probes on streams.

• "Why a stream over a list for ingestion?" — acknowledgement + consumer groups + replay.
• "How do you recover a crashed consumer's in-flight work?" — XAUTOCLAIM idle entries to a live consumer.
• "How do you stop a stream growing forever?" — MAXLEN ~ capped XADD or a periodic XTRIM.
• "Stream vs Kafka?" — a stream is a single-node, single-partition-per-key log; Kafka is a distributed, partitioned, replicated log. Streams win for "I already run Redis and need a buffer"; Kafka wins for multi-consumer, multi-TB, cross-datacentre.

Worked example — ingesting events with XADD and a capped stream

Detailed explanation. The producer side of a stream is a single command. You append structured events (field/value pairs) and let Redis assign ordered IDs. The one thing you must not forget is MAXLEN — a stream without a cap grows until it eats all memory. Approximate trimming (MAXLEN ~ N) is nearly free because Redis only trims at listpack-node boundaries.

• Append. XADD events * type click user 42 ts 1725545000 — * assigns the ID.
• Cap. MAXLEN ~ 1000000 keeps roughly the last million entries.
• Ordering. IDs are ms-seq, monotonic, so XRANGE always returns time order.

Question. Write a producer that appends click events to a capped stream and returns the assigned entry ID.

Input.

Parameter	Value
Stream key	events:clicks
Fields	type, user, url, ts
Cap	MAXLEN ~ 1000000
ID assignment	* (server-assigned)

Code.

import redis, time
r = redis.Redis(decode_responses=True)

def emit_click(user_id: int, url: str) -> str:
    entry_id = r.xadd(
        "events:clicks",
        {"type": "click", "user": user_id, "url": url, "ts": int(time.time())},
        maxlen=1_000_000,     # cap the stream ...
        approximate=True,     # ... with cheap approximate trimming (the '~')
    )
    return entry_id   # e.g. "1725545000123-0"

# Inspect
def stream_stats():
    return {
        "length": r.xlen("events:clicks"),
        "first":  r.xrange("events:clicks", count=1),
        "last":   r.xrevrange("events:clicks", count=1),
    }

Step-by-step explanation.

1. XADD with * assigns the entry an ID of <ms>-<seq>. Two entries added in the same millisecond get sequence -0, -1, … so ordering is total even under bursts.
2. maxlen=1_000_000, approximate=True emits MAXLEN ~ 1000000. The ~ lets Redis trim whole macro-nodes rather than counting to an exact boundary — trimming becomes O(1) amortised instead of O(entries-removed).
3. The event is stored as field/value pairs, not a serialised blob, so consumers can read individual fields and downstream tooling (XRANGE) shows a readable structure.
4. XLEN is O(1) — Redis tracks the length — so monitoring stream depth is cheap. XRANGE ... COUNT 1 and XREVRANGE ... COUNT 1 cheaply fetch the oldest and newest entries for lag calculations.
5. The producer knows nothing about consumers. Any number of consumer groups can attach later and read from 0 (all history still in the stream) or $ (only new entries) — the producer path is unchanged.

Output.

Call	Result
emit_click(42, "/pricing")	"1725545000123-0"
emit_click(42, "/docs")	"1725545000481-0"
XLEN events:clicks	2 (then grows)
after 1.2M adds	length capped near 1000000

Rule of thumb. Every XADD in production carries a MAXLEN ~ cap. An uncapped stream is an unbounded memory leak; approximate trimming makes the cap essentially free.

Worked example — a consumer-group worker loop with XREADGROUP and XACK

Detailed explanation. The consumer side is where streams earn their keep. A consumer group gives each worker a slice of new entries, tracks what each worker has in flight (the PEL), and holds unacknowledged entries until you XACK. The canonical loop: read new entries with >, process, ack. Add a second phase that first drains anything already pending for this consumer (ID 0) so a restarted worker finishes its in-flight work before taking new work.

• Create the group once. XGROUP CREATE events:clicks loaders $ MKSTREAM.
• Read new work. XREADGROUP GROUP loaders w1 COUNT 100 BLOCK 5000 STREAMS events:clicks >.
• Ack on success. XACK events:clicks loaders <id> per processed entry.

Question. Implement a worker in the loaders group that processes click events in batches and acknowledges only after the sink write succeeds.

Input.

Parameter	Value
Stream	events:clicks
Group	loaders
Consumer	w1
Batch	COUNT 100, BLOCK 5000 ms

Code.

import redis
r = redis.Redis(decode_responses=True)

STREAM, GROUP, CONSUMER = "events:clicks", "loaders", "w1"

def ensure_group():
    try:
        r.xgroup_create(STREAM, GROUP, id="$", mkstream=True)
    except redis.ResponseError as e:
        if "BUSYGROUP" not in str(e):   # group already exists is fine
            raise

def run_worker():
    ensure_group()
    while True:
        # '>' = entries never delivered to any consumer in this group
        resp = r.xreadgroup(GROUP, CONSUMER, {STREAM: ">"},
                            count=100, block=5000)
        if not resp:
            continue                      # timed out; loop and block again
        _, entries = resp[0]
        for entry_id, fields in entries:
            try:
                write_to_sink(fields)     # e.g. buffer into a warehouse loader
                r.xack(STREAM, GROUP, entry_id)   # ack ONLY after success
            except Exception:
                # do NOT ack: entry stays in the PEL and will be reclaimed
                # by XAUTOCLAIM (see next example) or retried on restart
                log_failure(entry_id, fields)

Step-by-step explanation.

1. XGROUP CREATE ... $ MKSTREAM creates the group at the current end (only new entries) and creates the stream if it does not exist. Catching BUSYGROUP makes group creation idempotent across worker restarts.
2. XREADGROUP ... STREAMS events:clicks > delivers new entries to w1 and records them in w1's PEL. If a second worker w2 runs the same loop, Redis splits entries between them — automatic load balancing with no coordinator.
3. BLOCK 5000 makes the read wait up to 5 seconds for new entries instead of busy-looping, so an idle worker costs almost nothing.
4. The worker XACKs each entry only after write_to_sink succeeds. This is the at-least-once contract: if the process crashes between read and ack, the entry stays pending and is redelivered — never silently lost.
5. On an exception, the worker deliberately does not ack. The entry remains in the PEL, visible to XPENDING, and is later reclaimed by XAUTOCLAIM (next example) — so a bad sink write becomes a retry, not a data loss.

Output.

Phase	PEL for w1	Action
read 100 new	100 pending	process batch
sink ok for 98	2 pending	98 acked, 2 failed
next loop	2 pending + new	failures await claim/retry
crash + restart	2 still pending	redelivered via 0/autoclaim

Rule of thumb. Acknowledge with XACK only after the downstream write commits. An entry read but unacked is your safety net — the stream will redeliver it, which is exactly what at-least-once means.

Worked example — recovering a dead consumer with XAUTOCLAIM

Detailed explanation. When a worker crashes mid-batch, its in-flight entries sit in the PEL forever unless someone claims them. XAUTOCLAIM (Redis 6.2+) scans the group's pending entries, finds those idle longer than a threshold, and reassigns them to a healthy consumer in one command — the modern replacement for the older XPENDING + XCLAIM dance.

• Detect. Entries idle > min-idle-time belong to a stalled consumer.
• Reassign. XAUTOCLAIM stream grp new_consumer min_idle 0 transfers them.
• Process + ack. The rescuer processes and acks reclaimed entries like normal work.

Question. Add a recovery routine that reclaims entries idle for more than 60 seconds and re-processes them under a live consumer.

Input.

Parameter	Value
Min idle	60000 ms
Rescuer consumer	w1
Start cursor	0-0
Batch	COUNT 100

Code.

import redis
r = redis.Redis(decode_responses=True)
STREAM, GROUP, CONSUMER = "events:clicks", "loaders", "w1"

def reclaim_stalled(min_idle_ms: int = 60_000):
    cursor = "0-0"
    while True:
        # Claim entries idle > min_idle_ms for THIS consumer to finish
        cursor, claimed, _deleted = r.xautoclaim(
            STREAM, GROUP, CONSUMER,
            min_idle_time=min_idle_ms,
            start_id=cursor,
            count=100,
        )
        for entry_id, fields in claimed:
            try:
                write_to_sink(fields)
                r.xack(STREAM, GROUP, entry_id)
            except Exception:
                log_failure(entry_id, fields)  # leave pending; try again later
        if cursor == "0-0":     # cursor wraps to 0-0 when the scan completes
            break

def pending_overview():
    # summary: total pending, min/max id, per-consumer counts
    return r.xpending(STREAM, GROUP)

Step-by-step explanation.

1. XPENDING (summary form) tells you the group's health: how many entries are pending and which consumers hold them. A consumer with a growing pending count that never shrinks is a crashed or stuck worker.
2. XAUTOCLAIM ... min_idle_time=60000 transfers only entries that have been pending (unacked) for over 60 seconds — long enough to be confident the original consumer is not merely slow. This avoids stealing work from a healthy-but-busy worker.
3. The command returns a cursor; you loop, passing the cursor back, until it wraps to 0-0, meaning the whole PEL has been scanned. This pages through large backlogs without loading them all at once.
4. Reclaimed entries are processed and acked exactly like fresh entries. The delivery count for each entry increments, so you can route entries redelivered too many times (a "poison message") to a dead-letter stream.
5. The net effect: a crashed worker's in-flight work is automatically picked up by a survivor within the idle threshold, so the pipeline self-heals without operator intervention or lost events.

Output.

State	Pending (w2, crashed)	After XAUTOCLAIM by w1
before	40 entries, idle 90s	—
claim	—	40 moved to w1
process + ack	—	40 acked, PEL drains
poison (delivery > 5)	—	routed to events:clicks:dead

Rule of thumb. Run an XAUTOCLAIM sweep on a timer in every consumer-group deployment. Pending entries with a high delivery count are poison messages — dead-letter them instead of retrying forever.

Data engineering interview question on Redis Streams

A senior interviewer might ask: "You need a durable ingestion buffer between a high-throughput producer and a warehouse loader that occasionally goes down for maintenance. Events must not be lost while the loader is offline, multiple loader workers should share the load, and a crashed worker's in-flight events must be retried. Design this with Redis Streams — the produce path, the consumer group, failure recovery, trimming, and idempotency at the sink."

Solution Using a capped stream + consumer group + XAUTOCLAIM + idempotent sink

import redis, json
r = redis.Redis(decode_responses=True)
STREAM, GROUP = "ingest:events", "warehouse-loaders"

# --- producer ---
def produce(event: dict) -> str:
    return r.xadd(STREAM, {"body": json.dumps(event)},
                  maxlen=5_000_000, approximate=True)

# --- consumer worker ---
def worker(name: str):
    try:
        r.xgroup_create(STREAM, GROUP, id="0", mkstream=True)
    except redis.ResponseError:
        pass
    while True:
        # 1) finish this worker's own pending work first (crash recovery)
        _drain(name, "0")
        # 2) then take new work
        _drain(name, ">")
        # 3) reclaim entries abandoned by dead workers
        _reclaim(name)

def _drain(name: str, cursor: str):
    resp = r.xreadgroup(GROUP, name, {STREAM: cursor}, count=200, block=2000)
    if not resp:
        return
    for entry_id, fields in resp[0][1]:
        event = json.loads(fields["body"])
        if sink_idempotent(entry_id, event):   # dedupe by entry_id at the sink
            r.xack(STREAM, GROUP, entry_id)

def _reclaim(name: str):
    cursor = "0-0"
    while True:
        cursor, claimed, _ = r.xautoclaim(STREAM, GROUP, name,
                                          min_idle_time=60_000,
                                          start_id=cursor, count=200)
        for entry_id, fields in claimed:
            event = json.loads(fields["body"])
            if sink_idempotent(entry_id, event):
                r.xack(STREAM, GROUP, entry_id)
        if cursor == "0-0":
            break

Step-by-step trace.

Step	Input	Effect
producer XADD	{order:42} → stream, capped 5M	durable, ordered, id t-0
loader offline	producer keeps XADD-ing	events accumulate in stream (bounded by MAXLEN)
worker reads >	200 entries → PEL	delivered, not yet acked
sink write ok	sink_idempotent dedupes by id	XACK removes from PEL
worker crash	200 entries stuck in PEL, idle 60s	survivor XAUTOCLAIMs them
survivor re-processes	idempotent sink skips already-written	XACK; no double-load

After deployment, the producer never blocks on the loader — events land in the capped stream regardless of loader health. Multiple loader workers in one group split the backlog; when one crashes, its in-flight entries are reclaimed after 60 seconds and retried; and because the sink dedupes by the stream entry ID, at-least-once delivery is made effectively-once at the warehouse.

Output:

Metric	Value
Delivery guarantee	at-least-once (effectively-once with idempotent sink)
Loader-outage tolerance	until stream hits MAXLEN ~ 5M
Load balancing	automatic across group consumers
Crash recovery	XAUTOCLAIM after 60 s idle
Ordering	total, by stream ID

Why this works — concept by concept:

• Capped stream as the buffer — XADD ... MAXLEN ~ 5M decouples producer from consumer: the producer writes at full speed even while the loader is down, and approximate trimming bounds memory. The stream is the durable, replayable backlog a list or pub/sub could never be.
• Consumer group load balancing — XREADGROUP ... > hands each worker a disjoint slice of new entries with no external coordinator, so adding a worker linearly increases throughput.
• Pending-first drain (0 before >) — reading cursor 0 first makes a restarted worker finish its own in-flight entries before taking new work, closing the window where a crash could strand entries.
• XAUTOCLAIM recovery — entries idle past 60 seconds are reassigned to a live worker, so a crashed consumer's work is retried automatically; delivery-count tracking lets you dead-letter poison messages.
• Idempotent sink keyed by entry ID — because at-least-once can redeliver, the sink deduplicates on the immutable stream ID, upgrading the guarantee to effectively-once without distributed transactions.
• Cost — O(1) XADD, O(1) XACK, O(log N) internal indexing, memory bounded by MAXLEN. Compared to standing up Kafka for a buffer you could serve from the Redis you already run, this is dramatically less operational surface for the same at-least-once semantics.

Streaming
Topic — streaming
Streaming ingestion and consumer-group problems

Practice →

Database Topic — database Durable buffer and delivery-semantics problems

Practice →

---

3. Sorted sets — leaderboards, time-series windows, rate limiting

sorted sets are the Swiss-army structure — one score-ordered set solves leaderboards, time-series windows, and rate limiting

The mental model in one line: a sorted set stores unique members each tagged with a floating-point score and keeps them permanently ordered by that score, so ZADD inserts in O(log N), ZRANGE/ZREVRANGE read ranked slices, ZRANK finds a member's position, and ZRANGEBYSCORE/ZREMRANGEBYSCORE operate on score windows — and because the score can be anything (points for a leaderboard, a Unix timestamp for a time-series or rate-limit window, a priority for a queue), the same structure covers a startling range of data-engineering problems. When an interviewer asks for a leaderboard, a sliding-window rate limiter, or a "give me events in this time range" index, the answer is almost always a sorted set.

The sorted-set primitives.

• ZADD key score member inserts or updates a member's score in O(log N). ZADD ... GT/LT update only if the new score is greater/less; ZADD ... NX/XX insert-only / update-only.
• ZRANGE / ZREVRANGE read members by rank (position) — ZREVRANGE lb 0 9 WITHSCORES is "top 10 with scores". ZRANGEBYSCORE / ZRANGEBYLEX read by score / lexical window.
• ZRANK / ZREVRANK / ZSCORE. A member's position (ascending / descending) and its score — O(log N). This is how you show "you are rank 4,217".
• ZINCRBY key delta member atomically bumps a member's score — the leaderboard "add points" primitive.
• ZREMRANGEBYSCORE / ZREMRANGEBYRANK delete a score / rank window — the trimming primitive behind sliding windows and capped leaderboards. ZCARD counts members; ZCOUNT counts a score range.

The three canonical uses — one structure, three problems.

• Leaderboards. Member = player, score = points. ZINCRBY to award points, ZREVRANGE 0 N-1 WITHSCORES for the top N, ZREVRANK for a player's rank. O(log N) writes and O(log N + M) top-M reads at any scale.
• Time-series windows. Member = event ID, score = event timestamp (ms). ZADD to record, ZRANGEBYSCORE key from to to fetch a time window, ZREMRANGEBYSCORE key -inf (cutoff to expire old events. A per-entity sorted set becomes a queryable, self-pruning time index.
• Sliding-window rate limiting. Member = a unique request marker, score = request timestamp. Drop entries older than now - window, count what remains, allow if under the limit, record the new request — all atomically. This is the textbook precise rate limiter, and it is a sorted set.

Leaderboard subtleties interviewers probe.

• Tie-breaking. Equal scores are ordered lexicographically by member. If you need "earlier submission wins ties", encode a timestamp into the score's fractional part or into the member.
• Descending display. Redis stores ascending; use ZREVRANGE/ZREVRANK for high-score-first, or store negated scores if you frequently need ascending semantics.
• Bounded leaderboards. To keep only the top 10,000, periodically ZREMRANGEBYRANK key 0 -10001 (drop everyone below rank 10,000). This caps memory on a viral leaderboard.

Common interview probes on sorted sets.

• "How do you get a user's rank in O(log N)?" — ZREVRANK; never scan.
• "How do you build a sliding-window rate limiter?" — sorted set scored by timestamp + ZREMRANGEBYSCORE + ZCARD, atomically.
• "How do you fetch events between two times?" — score by timestamp, ZRANGEBYSCORE.
• "How do you cap a leaderboard's memory?" — ZREMRANGEBYRANK to drop low ranks.

Worked example — a top-N leaderboard with ZINCRBY and ZREVRANGE

Detailed explanation. A leaderboard is the poster child for sorted sets. Awarding points is ZINCRBY (atomic, no lost updates); reading the top N is ZREVRANGE ... WITHSCORES; showing a player their rank is ZREVRANK. Every operation is O(log N) or better, so a leaderboard with ten million players is as fast as one with ten.

• Award points. ZINCRBY game:lb 50 player:42.
• Top N. ZREVRANGE game:lb 0 9 WITHSCORES.
• My rank. ZREVRANK game:lb player:42 (0-based; add 1 for display).

Question. Implement award-points, top-10, and get-my-rank for a game leaderboard.

Input.

Parameter	Value
Key	game:lb
Member	player:{id}
Score	cumulative points
Top-N read	ZREVRANGE 0 9 WITHSCORES

Code.

import redis
r = redis.Redis(decode_responses=True)
LB = "game:lb"

def award(player_id: int, points: int) -> float:
    # Atomic: no read-modify-write race, returns the new total score
    return r.zincrby(LB, points, f"player:{player_id}")

def top_n(n: int = 10) -> list[tuple[str, float]]:
    # Highest score first, with scores
    return r.zrevrange(LB, 0, n - 1, withscores=True)

def my_rank(player_id: int) -> int | None:
    rank = r.zrevrank(LB, f"player:{player_id}")
    return None if rank is None else rank + 1   # 1-based for humans

def cap_leaderboard(keep_top: int = 10_000):
    # Drop everyone below the top `keep_top` to bound memory
    r.zremrangebyrank(LB, 0, -keep_top - 1)

Step-by-step explanation.

1. ZINCRBY adds points to a player's score, creating the member at that score if absent, and returns the new total — atomically, so two concurrent point awards never lose one.
2. ZREVRANGE LB 0 9 WITHSCORES returns the ten highest-scoring members with their scores in one O(log N + 10) call. The set is always sorted, so there is no sort step at read time.
3. ZREVRANK returns a player's 0-based position from the top in O(log N); adding 1 gives the human-facing rank. This is how "You are #4,217 of 2.3M" renders instantly.
4. cap_leaderboard uses ZREMRANGEBYRANK LB 0 -(keep_top+1) to delete every member below the top keep_top. On a viral leaderboard this bounds memory without affecting the visible ranks.
5. Because scores are floats, you can encode tie-breaks: score = points - timestamp*1e-13 makes an earlier achiever outrank a later one at equal points, since the tiny timestamp term breaks ties deterministically.

Output.

Call	Result
award(42, 50)	50.0
award(42, 30)	80.0
award(7, 100)	100.0
top_n(2)	[("player:7",100.0),("player:42",80.0)]
my_rank(42)	2

Rule of thumb. A leaderboard is a sorted set: ZINCRBY to score, ZREVRANGE WITHSCORES for the top, ZREVRANK for a position. Never fetch all members and sort in application code — that throws away the whole point of the structure.

Worked example — a precise sliding-window rate limiter

Detailed explanation. Fixed-window counters (an INCR per minute bucket) allow bursts at window edges — 100 requests at 11:59:59 and 100 more at 12:00:00 pass a "100/min" limit but are 200 requests in one second. A sliding-window limiter using a sorted set scored by timestamp is precise: it counts exactly the requests in the trailing window at every instant. The four steps — drop old, count, add new, expire — must run atomically, which a Lua script or a MULTI/EXEC transaction guarantees.

• Drop old. ZREMRANGEBYSCORE key -inf (now-window) removes requests outside the window.
• Count. ZCARD key is how many requests remain in the window.
• Admit + record. If under the limit, ZADD key now unique_id.

Question. Build a 100-requests-per-60-seconds per-user rate limiter that is precise at window edges and atomic under concurrency.

Input.

Parameter	Value
Key	rl:user:{id}
Limit	100
Window	60 s
Score	request timestamp (ms)

Code.

import redis, time, uuid
r = redis.Redis(decode_responses=True)

# Atomic sliding-window limiter as a single Lua script (runs server-side)
_LUA = """
local key    = KEYS[1]
local now    = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit  = tonumber(ARGV[3])
local member = ARGV[4]

-- 1) drop entries older than the window
redis.call('ZREMRANGEBYSCORE', key, '-inf', now - window)
-- 2) count what remains
local count = redis.call('ZCARD', key)
if count < limit then
    -- 3) admit: record this request
    redis.call('ZADD', key, now, member)
    redis.call('PEXPIRE', key, window)   -- self-clean idle keys
    return 1        -- allowed
else
    redis.call('PEXPIRE', key, window)
    return 0        -- rejected
end
"""
_check = r.register_script(_LUA)

def allow_request(user_id: int, limit: int = 100, window_ms: int = 60_000) -> bool:
    now = int(time.time() * 1000)
    member = f"{now}-{uuid.uuid4().hex[:8]}"   # unique so ZADD never collides
    return _check(keys=[f"rl:user:{user_id}"],
                  args=[now, window_ms, limit, member]) == 1

Step-by-step explanation.

1. The entire check runs as one Lua script, which Redis executes atomically — no other command interleaves between the "drop old", "count", and "add new" steps, so two concurrent requests can never both slip past the limit on a stale count.
2. Step 1, ZREMRANGEBYSCORE key -inf (now-window), evicts every request timestamped before the window start. After this line the set contains exactly the requests in the trailing 60 seconds.
3. Step 2, ZCARD, counts the survivors. Because old entries were just removed, this count is the precise current window occupancy — no edge-burst loophole.
4. Step 3 admits the request only if count < limit, recording it with ZADD key now member. The member embeds a UUID so two requests in the same millisecond are distinct members (a plain timestamp would collide and undercount).
5. PEXPIRE key window sets the key to expire one window after the last activity, so a user who stops making requests has their rate-limit key reclaimed automatically — no key accumulation for idle users.

Output.

Request	Window count before	Decision	Set size after
#1..#100 in 60s	0..99	allow	grows to 100
#101 at +30s	100	reject (0)	still 100
#102 at +61s	old dropped → 40	allow	41

Rule of thumb. For a precise rate limiter, use a sorted set scored by timestamp and run drop-count-add as one Lua script. Fixed-window INCR counters are cheaper but allow 2× bursts at window edges — know which precision your SLA needs.

Worked example — time-bucketed metrics with score = timestamp

Detailed explanation. A per-entity sorted set scored by event time is a lightweight time-series index: record each event at score = ts, query any time range with ZRANGEBYSCORE, and prune history with ZREMRANGEBYSCORE. This gives you "last 24 hours of events for device 7" without a time-series database, and the pruning keeps memory bounded.

• Record. ZADD ts:device:7 <ts> <event_id>.
• Query range. ZRANGEBYSCORE ts:device:7 <from> <to>.
• Prune. ZREMRANGEBYSCORE ts:device:7 -inf (now-24h).

Question. Store device telemetry events and answer "how many events in the last hour" and "give me events between t1 and t2", pruning anything older than 24 hours.

Input.

Parameter	Value
Key	ts:device:{id}
Score	event timestamp (ms)
Member	event id
Retention	24 h

Code.

import redis, time
r = redis.Redis(decode_responses=True)

def record_event(device_id: int, event_id: str, ts_ms: int | None = None):
    ts = ts_ms or int(time.time() * 1000)
    key = f"ts:device:{device_id}"
    pipe = r.pipeline()
    pipe.zadd(key, {event_id: ts})
    pipe.zremrangebyscore(key, "-inf", f"({ts - 24*3600*1000}")  # prune >24h
    pipe.execute()

def count_last_hour(device_id: int) -> int:
    now = int(time.time() * 1000)
    return r.zcount(f"ts:device:{device_id}", now - 3600*1000, now)

def events_between(device_id: int, t1: int, t2: int) -> list[str]:
    return r.zrangebyscore(f"ts:device:{device_id}", t1, t2)

Step-by-step explanation.

1. ZADD key {event_id: ts} records the event with its timestamp as the score, so the set is automatically ordered by time — no separate index needed.
2. The ZREMRANGEBYSCORE key -inf (now-24h on every write prunes events older than the retention horizon incrementally, so the per-device set stays bounded instead of growing forever.
3. count_last_hour uses ZCOUNT key (now-1h) now to count members whose score (timestamp) falls in the last hour — O(log N), no scan.
4. events_between uses ZRANGEBYSCORE key t1 t2 to return exactly the event IDs in an arbitrary time window, in time order. This is the "range query" a time-series store would give you, from a sorted set.
5. Because both the query and the prune operate on score windows, the structure doubles as a self-maintaining rolling buffer: hot recent data stays, cold data is dropped, and every read is a logarithmic range operation.

Output.

Operation	Result
record e1@t, e2@t+5m, e3@t+2h	set has 3 members
count_last_hour (now=t+2h5m)	1 (only e3 in last hour)
events_between(t, t+10m)	["e1","e2"]
prune (>24h old)	old members removed on write

Rule of thumb. Score a sorted set by timestamp to get a queryable, self-pruning time index for free. Prune on write with ZREMRANGEBYSCORE so the set never outgrows your retention window.

Data engineering interview question on sorted sets

A senior interviewer might ask: "Design a per-user API rate limiter that enforces exactly 1,000 requests per hour with no burst loophole at the hour boundary, works correctly under many concurrent requests to the same user, cleans up state for idle users automatically, and adds negligible load to the primary database. Walk me through the data structure, the atomicity, and the failure behaviour if Redis restarts."

Solution Using a sorted-set sliding window driven by one atomic Lua script

import redis, time, uuid
r = redis.Redis(decode_responses=True)

_LIMITER = """
local key    = KEYS[1]
local now    = tonumber(ARGV[1])
local window = tonumber(ARGV[2])   -- ms
local limit  = tonumber(ARGV[3])
local member = ARGV[4]

redis.call('ZREMRANGEBYSCORE', key, '-inf', now - window)
local used = redis.call('ZCARD', key)
if used < limit then
    redis.call('ZADD', key, now, member)
    redis.call('PEXPIRE', key, window)
    return {1, limit - used - 1}      -- allowed, remaining
else
    redis.call('PEXPIRE', key, window)
    return {0, 0}                     -- rejected, 0 remaining
end
"""
_run = r.register_script(_LIMITER)

def rate_limit(user_id: int, limit=1000, window_ms=3_600_000):
    now = int(time.time() * 1000)
    member = f"{now}:{uuid.uuid4().hex[:8]}"
    allowed, remaining = _run(keys=[f"rl:{user_id}"],
                              args=[now, window_ms, limit, member])
    return bool(allowed), int(remaining)

Step-by-step trace.

Step	Input	Effect
req at t=0	window empty	ZREM (noop) → ZCARD 0 → admit → size 1
999 more by t=+50m	count climbs 1→999	all admitted; size 1000
req #1001 at t=+55m	ZCARD 1000 ≥ limit	rejected, remaining 0
req at t=+61m	entries < t+1m dropped	window shrinks → admit again
concurrent reqs	Lua runs atomically	no two admit on a stale count
Redis restart (no AOF)	key lost	limiter resets to empty (fail-open)

After deployment, each user's rate state is one sorted set keyed rl:{user_id}. The Lua script makes the drop-count-admit sequence indivisible, so concurrency cannot over-admit; the sliding window has no edge-burst loophole because old entries are removed before every count; and PEXPIRE reclaims idle users' keys. The database is never touched on the limit-check path.

Output:

Metric	Value
Precision	exact sliding window (no edge burst)
Concurrency safety	atomic (single Lua script)
Idle cleanup	PEXPIRE after one window
DB load on check	zero
Restart behaviour	fail-open unless AOF persists state

Why this works — concept by concept:

• Sorted set scored by timestamp — each request is a member scored by its arrival time, so "requests in the last hour" is a score-range operation and pruning old requests is ZREMRANGEBYSCORE — the structure is the sliding window.
• Atomic Lua script — bundling remove-count-add into one server-side script makes it indivisible under Redis's single-threaded execution, so two simultaneous requests cannot both read a stale count and both be admitted past the limit.
• UUID-tagged members — embedding a UUID in the member guarantees uniqueness within the same millisecond, so ZADD never overwrites a concurrent request's entry and the count stays exact.
• PEXPIRE self-cleanup — expiring the key one window after the last request means idle users cost no memory, replacing a reaper job with a TTL.
• Cost — O(log N) per ZADD/ZREM where N is requests-in-window (bounded by the limit), so the check is effectively O(log limit) and constant memory per active user. Compared to a database-backed limiter with a row lock per check, this offloads the entire hot path to Redis at microsecond latency.

Data structures
Topic — data-structures
Sorted-set, ranking, and window problems

Practice →

Database Topic — database Rate-limiting and time-index design problems

Practice →

---

4. Probabilistic structures — HyperLogLog and Bloom filters

hyperloglog and Bloom filters trade exactness for constant memory — count billions of uniques in 12 KB, test membership in kilobytes

The mental model in one line: probabilistic data structures answer "how many distinct?" and "have I seen this before?" using a fixed, tiny amount of memory that does not grow with the data, at the cost of a small, bounded, tunable error — HyperLogLog estimates cardinality (unique count) in ~12 KB regardless of whether you feed it a thousand items or a billion, and a Bloom filter tests set membership in a few kilobytes with a controllable false-positive rate and zero false negatives. For data engineers these are the difference between "store two billion user IDs to count uniques" (impossible on one box) and "store a 12 KB sketch" (trivial).

HyperLogLog — cardinality in constant memory.

• PFADD key element [element ...] adds elements; duplicates are absorbed — adding the same user a million times counts as one. Returns 1 if the estimated cardinality changed.
• PFCOUNT key [key ...] returns the estimated number of distinct elements, with a standard error of about 0.81%. Given multiple keys, it returns the cardinality of their union without materialising it.
• PFMERGE dest src1 src2 ... merges several HLLs into one — the union sketch. This is how you roll daily unique-visitor HLLs into a monthly one without re-scanning events.
• The magic. A Redis HLL is capped at 12 KB and estimates cardinalities up to ~2^64. Memory is constant — it does not grow with the number of distinct items. That fixed cost is the whole point.

Bloom filters — membership in bounded memory.

• BF.ADD key item / BF.MADD add items (RedisBloom module). BF.EXISTS key item tests membership.
• The guarantee. A Bloom filter answers "definitely not in the set" or "probably in the set" — it has zero false negatives (if it says no, it means no) and a tunable false-positive rate (if it says yes, there is a small chance it is wrong). BF.RESERVE key error_rate capacity sizes it.
• Why it fits. A Bloom filter for 100 million items at a 1% false-positive rate is roughly 120 MB — versus storing 100M full keys. It shines as a pre-filter: "have we probably seen this event / URL / email before?" before an expensive exact check.

Accuracy vs memory — the engineering decision.

• When approximate is correct. Dashboards ("~2.1M daily uniques"), cache-miss guards, dedup pre-filters, and "roughly how big" analytics rarely need exactness. A 0.81% error on a unique-visitor chart is invisible.
• When it is not. Billing, compliance counts, and anything where a single miscount has legal or financial consequences need exact structures. Never bill customers from a HyperLogLog.
• The false-positive contract. A Bloom "yes" must be safe to be occasionally wrong. Using it as a cache-miss filter is fine (a false "seen" just triggers a redundant check); using it as the sole gate on "have we already charged this card" is not.

Common interview probes.

• "Count daily unique visitors across two billion events cheaply." — HyperLogLog, one per day, PFCOUNT.
• "How much memory does HLL use?" — ~12 KB, constant, ~0.81% error.
• "Roll daily uniques into monthly without double-counting." — PFMERGE the daily HLLs.
• "Difference between a Bloom filter and a set?" — Bloom is constant-ish memory with false positives and no deletes (standard Bloom); a set is exact but O(n) memory.

Worked example — daily unique visitors with HyperLogLog

Detailed explanation. Counting unique visitors exactly means storing every distinct visitor ID — billions of them. HyperLogLog replaces that with a 12 KB sketch per day: PFADD every visitor (duplicates are free), PFCOUNT for the estimate. The memory does not grow with traffic, so a site with a billion daily events uses the same 12 KB as one with a thousand.

• Add. PFADD uv:2026-09-05 user:42.
• Count. PFCOUNT uv:2026-09-05 → ~unique visitors that day.
• Union. PFCOUNT uv:2026-09-05 uv:2026-09-06 → uniques across both days, de-duplicated.

Question. Track daily unique visitors and report both a single day and a two-day de-duplicated total.

Input.

Parameter	Value
Key pattern	uv:{yyyy-mm-dd}
Add op	PFADD
Count op	PFCOUNT
Memory per day	≤ 12 KB

Code.

import redis
from datetime import date
r = redis.Redis(decode_responses=True)

def track_visit(user_id: int, day: str | None = None):
    key = f"uv:{day or date.today().isoformat()}"
    # PFADD absorbs duplicates: same user added twice counts once
    r.pfadd(key, f"user:{user_id}")

def uniques_for_day(day: str) -> int:
    return r.pfcount(f"uv:{day}")

def uniques_across(days: list[str]) -> int:
    # PFCOUNT over multiple keys = cardinality of the UNION (deduped)
    keys = [f"uv:{d}" for d in days]
    return r.pfcount(*keys)

Step-by-step explanation.

1. PFADD uv:{day} user:{id} hashes the visitor into the day's HLL sketch. Adding the same visitor again does not change the estimate — de-duplication is intrinsic, so you never store the raw IDs.
2. The sketch is capped at 12 KB no matter how many distinct visitors it has seen, so memory per day is constant and predictable — you can keep years of daily HLLs in a few megabytes.
3. PFCOUNT uv:{day} returns the estimated distinct count with ~0.81% standard error — for a dashboard reading "2,143,000 daily uniques", the true value is within a few thousand, which is invisible at that scale.
4. PFCOUNT key1 key2 ... computes the cardinality of the union of several sketches on the fly, so "uniques across Saturday and Sunday" correctly counts a visitor who came both days only once — something you cannot do by adding two daily counts.
5. Because the operations are add-and-count only, the ingestion path is a single O(1) PFADD per event — cheap enough to call on every request without touching the database.

Output.

Operation	Result
PFADD uv:2026-09-05 user:42 (×3)	counts as 1
2.1M distinct users added	sketch still ≤ 12 KB
PFCOUNT uv:2026-09-05	~2,143,118 (±~0.8%)
PFCOUNT uv:09-05 uv:09-06	union, deduped

Rule of thumb. For "how many distinct" at scale, use one HyperLogLog per bucket and PFCOUNT. Never store raw IDs to count uniques — a 12 KB sketch replaces gigabytes and the 0.81% error is invisible on any dashboard.

Worked example — rolling daily HLLs into weekly/monthly with PFMERGE

Detailed explanation. You want monthly unique visitors, but summing 30 daily counts double-counts anyone who visited on multiple days. PFMERGE combines the daily sketches into a union sketch whose PFCOUNT is the true monthly distinct count — no event re-scan, no double counting.

• Merge. PFMERGE uv:2026-09 uv:2026-09-01 ... uv:2026-09-30.
• Count the union. PFCOUNT uv:2026-09 → monthly uniques.
• Idempotent. Re-merging is safe; HLL union is associative and commutative.

Question. Produce a monthly unique-visitor count from the daily HLLs without re-reading raw events.

Input.

Parameter	Value
Source keys	uv:2026-09-01 .. uv:2026-09-30
Merge op	PFMERGE
Dest key	uv:2026-09
Count	PFCOUNT uv:2026-09

Code.

import redis
from datetime import date, timedelta
r = redis.Redis(decode_responses=True)

def month_days(year: int, month: int) -> list[str]:
    d = date(year, month, 1)
    out = []
    while d.month == month:
        out.append(d.isoformat())
        d += timedelta(days=1)
    return out

def rollup_month(year: int, month: int) -> int:
    dest = f"uv:{year}-{month:02d}"
    srcs = [f"uv:{d}" for d in month_days(year, month)]
    # Union all daily sketches into one monthly sketch
    r.pfmerge(dest, *srcs)
    return r.pfcount(dest)   # true monthly distinct count (deduped across days)

Step-by-step explanation.

1. PFMERGE dest src1 ... srcN computes the union of all source HLLs into dest. Because HLL union takes the max register value across sketches, a visitor present on ten days contributes to the union exactly once.
2. The result PFCOUNT uv:2026-09 is the genuine monthly distinct count — not the sum of daily counts, which would over-count multi-day visitors, often by a large factor for sticky products.
3. The rollup reads only the 30 daily sketches (≤ 12 KB each), never the raw event log, so a month's rollup is kilobytes of I/O regardless of how many billions of events occurred.
4. The operation is idempotent and order-independent: re-running the merge, or merging weeks first and then months, yields the same count because HLL union is associative and commutative.
5. You can build a hierarchy — daily → weekly → monthly → yearly — each level a PFMERGE of the level below, giving cheap drill-down uniques at every granularity from a few kilobytes of sketches.

Output.

Operation	Result
sum of 30 daily counts	over-counts multi-day visitors
PFMERGE uv:2026-09 <30 days>	union sketch, ≤ 12 KB
PFCOUNT uv:2026-09	true monthly uniques
weekly rollups	PFMERGE of 7 daily keys each

Rule of thumb. Never sum daily unique counts for a monthly total — that double-counts returning visitors. PFMERGE the daily sketches and PFCOUNT the union for the true distinct count.

Worked example — a Bloom filter as a dedup / cache-miss pre-filter

Detailed explanation. Before an expensive check ("is this the first time we have seen this event ID?" or "is this URL already crawled?"), a Bloom filter gives a cheap probabilistic pre-answer. If it says "definitely not seen", you skip the expensive lookup entirely; if it says "probably seen", you do the exact check. Because it has zero false negatives, a "no" is trustworthy and safe to act on.

• Reserve. BF.RESERVE seen 0.001 100000000 — 0.1% false positives, 100M capacity.
• Add + test. BF.ADD seen <id>, BF.EXISTS seen <id>.
• Contract. "no" is exact; "yes" is probably-yes (do the exact check).

Question. Use a Bloom filter to skip the expensive "already processed?" database lookup for events that are definitely new.

Input.

Parameter	Value
Filter key	seen:events
Error rate	0.001 (0.1%)
Capacity	100,000,000
Ops	BF.ADD, BF.EXISTS

Code.

import redis
r = redis.Redis(decode_responses=True)   # requires the RedisBloom module

def ensure_filter():
    try:
        r.execute_command("BF.RESERVE", "seen:events", 0.001, 100_000_000)
    except redis.ResponseError:
        pass   # already exists

def process_event(event_id: str, payload: dict) -> str:
    # 1) cheap probabilistic pre-check
    maybe_seen = r.execute_command("BF.EXISTS", "seen:events", event_id)
    if not maybe_seen:
        # definitely NEW (zero false negatives) -> skip the exact DB lookup
        handle_new_event(event_id, payload)
        r.execute_command("BF.ADD", "seen:events", event_id)
        return "processed-new"

    # 2) 'probably seen' -> confirm with the authoritative store
    if db_already_processed(event_id):
        return "skipped-duplicate"
    else:                       # rare false positive: was NOT actually seen
        handle_new_event(event_id, payload)
        r.execute_command("BF.ADD", "seen:events", event_id)
        return "processed-after-fp"

Step-by-step explanation.

1. BF.EXISTS is the fast path. When it returns false, the event is definitely new — Bloom filters never have false negatives — so we process it and skip the expensive database dedup lookup entirely.
2. The overwhelming majority of genuinely new events take this fast path, so the expensive exact check runs only for events the filter thinks it has probably seen — a small fraction of traffic plus the ~0.1% false-positive rate.
3. When BF.EXISTS returns true ("probably seen"), we fall back to the authoritative db_already_processed check because a Bloom "yes" can be wrong. This preserves correctness: the filter optimises the common case, the database guarantees the answer.
4. The rare false positive (filter says seen, database says new) is handled by processing the event and adding it — correctness is never sacrificed, only a redundant lookup is occasionally spent.
5. BF.RESERVE seen 0.001 100000000 sizes the filter for 100M items at 0.1% false positives — roughly 180 MB, versus storing 100M raw IDs. The pre-filter turns most dedup checks into a single in-memory bit test.

Output.

Event	BF.EXISTS	Path taken	Result
brand-new id	false	fast path	processed-new
true duplicate	true	DB confirms	skipped-duplicate
false positive (~0.1%)	true	DB says new	processed-after-fp
memory (100M @ 0.1%)	—	~180 MB	vs GBs of raw ids

Rule of thumb. Use a Bloom filter as a pre-filter in front of an exact store, never as the sole source of truth. Its zero-false-negative guarantee makes "no" trustworthy; treat every "yes" as "probably — go confirm".

Data engineering interview question on probabilistic structures

A senior interviewer might ask: "You need daily and monthly unique-visitor counts for a site doing two billion events a day. Storing raw visitor IDs is out of the question on memory grounds, dashboards tolerate a sub-1% error, and the monthly number must not double-count visitors who came on many days. Design the counting system, explain the memory footprint, and state exactly where an approximate answer is acceptable and where it is not."

Solution Using one HyperLogLog per day and PFMERGE rollups

import redis
from datetime import date, timedelta
r = redis.Redis(decode_responses=True)

# --- ingest: O(1) per event, constant memory per day ---
def on_event(visitor_id: str, day: str | None = None):
    key = f"uv:{day or date.today().isoformat()}"
    r.pfadd(key, visitor_id)          # duplicates absorbed; sketch <= 12 KB

# --- daily count ---
def daily(day: str) -> int:
    return r.pfcount(f"uv:{day}")     # ~0.81% standard error

# --- monthly rollup: union of daily sketches, deduped ---
def monthly(year: int, month: int) -> int:
    d, srcs = date(year, month, 1), []
    while d.month == month:
        srcs.append(f"uv:{d.isoformat()}")
        d += timedelta(days=1)
    dest = f"uv:{year}-{month:02d}"
    r.pfmerge(dest, *srcs)            # union across all days
    return r.pfcount(dest)            # true monthly uniques

Step-by-step trace.

Step	Input	Effect
2B events/day → PFADD	many dup visitor ids	each day's sketch stays ≤ 12 KB
daily("2026-09-05")	one sketch	PFCOUNT → ~uniques, ±0.81%
visitor on 10 days	10 daily sketches	PFMERGE counts them once
monthly(2026, 9)	30 daily sketches	union sketch → true monthly uniques
sum of daily counts (wrong)	30 numbers	over-counts returning visitors
billing needs exact	—	do NOT use HLL; use exact store

After deployment, ingestion is a single PFADD per event with constant per-day memory, so two billion events a day cost 12 KB of unique-count state per day. Dashboards read PFCOUNT for any day and PFMERGE+PFCOUNT for any week or month, always de-duplicating returning visitors. The only place the approximation is disallowed — billing or compliance counts — is served by a separate exact pipeline.

Output:

Metric	Value
Memory per day	≤ 12 KB (constant)
Ingest cost	O(1) PFADD per event
Count error	~0.81% standard error
Monthly dedup	correct via PFMERGE
Exactness boundary	dashboards yes; billing no

Why this works — concept by concept:

• HyperLogLog constant memory — an HLL estimates cardinality from the distribution of hash-value leading zeros, so it needs only ~12 KB of registers regardless of how many distinct items it has seen, replacing gigabytes of raw IDs with a fixed sketch.
• PFADD idempotence — adding the same visitor repeatedly leaves the estimate unchanged, so the ingest path can fire on every event without any application-side dedup.
• PFMERGE union for rollups — merging daily sketches takes the register-wise maximum, which is exactly set union, so a multi-day visitor is counted once and monthly totals are correct without re-scanning events.
• Bounded, known error — the ~0.81% standard error is quantified and stable, so you can state up front that dashboards tolerate it and billing does not — the accuracy budget is an explicit engineering decision, not an accident.
• Cost — O(1) time per add and per count, O(1) (12 KB) memory per bucket, O(days) per rollup merge. Compared to an exact COUNT(DISTINCT) over two billion rows — which needs to hold or sort the distinct set — this is a constant-memory, constant-time alternative that is wrong by less than a percent.

Data structures
Topic — data-structures
Probabilistic structure and cardinality problems

Practice →

Database Topic — database Approximate analytics and dedup design problems

Practice →

---

5. Persistence, patterns and pitfalls

Redis is durable if you configure it — RDB, AOF, eviction, and atomicity decide whether your data survives a restart

The one-sentence invariant: Redis holds data in memory, so its durability, memory bounds, and atomicity are configuration choices you must make deliberately — RDB snapshots and the append-only file (AOF) decide what survives a crash, maxmemory plus an eviction policy decide what happens when you run out of RAM, and MULTI/EXEC, WATCH, and Lua scripts decide whether multi-step operations are atomic — and the difference between a Redis you can trust as a durable buffer and one that silently loses data is entirely in these settings. Treating Redis as "just a cache" is fine until you use it as a stream buffer or a rate-limit store, at which point persistence and eviction stop being optional.

Persistence — RDB vs AOF.

• RDB (snapshotting). Point-in-time binary dumps written every N seconds / M changes (save 900 1). Compact, fast to load, great for backups — but a crash loses everything since the last snapshot (seconds to minutes of data). BGSAVE forks and writes in the background.
• AOF (append-only file). Logs every write command; on restart Redis replays the log. Durability is governed by appendfsync: always (fsync every write — safest, slowest), everysec (fsync once a second — the standard trade-off, ≤ 1 s loss), no (let the OS decide). AOF files are larger and rewrite-compacted periodically (BGREWRITEAOF).
• Hybrid (recommended default). aof-use-rdb-preamble yes writes an RDB snapshot as the AOF's base plus the command tail — fast loads and ≤ 1 s durability. Most production Redis runs AOF everysec with the RDB preamble.

Eviction — what happens at maxmemory.

• Set a bound. maxmemory 8gb caps memory; without it, Redis grows until the OS OOM-kills it. Always set maxmemory in production.
• Pick a policy. noeviction (reject writes when full — correct for a durable buffer you must not silently drop), allkeys-lru / allkeys-lfu (evict least-recently/frequently-used across all keys — correct for a pure cache), volatile-lru / volatile-ttl (evict only keys that have a TTL, preferring shortest TTL — correct when cache and durable data share one instance).
• The trap. Running a cache-style allkeys-lru on an instance that also holds your only copy of stream/rate-limit data means Redis will happily evict that data under pressure. Separate durable and cache workloads, or use volatile-* so only expendable (TTL'd) keys are evicted.

Atomicity — transactions, scripts, and pipelines.

• MULTI/EXEC. Queues commands and runs them as one atomic unit — no other client interleaves. But queued commands cannot see each other's results (no conditional logic mid-transaction).
• WATCH (optimistic locking). WATCH key before MULTI; if the key changes before EXEC, the transaction aborts — the check-and-set primitive for "increment only if unchanged".
• Lua scripts (EVAL). A script runs atomically server-side and can branch on intermediate results — the right tool when you need read-decide-write in one indivisible step (the rate limiter in section 3).
• Pipelining is not a transaction. A pipeline batches commands into one round trip for throughput, but they are not atomic — other clients' commands can interleave. Use pipelines for speed, MULTI/Lua for atomicity.

The pitfalls that page you at 3 AM.

• Big keys. A single 5 GB list or a hash with 50M fields makes every operation on it slow and blocks the single-threaded server; DEL of a huge key can stall Redis (use UNLINK for async free). Keep collections bounded.
• Hot keys. One key taking the majority of traffic can saturate a single core / shard. Shard hot counters across N sub-keys and sum on read.
• KEYS in production. KEYS pattern scans the entire keyspace and blocks the server — never run it in prod. Use SCAN (cursor-based, incremental) instead.
• TTL-less keys. Keys written without a TTL live forever and are the most common cause of slow memory creep. Default to a TTL unless the data is genuinely permanent.
• Cache stampede. When a hot cached key expires, thousands of requests miss simultaneously and hammer the database. Mitigate with a short lock ("one rebuilder"), a probabilistic early refresh, or staggered TTLs.

Common interview probes on operations.

• "RDB vs AOF — which and why?" — hybrid (AOF everysec + RDB preamble) for durability + fast load.
• "What eviction policy for a mixed cache + durable instance?" — volatile-* so only TTL'd keys go.
• "How do you make read-decide-write atomic?" — Lua script (or WATCH/MULTI).
• "Why is KEYS dangerous?" — O(N) blocking scan; use SCAN.

Worked example — choosing RDB vs AOF for a durable buffer

Detailed explanation. When Redis backs a stream ingestion buffer, a restart must not lose acknowledged-but-not-yet-loaded events. RDB alone loses everything since the last snapshot; AOF everysec caps loss at one second; the hybrid gives fast restart loads plus that one-second bound. This example shows the config and the reasoning.

• RDB only. Fast load, up to save-interval data loss on crash.
• AOF everysec. ≤ 1 s loss, slower load without a preamble.
• Hybrid. RDB preamble + AOF tail = fast load and ≤ 1 s loss.

Question. Configure a Redis instance used as a durable stream buffer for ≤ 1 second of data loss and fast restart.

Input.

Requirement	Choice
Max data loss on crash	≤ 1 s
Restart load speed	fast
Persistence mode	AOF everysec + RDB preamble
Eviction	noeviction (must not drop buffer)

Code.

# redis.conf — durable buffer profile

# --- AOF for ≤ 1s durability ---
appendonly            yes
appendfsync           everysec        # fsync once per second (≤ 1s loss)
aof-use-rdb-preamble  yes             # RDB snapshot as AOF base -> fast load

# --- RDB snapshots as backups (also the AOF preamble source) ---
save 900 1
save 300 100
save 60  10000

# --- memory: never silently drop the buffer ---
maxmemory             12gb
maxmemory-policy      noeviction      # reject writes when full, do NOT evict

# --- safety ---
rename-command        KEYS ""         # disable KEYS in production

Step-by-step explanation.

1. appendonly yes + appendfsync everysec turns on the append-only log with a one-second fsync cadence, capping worst-case data loss at the last second of writes — the standard durability/throughput trade-off.
2. aof-use-rdb-preamble yes makes Redis write a compact RDB snapshot as the base of the AOF file and append recent commands after it, so restart loads the snapshot fast and replays only the short command tail — you get AOF durability without slow full-log replay.
3. maxmemory-policy noeviction is the critical choice for a durable buffer: when memory is full, Redis rejects new writes with an error rather than silently evicting buffered events. The producer sees backpressure instead of data loss.
4. The save lines keep RDB snapshots as an independent backup channel and feed the AOF preamble. They are cheap insurance and enable point-in-time restores.
5. rename-command KEYS "" disables the KEYS command entirely, removing the single most common way an operator accidentally blocks the whole server during an incident.

Output.

Failure	RDB only	AOF everysec	Hybrid
crash	lose since last snapshot	lose ≤ 1 s	lose ≤ 1 s
restart load	fast	slow (replay log)	fast (preamble)
memory full	evict/OOM	evict/OOM	reject (noeviction)

Rule of thumb. For anything you cannot afford to lose on restart, run AOF everysec with the RDB preamble and maxmemory-policy noeviction. RDB-only is for caches and backups, not for buffers that hold your only copy of the data.

Worked example — cache-aside with stampede protection

Detailed explanation. The cache-aside pattern (read cache; on miss, load from DB and populate cache) has a classic failure: when a hot key expires, thousands of concurrent requests all miss and all hit the database at once — a stampede that can topple the DB. The fix is to let exactly one request rebuild the value while others briefly wait or serve stale, using a short SET NX lock.

• Read. GET key; hit → return.
• Miss with lock. SET lock:key 1 NX EX 5 — only one request wins the lock and rebuilds.
• Losers. Wait briefly and re-read, or serve the last-known value.

Question. Implement cache-aside with single-flight rebuild so a hot-key expiry cannot stampede the database.

Input.

Parameter	Value
Value key	cache:{id}
Lock key	lock:{id} (NX, EX 5)
Value TTL	300 s
Loser behaviour	short backoff + re-read

Code.

import redis, time, json
r = redis.Redis(decode_responses=True)

def get_with_stampede_guard(obj_id: str, ttl: int = 300):
    key, lock = f"cache:{obj_id}", f"lock:{obj_id}"

    cached = r.get(key)
    if cached is not None:
        return json.loads(cached)              # fast path: cache hit

    # Cache miss: try to become the single rebuilder
    if r.set(lock, "1", nx=True, ex=5):        # only ONE request wins
        try:
            value = load_from_db(obj_id)        # expensive
            r.set(key, json.dumps(value), ex=ttl)
            return value
        finally:
            r.delete(lock)
    else:
        # Lost the lock: another request is rebuilding. Back off + re-read.
        for _ in range(10):
            time.sleep(0.05)
            cached = r.get(key)
            if cached is not None:
                return json.loads(cached)
        return load_from_db(obj_id)             # last-resort fallback

Step-by-step explanation.

1. The fast path is a plain GET; on a hit it returns immediately with no locking overhead — the overwhelming majority of requests take this path.
2. On a miss, SET lock NX EX 5 atomically grants the rebuild lock to exactly one request. NX means "only if absent", so concurrent missers race and only the winner proceeds to the expensive load_from_db.
3. The winner loads from the database, repopulates the cache with a fresh TTL, and releases the lock in a finally so a crash mid-rebuild cannot leave the lock stuck (the EX 5 also auto-expires it as a backstop).
4. The losers do not hit the database. They back off in short sleeps and re-read the cache, picking up the value the winner just wrote — so N concurrent missers produce exactly one database load, not N.
5. The EX 5 lock TTL and the bounded retry loop guarantee liveness: even if the rebuilder dies, the lock frees within 5 seconds and a fallback load prevents an indefinite stall.

Output.

Scenario	DB loads	Behaviour
cache hit	0	immediate return
single miss	1	rebuild + populate
1000 concurrent misses	1	one rebuilds, 999 re-read
rebuilder crash	≤ 1 + fallback	lock expires in 5 s

Rule of thumb. Protect hot cached keys with a single-flight SET NX rebuild lock. Without it, a popular key's expiry turns into a synchronized thundering herd against your database.

Worked example — atomic read-decide-write with a Lua script

Detailed explanation. Some operations must read a value, decide based on it, and write — atomically. MULTI/EXEC cannot branch on intermediate results, and WATCH retries can starve under contention. A Lua script runs on the server as one indivisible unit and can branch, making it the cleanest tool for conditional atomic mutations like "decrement inventory only if stock remains".

• Read. GET stock:{sku} inside the script.
• Decide. Branch: enough stock or not.
• Write. DECRBY only on the success branch — all atomic.

Question. Implement "reserve K units of a SKU only if at least K are in stock" atomically.

Input.

Parameter	Value
Key	stock:{sku}
Reserve amount	K
Success	DECRBY and return remaining
Failure	return -1, no change

Code.

import redis
r = redis.Redis(decode_responses=True)

_RESERVE = """
local stock = tonumber(redis.call('GET', KEYS[1]) or '0')
local want  = tonumber(ARGV[1])
if stock >= want then
    return redis.call('DECRBY', KEYS[1], want)   -- reserve, return remaining
else
    return -1                                     -- insufficient stock
end
"""
_reserve = r.register_script(_RESERVE)

def reserve(sku: str, qty: int) -> int:
    # returns remaining stock, or -1 if not enough
    return int(_reserve(keys=[f"stock:{sku}"], args=[qty]))

Step-by-step explanation.

1. The whole read-decide-write runs inside one Lua script, which Redis executes atomically — no other client can change the stock between the GET and the DECRBY, eliminating the oversell race entirely.
2. The script reads the current stock, coercing a missing key to 0, then compares against the requested quantity. All logic runs server-side, so there is no network round trip between read and decide.
3. On the success branch, DECRBY reserves the units and returns the new remaining count in the same atomic step. Two concurrent reservations cannot both see the same "enough stock" and both decrement past zero.
4. On the failure branch, the script returns -1 and makes no change, so an over-request is a clean rejection, not a partial or negative decrement.
5. This is strictly stronger than WATCH/MULTI, which would abort-and-retry under contention (wasting round trips) and can livelock on a hot SKU; the Lua approach resolves each request in one server-side pass.

Output.

Stock before	reserve(sku, 3)	Stock after	Return
10	ok	7	7
2	insufficient	2	-1
concurrent (10, two ×6)	one wins	4 then -1	4, -1

Rule of thumb. For read-decide-write that must be atomic and conditional, use a Lua script — it is indivisible, branch-capable, and contention-proof, unlike MULTI/EXEC (no branching) or WATCH (retry storms under contention).

Data engineering interview question on persistence and patterns

A senior interviewer might ask: "You are running one Redis instance as both a cache and the durable buffer for a rate limiter and a stream ingestion pipeline. It has been OOM-killed twice and lost rate-limit state on the last restart. Harden it: choose the persistence mode, the memory bound and eviction policy that protects durable data while still evicting cache entries, make the write path atomic, and list the pitfalls you would audit for."

Solution Using AOF + noeviction on TTL-tagged caching + Lua-atomic writes

# redis.conf — mixed cache + durable-buffer instance

appendonly            yes
appendfsync           everysec         # ≤ 1s loss for durable buffer + limiter
aof-use-rdb-preamble  yes              # fast restart load

maxmemory             16gb
maxmemory-policy      volatile-lru     # evict ONLY keys that have a TTL (caches)
                                       # durable keys carry NO ttl -> never evicted

rename-command        KEYS   ""        # no accidental full-keyspace scans
rename-command        FLUSHALL ""      # no accidental wipe

import redis
r = redis.Redis(decode_responses=True)

# Cache entries ALWAYS carry a TTL -> eligible for volatile-lru eviction
def cache_put(key: str, value: str, ttl: int = 300):
    r.set(f"cache:{key}", value, ex=ttl)

# Durable buffer / limiter keys carry NO ttl (except sliding windows) ->
# volatile-lru will never evict them, so they survive memory pressure.
def buffer_append(stream: str, fields: dict):
    r.xadd(stream, fields, maxlen=5_000_000, approximate=True)

# Atomic write path via Lua (see rate limiter / reserve examples)

Step-by-step trace.

Concern	Setting	Result
data loss on crash	AOF everysec + preamble	≤ 1 s loss, fast reload
OOM kill	maxmemory 16gb	bounded; no OS OOM
protect durable keys	volatile-lru	only TTL'd cache keys evicted
cache keys	always SET ... EX	evictable under pressure
durable keys	no TTL	never eviction targets
atomic writes	Lua scripts	no read-modify-write races
operator safety	rename-command KEYS/FLUSHALL	can't block or wipe

After hardening, the instance survives restarts with at most one second of loss, never gets OOM-killed because maxmemory bounds it, and under memory pressure sheds only the disposable cache entries (which carry TTLs) while the TTL-less stream and limiter data are ineligible for eviction. The write paths that must be race-free run as Lua scripts, and the two most dangerous commands are disabled.

Output:

Metric	Before	After
Restart data loss	total (no persistence)	≤ 1 s
OOM kills	recurring	none (bounded)
Durable data under pressure	evicted	protected (no TTL)
Cache under pressure	—	evicted (volatile-lru)
Write-path races	possible	eliminated (Lua)

Why this works — concept by concept:

• AOF everysec + RDB preamble — the append-only log caps crash loss at one second while the RDB preamble makes restart loads fast, giving the durable buffer and rate limiter the persistence a cache never needed.
• volatile-lru with a TTL convention — tagging only cache entries with a TTL and leaving durable keys TTL-less makes eviction target exactly the expendable data; the policy physically cannot evict the stream or limiter state.
• maxmemory bound — capping memory turns "OS OOM-kills the process" (total loss) into "Redis manages its own eviction" (controlled, policy-driven), which is the difference between a graceful and a catastrophic memory event.
• Lua-atomic write paths — the limiter and inventory writes run as indivisible server-side scripts, so concurrency cannot corrupt counts even while the same instance serves cache traffic.
• Disabling KEYS and FLUSHALL — removing the two commands most likely to block or wipe the server during an incident closes the biggest operational footguns on a shared instance.
• Cost — AOF adds ~1 s fsync latency amortised and some disk I/O; volatile-lru adds negligible bookkeeping. In return you get restart durability, OOM safety, and eviction that respects the cache/durable boundary — O(1) on the hot path throughout.

Database
Topic — database
Persistence, eviction, and atomicity design problems

Practice →

Streaming
Topic — streaming
Durable buffer and backpressure problems

Practice →

---

Cheat sheet — Redis for data engineers recipes

• Structure-selection matrix. Counter → string + INCR. Object / row you update field-by-field → hash + HSET/HINCRBY. Last-N buffer or simple queue → list + LPUSH/LTRIM/BRPOP. De-dup / membership / tags → set + SADD/SISMEMBER. Ranked or time-windowed anything → sorted set + ZADD/ZRANGE. Durable replayable log with groups → stream + XADD/XREADGROUP. Unique count at scale → HyperLogLog + PFADD/PFCOUNT. Membership pre-filter → Bloom (BF.ADD/BF.EXISTS).
• Atomic counter template. INCR key (never GET+add+SET); bucket by time in the key name for windowed counts (views:{id}:{yyyymmddHH}); arm EXPIRE key ttl only on the first increment (if count == 1) for a fixed window, or on every access for a sliding one.
• Hash-per-object rule. Store a session / profile / feature row as one hash, not a serialised JSON string — HGET/HSET touch single fields in O(1), HINCRBY gives race-free in-row counters, and the object expires and replicates as a unit. Small rows stay in compact listpack encoding.
• Streams consumer-group loop. XGROUP CREATE stream grp $ MKSTREAM; workers XREADGROUP GROUP grp me COUNT n BLOCK ms STREAMS stream >; process; XACK stream grp id only after the sink commits; drain your own pending with cursor 0 first on restart; run XAUTOCLAIM ... min-idle 60000 on a timer to rescue crashed workers; always XADD ... MAXLEN ~ N to cap the stream.
• Sliding-window rate limiter (sorted set). One Lua script: ZREMRANGEBYSCORE key -inf (now-window) → ZCARD key → if under limit ZADD key now uuid → PEXPIRE key window. Precise at edges, atomic under concurrency, self-cleaning for idle users. Fixed-window INCR is cheaper but allows 2× edge bursts.
• Leaderboard recipe. ZINCRBY lb points member to score, ZREVRANGE lb 0 N-1 WITHSCORES for the top N, ZREVRANK lb member for a rank (O(log N)); cap memory with ZREMRANGEBYRANK lb 0 -(keep+1); break score ties by encoding a timestamp into the score's fractional part.
• Unique-count rollup (HLL). One HyperLogLog per bucket (uv:{day}), PFADD per event (duplicates free, ≤ 12 KB, ~0.81% error), PFCOUNT for a day, PFMERGE+PFCOUNT to roll days into weeks/months without double-counting. Never sum daily counts; never bill from an HLL.
• Bloom pre-filter contract. BF.RESERVE key error_rate capacity; BF.EXISTS "no" is exact (zero false negatives) → act on it; "yes" is probably-yes → confirm against the authoritative store. Use as a cheap gate in front of an expensive dedup lookup, never as the source of truth.
• Persistence default. AOF appendfsync everysec + aof-use-rdb-preamble yes = ≤ 1 s crash loss and fast restart loads. RDB-only is for caches and backups. For a buffer you must not drop, set maxmemory-policy noeviction so a full instance rejects writes instead of silently evicting.
• Eviction for mixed instances. Set maxmemory always. On an instance holding both cache and durable data, use volatile-lru/volatile-ttl and give only cache keys a TTL — durable keys stay TTL-less and are never eviction targets. Pure caches use allkeys-lru/allkeys-lfu.
• Atomicity rule. Pipelining = throughput, not atomicity (commands can interleave). MULTI/EXEC = atomic but no branching. WATCH = optimistic CAS (retry storms under contention). Lua EVAL = atomic and branch-capable — the tool for conditional read-decide-write (rate limits, inventory reserve).
• Pitfall audit. No big keys (bound every collection; UNLINK not DEL for huge keys); shard hot keys across sub-keys; SCAN never KEYS in prod (rename-command KEYS ""); default a TTL on every key unless truly permanent; guard hot cached keys with a single-flight SET NX rebuild lock to prevent cache stampedes.

Frequently asked questions

Is Redis just a cache?

No — Redis is an in-memory data-structure server that happens to be excellent at caching. Beyond GET/SET strings, it ships hashes (object rows), lists (queues, capped buffers), sets (membership, dedup), sorted sets (leaderboards, time windows, rate limiting), streams (durable replayable logs with consumer groups), and probabilistic sketches (HyperLogLog for cardinality, Bloom filters for membership). For data engineers, the same Redis you use as a cache is often the cheapest way to build a rate limiter, a unique-visitor counter, or a durable ingestion buffer — the caching use case is just the most visible one.

Redis Streams vs Kafka — when do I pick each?

Pick Redis Streams when you already run Redis and need a durable, ordered, replayable buffer with consumer groups and at-least-once delivery — it is dramatically less operational surface than a Kafka cluster, with XADD/XREADGROUP/XACK/XAUTOCLAIM covering produce, group fan-out, acknowledgement, and crash recovery. Pick Kafka when you need a distributed, partitioned, replicated log across many brokers and datacentres, multi-terabyte retention, a large ecosystem of connectors, or hundreds of independent consumer groups at very high throughput. A Redis Stream is a single-node log (bounded by one machine's memory and MAXLEN); Kafka is a horizontally-scaled commit log. For "buffer between a producer and a warehouse loader on infrastructure I already have", streams usually win.

When should I use a sorted set instead of a plain set or list?

Use a sorted set whenever you need order by a numeric key — because the score can be anything, one structure covers three big use cases: points for a leaderboard (ZREVRANGE, ZREVRANK), a timestamp for a time-series window (ZRANGEBYSCORE, ZREMRANGEBYSCORE), and a timestamp for a sliding-window rate limiter (drop old, count, admit — atomically in Lua). If you only need membership with no order, a plain set is cheaper; if you need insertion order with no ranking or windowing, a list is simpler. Reach for the sorted set the moment "ranked", "top N", "in this time range", or "how many in the last X" enters the requirements.

How much memory does HyperLogLog use, and how accurate is it?

A Redis HyperLogLog is capped at about 12 KB and estimates cardinalities up to ~2^64 with a standard error of roughly 0.81% — and crucially that memory is constant: it does not grow whether you add a thousand distinct items or a billion. That makes it the right tool for unique counts at scale (daily/monthly unique visitors over billions of events) where storing raw IDs is infeasible. PFMERGE unions daily sketches into weekly/monthly ones without double-counting returning visitors. The catch is the ~0.8% error: perfect for dashboards and analytics, unacceptable for billing, compliance, or any count with legal or financial consequences — those need an exact structure.

RDB vs AOF — which persistence should I run?

Run the hybrid: AOF with appendfsync everysec plus aof-use-rdb-preamble yes. That caps crash data loss at about one second (AOF fsyncs writes once per second) while keeping restarts fast (the RDB preamble loads a compact snapshot, then a short command tail replays). RDB-only snapshotting is compact and fast to load but loses everything since the last snapshot on a crash — fine for a pure cache or as a backup channel, not for a durable buffer. AOF always fsyncs every write for maximum durability at a large throughput cost — reserve it for data where even one second of loss is unacceptable. For anything you cannot rebuild on restart, also set maxmemory-policy noeviction so a full instance rejects writes instead of silently dropping your data.

How do I make a multi-step Redis operation atomic?

Use a Lua script (EVAL) when the operation must read, decide, and write in one indivisible step — the script runs server-side in Redis's single-threaded loop, so nothing interleaves, and unlike MULTI/EXEC it can branch on intermediate values (the sliding-window rate limiter and the "reserve stock only if available" pattern both need this). Use MULTI/EXEC for a fixed batch of commands that must all apply atomically but need no conditional logic. Use WATCH + MULTI for optimistic check-and-set ("update only if unchanged"), accepting that it retries under contention. Do not rely on pipelining for atomicity — a pipeline only batches commands into one round trip for throughput; other clients' commands can still interleave between them.

Practice on PipeCode

• Drill the data-structures practice library → for the sorted-set ranking, sliding-window, HyperLogLog, and structure-selection problems senior interviewers love.
• Rehearse on the database practice library → for key-value modelling, persistence and eviction trade-offs, atomicity, and rate-limiter design.
• Sharpen the ingestion axis with the streaming practice library → for Redis Streams consumer groups, at-least-once delivery, and durable-buffer scenarios.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the structure-selection matrix against real graded inputs.

Lock in Redis-beyond-caching muscle memory

Docs list the commands. PipeCode drills make you pick the structure under pressure — when a sorted set beats a fixed-window counter, when a stream beats a list, when a HyperLogLog replaces a billion stored IDs, when AOF and noeviction are the difference between a durable buffer and silent data loss. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice data-structure problems →
Practice streaming problems →