Redis Streams for Real-Time Pipelines: Consumer Groups, XADD/XREAD, Persistence

redis streams is the sub-millisecond append-only log that quietly took over "the Kafka lite tier" of 2026 real-time pipelines — small enough that a two-node deployment fits on a laptop, fast enough that clickstream and cache-warm workloads hit p99 under a millisecond, and just structured enough that consumer-group semantics work exactly the same way as Kafka's. The one-sentence architectural claim: a Redis Stream is an ordered, append-only sequence of entries keyed by monotonically-increasing millisecond IDs, with server-side fan-out to consumer groups, per-entry acknowledgement, and pending-entry-list bookkeeping — all delivered by a single-threaded C runtime that most teams already run for caching. The engineering trade-off lives in when Streams beat Kafka and how you tune persistence, not in whether Streams are "real" streaming.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "walk me through xadd versus xread versus xreadgroup" or "what does a healthy redis consumer group look like when one consumer dies mid-batch?" or "how do you reason about redis streams persistence when appendonly=yes and fsync=everysec are the pragmatic defaults?" It walks through why sub-ms real-time and a boring ops story make Streams the default choice for single-region ingest, the xadd/xread/xreadgroup triad and the MAXLEN ~ trim policy that caps memory, the consumer-group state machine (xack, xpending, XCLAIM, XAUTOCLAIM) that recovers pending entries when a consumer dies, the AOF vs RDB persistence model and the redis appendonly fsync policies you actually pick, and the redis stream vs kafka decision framework plus cluster-mode hash-slot layout that senior engineers ship into production. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library →, rehearse on the ETL practice library →, and sharpen the tuning axis with the optimization practice library →.

---

On this page

• Why Redis Streams became the Kafka Lite of 2026
• XADD + XREAD + XREADGROUP
• Consumer groups + XACK + XPENDING
• Persistence — AOF, RDB, streams-in-memory
• Redis Streams vs Kafka + production patterns
• Cheat sheet — Redis Streams recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Redis Streams became the Kafka Lite of 2026

Sub-millisecond append plus a boring ops story — that is why Streams beat "just add Kafka" for single-region ingest

The one-sentence invariant: a Redis Stream is an ordered, in-memory append-only log with server-side fan-out to consumer groups, keyed by monotonically-increasing millisecond IDs (<ms>-<seq>), where every entry is a small hash of field/value pairs and every append and every read is answered by a single-threaded C runtime with no ZooKeeper, no controller quorum, and no partition rebalance dance. That is a very different physical model from Kafka's replicated, disk-backed, per-partition log — and it changes the operational cost, the latency floor, and the failure modes in ways senior interviewers actively probe. The 2026 rule of thumb is boring: if the workload is single-region and under about 200k events per second per stream, Streams beats Kafka on ops cost by a wide margin; above that scale or across regions, Kafka wins.

The four axes interviewers actually probe.

• xadd / xread / xreadgroup semantics. Every interview starts here — what does XADD do, how does XREAD differ from XREADGROUP, when do you use $ versus > as the read ID, and how does the millisecond-plus-sequence entry ID guarantee monotonicity even under high append rates. Knowing the three commands cold is the senior signal.
• Consumer-group state — xack, xpending, XCLAIM, XAUTOCLAIM. A consumer group is a client-visible state machine, not an opaque abstraction. The senior answer names the pending-entry list (PEL), explains how xack clears it, and walks through the XCLAIM/XAUTOCLAIM handoff when a consumer dies mid-batch.
• redis streams persistence. The Streams data lives in memory; durability is provided by AOF (append-only file) with everysec / always / no fsync policies, plus periodic RDB snapshots. The senior answer names the trade-off — RAM budget vs durability window vs write amplification — and picks appendonly yes + appendfsync everysec as the pragmatic default.
• redis stream vs kafka. The comparison question every interviewer asks. Streams for single-region, sub-ms, small team; Kafka for multi-region, exactly-once semantics, ecosystem depth, high-throughput. The senior answer maps the decision to a four-row matrix (latency, durability, throughput, ops) rather than the vague "Streams is simpler."

Why 2026 pipelines default to Streams for single-region ingest.

• Latency floor. Redis is a single-threaded event loop over epoll/kqueue on Linux; a co-located producer sees XADD round-trip in 60–200 microseconds on a warm socket. Kafka's acks=all producer sees 1–5 ms on the same wire because of the replicated write path. For sub-100 ms end-to-end SLOs, Streams removes a full millisecond of headroom.
• Ops surface. A production Redis Streams deployment is one process (or a small cluster of shard-owner processes). Kafka is a Kafka broker cluster plus KRaft controllers (or ZooKeeper), plus Schema Registry, plus Kafka Connect if you want connectors, plus rebalance-aware consumers. Small teams ship Streams and never see 3 AM controller-election pages.
• Team familiarity. Every backend team already runs Redis for caching. Adding a Stream is a XADD call from the same client library and one config line for AOF. Adding Kafka is a whole platform team.
• Sizing. Streams are memory-bound — the practical cap is roughly MAXLEN ~ 10_000_000 entries per stream for a stream of ~200-byte entries on a 16 GB Redis instance. That is plenty for clickstream, cache-warm, and background-job workloads. Above that, Kafka's disk-backed logs win.

What XADD actually does — the primitive senior interviewers probe first.

• Command shape. XADD <stream-key> [NOMKSTREAM] [MAXLEN|MINID [~|=] <threshold> [LIMIT <count>]] <id>|* <field> <value> [<field> <value> ...].
• ID generation. The * argument tells Redis to generate the ID as <current-ms>-<seq> where <seq> counts appends inside the same millisecond. IDs are strictly monotonically increasing per stream, even across clients.
• NOMKSTREAM. If the stream key does not exist, XADD normally creates it. NOMKSTREAM opts out — the append fails with nil if the key does not exist. Useful when you want producers to fail fast if a consumer has never registered against the stream.
• MAXLEN trim. MAXLEN ~ 10000 caps the stream at approximately 10,000 entries (the ~ accepts slack for radix-tree page efficiency). MAXLEN = 10000 is exact but slower. Almost every production stream uses ~.
• Return. The generated entry ID (as a string).

What XREAD does — the read-by-cursor primitive.

• Command shape. XREAD [COUNT <n>] [BLOCK <ms>] STREAMS <stream1> <stream2> ... <id1> <id2> ....
• The cursor. The client tracks the last ID it consumed and passes it as the "start after" ID. Pass $ for "only new entries after the current end" (typically used in the first BLOCK call).
• Non-blocking vs blocking. Without BLOCK, XREAD returns immediately with whatever entries are already past the cursor. With BLOCK <ms>, XREAD suspends the connection until a new entry lands or the timeout fires — the primitive that lets consumers wait sub-ms for the next event.
• No consumer-group state. XREAD is stateless on the server side. It has no idea whether the client "successfully processed" the entry. That is what XREADGROUP + XACK are for.

What XREADGROUP adds — the consumer-group-aware primitive.

• Command shape. XREADGROUP GROUP <group> <consumer> [COUNT <n>] [BLOCK <ms>] [NOACK] STREAMS <stream> ... <id> ....
• The special IDs. > means "only entries never delivered to any consumer in this group" — the default read cursor for live consumption. Any explicit ID means "re-read entries in my pending list starting from that ID" — used for recovery.
• Delivery guarantee. Each entry is delivered to exactly one consumer in the group. The consumer must XACK to remove the entry from the group's pending-entry list (PEL).
• PEL. The pending-entry list is the group's server-side bookkeeping: (entry-id, consumer, delivery-count, last-delivery-time). XPENDING reads the PEL; XCLAIM reassigns an entry from one consumer to another.

What interviewers listen for.

• Do you say "Redis Streams is an ordered, in-memory, append-only log with server-side fan-out via consumer groups" in the first sentence? — senior signal.
• Do you name XADD * MAXLEN ~ 10000 as the canonical producer shape without prompting? — senior signal.
• Do you distinguish > (never-delivered) versus $ (only-new) versus explicit-ID (my-pending) on the read side? — required answer.
• Do you push back on "just use Kafka for streaming" with the latency + ops + team-size argument? — required answer.

Worked example — sizing a Redis Stream for a 50k events/sec clickstream

Detailed explanation. A textbook sizing problem: a marketing site emits 50,000 clickstream events per second, average payload 180 bytes, and the downstream analytics warehouse consumes with a 60-second window. The team wants to pick between Streams and Kafka. Walk through the memory math, the MAXLEN cap, and the failure mode if MAXLEN is missing.

• Event rate. 50,000 events per second.
• Payload. 180 bytes per event.
• Retention window. 60 seconds (analytics consumer lag budget).
• Naive expectation. 50,000 × 60 × 180 = 540 MB of stream data at steady state.

Question. Compute the MAXLEN needed to bound retention at 60 seconds, the RAM footprint of the stream, and the failure mode if MAXLEN is set to 0 (no trim).

Input.

Parameter	Value
Event rate	50,000 events/sec
Avg payload	180 bytes
Retention window	60 sec
MAXLEN threshold	3,000,000 entries
Redis instance RAM	16 GB

Code.

# Producer — XADD with MAXLEN ~ trim
redis-cli XADD clickstream \
  MAXLEN "~" 3000000 \
  "*" \
  user_id "u_42" \
  page "/pricing" \
  ts "1720214400123" \
  session "s_abc"

# Verify stream length + memory footprint
redis-cli XLEN clickstream
redis-cli MEMORY USAGE clickstream

# Producer — batch XADD via pipeline for throughput
import redis
r = redis.Redis(host="cache-1.internal", decode_responses=True)

def emit_batch(events):
    with r.pipeline(transaction=False) as pipe:
        for ev in events:
            pipe.xadd(
                "clickstream",
                fields=ev,
                id="*",
                maxlen=3_000_000,
                approximate=True,   # the ~ modifier
            )
        pipe.execute()

# Emit 1000 events per pipeline flush → 50 flushes/sec at 50k events/sec
emit_batch([
    {"user_id": "u_42", "page": "/pricing",   "ts": "1720214400123"},
    {"user_id": "u_43", "page": "/checkout",  "ts": "1720214400124"},
    # ...
])

Step-by-step explanation.

1. The retention math: 50,000 events/sec × 60 sec = 3,000,000 entries. That is the MAXLEN target — the stream keeps roughly the last 60 seconds of data, sized in entry count rather than time (Streams do not natively expose a time-based retention knob).
2. The memory math: 3,000,000 entries × ~250 bytes overhead-per-entry (payload + Redis's radix-tree overhead) ≈ 750 MB. That fits comfortably on a 16 GB Redis instance with plenty of headroom for other keys and OS cache.
3. The ~ modifier on MAXLEN tells Redis to trim to approximately 3,000,000 — the actual length may be a few thousand entries higher because Redis trims by radix-tree page. This is 3–10× faster than the exact = variant and matters at 50k appends/sec.
4. The pipeline pattern batches 1000 XADDs per round-trip. Without pipelining, 50,000 XADDs/sec would need 50,000 round-trips/sec — feasible over Unix socket, painful over the network. With a 1000-batch pipeline, the effective network cost is 50 round-trips/sec.
5. The failure mode without MAXLEN: the stream grows unboundedly. At 50,000 events/sec × 180 bytes = 9 MB/sec = 32 GB/hour. Within 30 minutes the Redis instance runs out of memory and the OS OOM-kills the process. MAXLEN ~ is not optional; it is the memory safety belt.

Output.

Setting	Stream length	Memory footprint	Outcome
No MAXLEN	unbounded	grows at 9 MB/sec	OOM in ~30 min
MAXLEN ~ 3,000,000	~3.0–3.02 M	~750 MB	Stable, 60s window
MAXLEN = 3,000,000	exactly 3.0 M	~750 MB	Stable but 3× slower trim

Rule of thumb. Every production redis streams producer sets MAXLEN ~ <target> on every XADD call. Compute the target as retention_seconds × events_per_second; leave 20% headroom for burst; verify with MEMORY USAGE <stream-key> after the first hour of traffic.

Worked example — the four axes on a whiteboard

Detailed explanation. A senior interviewer draws four boxes and asks you to fill them in for Streams: XADD/XREAD, consumer groups, persistence, dead-letter. This is a compression exercise — you have thirty seconds per box to prove you can operate the primitive without the docs open. Walk through each box.

• Box 1: XADD/XREAD. Append and read cursor.
• Box 2: Consumer groups. Fan-out to N workers with exactly-one-of-group delivery + PEL.
• Box 3: Persistence. In-memory + AOF + RDB.
• Box 4: Dead-letter. Secondary stream via XCLAIM on delivery-count > threshold.

Question. Fill in the four boxes in bullet form, with the one-line answer per axis and the config knob or command that ships it.

Input.

Axis	What interviewer wants
XADD/XREAD	Command shape + ID model
Consumer groups	Delivery semantics + PEL
Persistence	AOF/RDB/fsync
Dead-letter	Poison-pill escape hatch

Code.

# Box 1 — XADD + XREAD
redis-cli XADD events "*" MAXLEN "~" 1000000 user "u_42" event "click"
redis-cli XREAD COUNT 100 BLOCK 5000 STREAMS events "$"

# Box 2 — consumer group + XREADGROUP
redis-cli XGROUP CREATE events grp1 "$" MKSTREAM
redis-cli XREADGROUP GROUP grp1 consumer-a COUNT 10 BLOCK 5000 STREAMS events ">"
redis-cli XACK events grp1 1720214400123-0

# Box 3 — persistence
# redis.conf
# appendonly yes
# appendfsync everysec
# save 3600 1 300 100 60 10000     # RDB snapshot triggers

# Box 4 — dead-letter via XCLAIM on delivery-count
redis-cli XPENDING events grp1 IDLE 30000 - + 100 consumer-a
redis-cli XCLAIM events grp1 dlq-worker 30000 1720214400123-0
redis-cli XADD events:dlq "*" original_id 1720214400123-0 reason "poison"

Step-by-step explanation.

1. XADD takes the stream key, an ID (* = auto-generate as <ms>-<seq>), an optional MAXLEN ~ trim threshold, and the field/value pairs. XREAD takes the stream key and a start-after ID; $ means "only entries created after this call registers"; BLOCK <ms> suspends the connection waiting for new data.
2. XGROUP CREATE registers a consumer group; MKSTREAM creates the stream if missing. XREADGROUP reads on behalf of a named consumer inside the group; > means "never-delivered to any consumer in this group." XACK removes the entry from the group's PEL after processing.
3. Persistence ships via appendonly yes (AOF turned on) and appendfsync everysec (fsync the AOF once per second — the pragmatic default). The save line drives periodic RDB snapshots for point-in-time recovery.
4. Dead-letter. Poll XPENDING for entries idle > 30 seconds; use XCLAIM to reassign them to a DLQ worker; the worker XADDs to a secondary stream events:dlq with the failure metadata and XACKs the original. This is the standard poison-pill pattern.
5. Each box has one command family + one config line — that is what makes Streams operationally cheap. Kafka's equivalent whiteboard would need Producer + Consumer + Broker + Controller + Topic + Partition + Retention + Compaction lines.

Output.

Box	Command	Config knob
XADD/XREAD	XADD ... MAXLEN ~ ... / XREAD BLOCK	none — client-side
Consumer groups	XGROUP CREATE / XREADGROUP / XACK	stream-node-max-entries
Persistence	(none — server-side)	appendonly yes, appendfsync everysec, save
Dead-letter	XPENDING IDLE / XCLAIM / XADD :dlq	none — client-side pattern

Rule of thumb. If you can fill in these four boxes cold, you can pass any Redis-Streams-vs-Kafka interview round. Everything else is elaboration on these four primitives.

Worked example — Redis Streams versus Redis Pub/Sub for the same click event

Detailed explanation. A common source of confusion — Redis has two messaging primitives: Streams and Pub/Sub. They look similar at the API level but have opposite semantics on persistence, delivery guarantees, and consumer state. Show the difference with the same click event pushed through each, and explain why Streams beats Pub/Sub for every real-time pipeline in 2026.

• Pub/Sub. Fire-and-forget broadcast. Every currently-subscribed client sees the message; nothing is stored; nothing is acknowledged.
• Streams. Persistent append-only log. New consumers can replay from any historical ID; consumer groups track PEL; entries survive restarts if AOF/RDB is on.
• The pipeline reality. Pub/Sub is fine for cache-invalidation broadcasts and websocket fan-out to connected clients; for anything you cannot lose (clickstream, order events, payment webhooks), you need Streams.

Question. Push the same click event through Pub/Sub and Streams. Show what happens when the consumer restarts mid-flight. Explain why Streams is the correct primitive for ingest pipelines.

Input.

Attribute	Pub/Sub	Streams
Persistence	none (in-flight only)	in-memory + AOF/RDB
Delivery	broadcast to subscribers	consumer group: one-of-N
Replay	not possible	XREADGROUP with explicit ID
Consumer restart	events during downtime are lost	PEL survives; consumer resumes

Code.

# Pub/Sub — fire and forget
redis-cli PUBLISH clicks '{"user":"u_42","page":"/pricing"}'
# Subscriber (must be connected BEFORE the publish):
redis-cli SUBSCRIBE clicks

# Streams — persistent
redis-cli XADD clicks:stream "*" MAXLEN "~" 1000000 \
  user u_42 page "/pricing" ts 1720214400123
# Consumer (can subscribe at any time, even after publish):
redis-cli XREADGROUP GROUP analytics worker-a \
  COUNT 100 BLOCK 5000 STREAMS clicks:stream ">"

# Restart scenario — same producer, two consumers
import redis, time
r = redis.Redis(decode_responses=True)

# Producer emits 100 clicks
for i in range(100):
    r.xadd("clicks:stream", {"user": f"u_{i}", "page": "/pricing"}, id="*",
           maxlen=1_000_000, approximate=True)
    r.publish("clicks", f"user=u_{i} page=/pricing")

# --- 1 second later, restart both consumers ---
time.sleep(1)

# Pub/Sub consumer — 100% loss (never subscribed during the publish)
r.pubsub().subscribe("clicks")

# Streams consumer — 0% loss, replays from the group's PEL
r.xgroup_create("clicks:stream", "analytics", id="0", mkstream=True)
result = r.xreadgroup("analytics", "worker-a",
                      {"clicks:stream": ">"}, count=100, block=5000)
print(f"Replayed {sum(len(entries) for _, entries in result)} clicks")

Step-by-step explanation.

1. Pub/Sub broadcasts each PUBLISH to whichever clients are currently SUBSCRIBEd. If no client is listening at the exact moment of publish, the message is gone forever. There is no storage, no queue, no delivery confirmation. This is fire-and-forget in the most literal sense.
2. Streams stores every XADD in an in-memory radix tree (persisted via AOF/RDB if enabled). New consumers can XREADGROUP with > to get never-delivered entries or with an explicit ID to replay historical entries. The stream is a durable log, not a broadcast.
3. In the restart scenario, both producers emit 100 events. The Pub/Sub consumer sees zero events after restart — the events were emitted before the subscribe, so nothing was buffered for it.
4. The Streams consumer, after restart, calls XGROUP CREATE ... id="0" (or was already registered) and calls XREADGROUP ... ">". Redis returns every entry not previously delivered to any consumer in the group — all 100 events replay to the fresh consumer. Zero loss.
5. The architectural takeaway: Pub/Sub is a broadcast bus with in-flight-only semantics; Streams is a durable log with consumer-group semantics. For every real-time pipeline in 2026 — clickstream, order events, cache-warm — Streams is the primitive; Pub/Sub is for websocket-style fan-out to connected clients.

Output.

Scenario	Pub/Sub result	Streams result
Subscribed before publish	delivered	delivered
Subscribed after publish	lost	replayed via > cursor
Consumer restart	100% loss during downtime	0% loss (PEL survives)
Multiple consumer groups	broadcast to all subs	independent PELs per group

Rule of thumb. For any pipeline where losing an event is a bug, use Streams. Pub/Sub belongs to the "cache invalidation" and "websocket broadcast" categories only. Interviewers listen for this distinction — mixing them up on a whiteboard is an instant negative signal.

Senior interview question on the four axes of Redis Streams

A senior interviewer often opens with: "You have a clickstream at 100k events per second, a single-region deployment, a small platform team of three engineers, and an existing Redis cluster you already run for caching. Walk me through why you'd choose Redis Streams over Kafka, and cover the four axes I'd expect a senior engineer to name."

Solution Using Streams with MAXLEN, consumer groups, AOF persistence, and a DLQ pattern

Decision — Redis Streams for the 100k-clickstream workload
==========================================================

Axis 1 — XADD/XREAD
   Producer: XADD clickstream "*" MAXLEN "~" 6000000 <fields...>
             (100k evt/sec × 60s retention window = 6M entries)
   Consumer: XREADGROUP GROUP analytics worker-N COUNT 500 BLOCK 5000
             STREAMS clickstream ">"

Axis 2 — Consumer groups
   XGROUP CREATE clickstream analytics "$" MKSTREAM
   XACK clickstream analytics <entry-id> after processing
   XPENDING clickstream analytics IDLE 30000 - + 100
   XCLAIM clickstream analytics recovery-worker 30000 <entry-id>

Axis 3 — Persistence
   appendonly yes
   appendfsync everysec              # 1s durability window
   save 3600 1 300 100 60 10000      # RDB snapshot triggers

Axis 4 — Dead-letter
   XPENDING → deliveries > 5 → XCLAIM to :dlq-worker
   worker: XADD clickstream:dlq "*" original_id X reason "..."
           XACK clickstream analytics X

# Producer — batch pipeline for 100k events/sec
redis-cli --pipe <<EOF
XADD clickstream "*" MAXLEN "~" 6000000 user u_42 page /a
XADD clickstream "*" MAXLEN "~" 6000000 user u_43 page /b
...
EOF

# Consumer skeleton
XGROUP CREATE clickstream analytics "$" MKSTREAM

# per-consumer loop
XREADGROUP GROUP analytics worker-1 \
  COUNT 500 BLOCK 5000 STREAMS clickstream ">"
# process batch...
XACK clickstream analytics <id1> <id2> ...

# Recovery loop (runs every 5 seconds)
XAUTOCLAIM clickstream analytics recovery-worker 30000 0 COUNT 100

Step-by-step trace.

Layer	Choice	Reasoning
Ingest primitive	Redis Streams	100k evt/sec, single-region, sub-ms latency → Streams beats Kafka on ops
MAXLEN	~ 6,000,000	100k × 60s retention window
Consumer group	analytics	one group; N worker consumers
Persistence	appendonly yes, everysec	1s durability window is acceptable for clickstream
Dead-letter	events:dlq via XCLAIM	poison pills reroute after 5 deliveries
Memory footprint	~1.5 GB (6M × 250 bytes)	fits in a 16 GB Redis instance

After the rollout, the platform ships one process instead of a Kafka broker cluster, hits sub-ms p99 append latency, and never sees a 3 AM controller-election page. The workload lives comfortably within the existing Redis cluster budget; the team of three engineers can operate it without a dedicated platform team.

Output:

Metric	Streams	Kafka (hypothetical)
p99 append latency	0.2 ms	2–5 ms
Operational surface	1 process + AOF	broker cluster + KRaft + Schema Reg
Memory footprint	~1.5 GB	15–30 GB disk (7d retention)
Team size to operate	1–2 engineers	3–5 engineers
Multi-region story	manual replication	native MirrorMaker

Why this works — concept by concept:

• Match the tool to the topology — single-region + sub-ms latency + small team is the exact fit for Streams. Kafka's replicated log is over-engineering for this shape. The decision framework is a four-row matrix (latency, durability, throughput, ops), not a coin flip.
• MAXLEN as the memory safety belt — every XADD must include MAXLEN ~ <target>. Without it, a single misbehaving producer OOMs the Redis instance in minutes. The ~ (approximate) variant is 3–10× faster than = and matters at 100k appends/sec.
• Consumer groups over raw XREAD — XREAD is stateless; XREADGROUP tracks PEL and delivers each entry to exactly one consumer in the group. For fan-out to N workers, XREADGROUP is the only correct primitive.
• appendonly yes + fsync everysec — the pragmatic durability choice. always fsyncs every write (safest, ~5× slower on hot workloads); no lets the OS batch (fastest, ~30-second data loss on crash); everysec is the middle ground every production deployment ships.
• Cost — one Redis process, one AOF file, one RDB snapshot, one consumer-group PEL. Operational cost is O(1) in shard count. Kafka's equivalent is O(broker × partition) with a controller quorum on top. The Streams choice pays for itself the first time you don't get paged at 3 AM.

Streaming
Topic — streaming
Streaming problems on Redis Streams and consumer groups

Practice →

ETL Topic — etl ETL problems on real-time ingest pipelines

Practice →

---

2. XADD + XREAD + XREADGROUP

Three commands, three semantics — the append, the cursor read, and the consumer-group-aware read

The mental model in one line: xadd appends a field/value entry to the stream and returns the auto-generated <ms>-<seq> ID; xread is a cursor-based read that returns entries after a client-tracked ID (with optional BLOCK); xreadgroup is the consumer-group-aware read that returns entries the group has never delivered to any consumer (>) or entries in the current consumer's pending list (explicit ID) — with server-side PEL bookkeeping. Every other Redis Streams interview question is a variation on which of these three you should be using.

XADD — the append primitive.

• Command shape. XADD <key> [NOMKSTREAM] [MAXLEN|MINID [~|=] <threshold>] * <field> <value> [<field> <value> ...].
• Auto-generated IDs. * is the canonical ID — Redis returns <current-ms>-<seq>. IDs are strictly monotonic per stream: within the same millisecond, <seq> counts up (1720214400123-0, 1720214400123-1, ...); across milliseconds, the leading <ms> component ensures monotonicity.
• NOMKSTREAM. Fail the append with nil if the stream key does not exist. Useful for pipelines where the consumer registers the stream first; producers should not create streams the consumer has never subscribed to.
• MAXLEN vs MINID. MAXLEN ~ 10000 trims by count; MINID ~ 1720214000000-0 trims by minimum ID (typically a "keep last N minutes" window). Almost every production deployment uses MAXLEN.
• The ~ trick. MAXLEN ~ allows the trim to run in O(1) amortised time by trimming whole radix-tree pages instead of individual entries. Use ~ for high-throughput streams; use = only when the exact bound matters (rare).
• Pipelining. For high append rates (>10k/sec), pipeline XADD calls to amortise the network round-trip. A 1000-entry pipeline collapses 1000 XADDs into 1 round-trip; measured throughput scales roughly linearly to the CPU limit of the single Redis thread.

XREAD — the stateless cursor read.

• Command shape. XREAD [COUNT <n>] [BLOCK <ms>] STREAMS <key1> ... <id1> ....
• The cursor. The client tracks the last ID it consumed. Pass that ID (say 1720214400123-0) and Redis returns entries with ID > that value. This is cursor-based, not offset-based like Kafka.
• The special IDs. $ means "only entries created after this call registers with the stream" — used in the first BLOCK call to skip historical entries. + is "the highest possible ID" (rarely useful for reads). Any explicit <ms>-<seq> is "start after this ID."
• Non-blocking. Without BLOCK, XREAD returns immediately with whatever is available past the cursor (or empty).
• Blocking. BLOCK 5000 suspends the connection for up to 5 seconds waiting for new data. A new XADD by any producer wakes the blocked reader immediately (server-side notification).
• No server-side state. XREAD does not remember which entries the client processed. The client is responsible for tracking the cursor. Two clients doing XREAD on the same stream see the same entries — not a fan-out primitive.

XREADGROUP — the consumer-group-aware read.

• Command shape. XREADGROUP GROUP <group> <consumer> [COUNT <n>] [BLOCK <ms>] [NOACK] STREAMS <key> ... <id> ....
• The special IDs. > means "entries never delivered to any consumer in this group." Any explicit ID means "entries in this consumer's pending list starting after that ID" (used for recovery reads).
• Delivery semantics. Each entry the group has not yet delivered is handed to exactly one consumer. Redis picks the consumer that issued the XREADGROUP call — round-robin, not stickiness.
• PEL update. When XREADGROUP delivers entry E to consumer C, Redis adds (E, C, now, delivery-count=1) to the group's pending-entry list. The entry stays in the PEL until XACK.
• NOACK. Rarely used; skips the PEL update. Use only for "fire-and-forget within a group" workloads (e.g., metrics fan-out where re-delivery on failure is not required).

How XADD generates its IDs — the monotonicity guarantee.

• Millisecond component. The server's current epoch-millisecond clock.
• Sequence component. A counter that increments if multiple XADDs land in the same millisecond, guaranteeing strict order.
• Clock-skew handling. If the server's clock ever goes backwards (NTP adjustment, VM live migration), Redis clamps the ID to last_ms + 1 to preserve monotonicity. Never trust the ID as a wall-clock timestamp beyond ~1s resolution.
• Client-supplied IDs. You can pass an explicit <ms>-<seq> instead of *, but the ID must be strictly greater than the previous entry's ID or the XADD fails. Almost always let Redis generate the ID.

What BLOCK really does.

• The wait model. The reader's connection is added to a per-stream blocked-clients list. It does not consume any CPU while blocked.
• The wake path. When a producer XADDs a new entry, Redis walks the blocked-clients list and wakes anyone waiting on that stream. Wake latency is well under a millisecond on a warm server.
• The timeout. BLOCK 0 blocks forever; BLOCK 5000 returns nil after 5 seconds. Choose the timeout based on your consumer's health-check interval.
• The COUNT-BLOCK interaction. COUNT caps the batch size; BLOCK returns as soon as any entry is available, up to COUNT. You do not wait for COUNT entries — you wait for the first entry, then take up to COUNT.

Common interview probes on XADD/XREAD.

• "Walk me through the <ms>-<seq> ID model" — required answer.
• "What does $ mean in XREAD?" — only-new (skip historical).
• "How is XREAD different from XREADGROUP?" — XREAD is stateless; XREADGROUP has PEL.
• "Why use MAXLEN ~ instead of MAXLEN =?" — O(1) amortised trim vs O(n) exact trim.

Worked example — batching XADD for 100k appends/sec

Detailed explanation. A producer feeds 100,000 clickstream events per second to Redis Streams. A naive per-event XADD hits the Linux syscall + network round-trip cost per call and caps at roughly 20,000 XADDs/sec over TCP. Pipelining collapses the round-trips and gets to the CPU limit of the single Redis thread. Walk through the numbers.

• Naive throughput. 100k XADDs/sec × 1 round-trip = 100k round-trips/sec, each ~30 µs — 3 seconds of latency budget per real second of traffic. Never works.
• Pipeline throughput. 100k XADDs/sec in 1000-batch pipelines = 100 round-trips/sec — well under 1 ms/sec of round-trip overhead. Works comfortably.

Question. Write the Python producer that pipelines XADD in 1000-batch flushes and computes the resulting throughput and latency.

Input.

Parameter	Value
Event rate	100,000 events/sec
Batch size	1,000
Flush rate	100 flushes/sec
Network RTT	30 µs
Per-XADD server CPU cost	~5 µs

Code.

import redis
import time
import queue

# Single Redis client, connection pool of 16
r = redis.Redis(
    host="cache-1.internal",
    port=6379,
    decode_responses=True,
    socket_keepalive=True,
    health_check_interval=30,
)

def producer_loop(event_source: queue.Queue, batch_size: int = 1000,
                  flush_interval_s: float = 0.010):
    """Batch XADD into pipelines of `batch_size` or flush every 10 ms."""
    buf = []
    last_flush = time.monotonic()
    while True:
        try:
            ev = event_source.get(timeout=flush_interval_s)
            buf.append(ev)
        except queue.Empty:
            pass

        should_flush = (
            len(buf) >= batch_size
            or (buf and time.monotonic() - last_flush >= flush_interval_s)
        )
        if not should_flush:
            continue

        with r.pipeline(transaction=False) as pipe:
            for e in buf:
                pipe.xadd(
                    "clickstream",
                    fields=e,
                    id="*",
                    maxlen=6_000_000,
                    approximate=True,
                )
            pipe.execute()
        buf.clear()
        last_flush = time.monotonic()

# Fan-in: an in-process queue producers write to
event_q: queue.Queue = queue.Queue(maxsize=50_000)

# Bench — throughput comparison
# Naive per-event XADD
redis-benchmark -h cache-1 -p 6379 -n 100000 -q -P 1 XADD clickstream '*' MAXLEN '~' 6000000 field1 value1

# Pipelined XADD (batch 1000)
redis-benchmark -h cache-1 -p 6379 -n 100000 -q -P 1000 XADD clickstream '*' MAXLEN '~' 6000000 field1 value1

Step-by-step explanation.

1. The producer loop maintains a bounded buf of pending events. It reads from an in-process queue; if no event arrives within flush_interval_s, it flushes whatever is buffered anyway to bound tail latency.
2. The pipeline batches batch_size XADDs. transaction=False avoids wrapping the batch in MULTI/EXEC — pipelining does not require atomicity, and skipping the transaction wrapper saves CPU on both client and server.
3. The flush_interval_s = 10 ms is a floor on max end-to-end latency. If traffic drops (fewer than 100k events/sec), the loop still flushes every 10 ms so individual events do not sit in buf waiting for the batch to fill.
4. The throughput math: 100k events/sec at 1000-batch pipelines = 100 round-trips/sec. Redis's single thread services roughly 250k operations/sec of pipelined XADDs on modest hardware — comfortable 2.5× headroom.
5. The alternative — many producer processes each doing per-event XADD — would need ~40 producer connections to hit 100k/sec throughput, and each connection would still see round-trip latency per event. Pipelining from one producer beats fanning out to many un-pipelined producers.

Output.

Producer shape	RTT/event	Throughput	CPU headroom
Per-event XADD, 1 client	30 µs	~20k/sec	5× under-utilised
Per-event XADD, 40 clients	30 µs	~100k/sec	0× (network bound)
Pipeline 1000, 1 client	0.03 µs amortised	~100–150k/sec	2.5× headroom
Pipeline 1000, 4 clients	0.03 µs amortised	~500k/sec	Redis CPU bound

Rule of thumb. For any Streams producer above ~10k events/sec, pipeline in batches of 500–1000 with a 10 ms flush interval. Naive per-event XADD is the #1 cause of "why is Redis suddenly slow?" incidents — it is not slow; you are just paying network round-trip per event instead of amortising.

Worked example — the BLOCK pattern for a real-time consumer

Detailed explanation. A real-time consumer wants to react to each new stream entry within a millisecond of arrival. The naive polling loop (XREAD without BLOCK, sleep for 100 ms, repeat) adds 100 ms of median latency and burns CPU when no data arrives. The BLOCK pattern uses the server's blocked-clients mechanism to wake the reader immediately when a new entry lands.

• Polling latency. Sleep 100 ms → median 50 ms latency to first delivery, p99 100 ms.
• BLOCK latency. Sub-millisecond — the reader wakes as soon as any producer XADDs.
• CPU cost. Polling burns a CPU core per client; BLOCK is nearly zero CPU while suspended.

Question. Write the Python consumer loop using XREAD BLOCK, with a 5-second timeout for health-check purposes, and show how it processes batches.

Input.

Parameter	Value
Stream	clickstream
Batch size	100
Block timeout	5000 ms
Cursor	starts at $, then last-seen ID

Code.

import redis
import logging

log = logging.getLogger("consumer.xread")
r = redis.Redis(host="cache-1.internal", decode_responses=True)

def xread_consumer(stream_key: str = "clickstream"):
    """Cursor-based blocking read loop. Not consumer-group aware."""
    cursor = "$"          # start at end-of-stream on first call
    while True:
        result = r.xread(
            streams={stream_key: cursor},
            count=100,
            block=5000,   # 5s timeout for health check
        )
        if not result:
            log.debug("xread heartbeat — no new entries in 5s")
            continue

        # result: [(stream_key, [(entry_id, {field: value, ...}), ...])]
        for _stream, entries in result:
            for entry_id, fields in entries:
                try:
                    process_event(fields)
                except Exception as e:
                    log.exception("processing failed for %s: %s", entry_id, e)
                cursor = entry_id     # advance cursor

def process_event(fields: dict):
    # user-supplied handler
    pass

if __name__ == "__main__":
    xread_consumer()

# Verify the block wake latency with a producer-consumer race
# Terminal A — consumer
redis-cli XREAD BLOCK 0 COUNT 1 STREAMS test $

# Terminal B — producer, sends one event
redis-cli XADD test "*" k v

Step-by-step explanation.

1. The consumer starts with cursor = "$", meaning "only entries created after this call registers." This is the standard "live tail" behaviour — historical entries are skipped. If you want to replay historical entries, start with cursor = "0" instead.
2. The block=5000 argument tells Redis to suspend the connection for up to 5 seconds waiting for new data. During the wait, the client uses zero CPU; Redis holds the connection in its blocked-clients list.
3. When any producer XADDs to clickstream, Redis wakes the blocked reader and returns up to count=100 entries. Wake latency is well under a millisecond — the wake path is a linked-list walk over the small number of blocked clients on that stream.
4. After processing the batch, the client updates cursor = entry_id to the last-processed entry's ID. The next XREAD call passes this cursor, so the reader picks up strictly after the last-processed entry.
5. The 5-second timeout returning nothing is the heartbeat path. Even if the stream is idle, the consumer wakes every 5 seconds — a natural point to emit liveness metrics or check a shutdown flag. Never use BLOCK 0 (block forever) in production; the process becomes hard to stop cleanly.

Output.

Pattern	Median latency to first delivery	CPU cost while idle
Polling every 100 ms	50 ms	1 core (busy loop)
Polling every 10 ms	5 ms	1 core (partial)
XREAD BLOCK 5000	< 1 ms	~0 CPU (suspended)
XREAD BLOCK 0	< 1 ms	~0 CPU but no heartbeat

Rule of thumb. Never poll Redis Streams. Always use BLOCK with a 1–5 second timeout. The BLOCK path is nearly free while suspended and delivers sub-ms wake latency — polling is strictly worse on both axes.

Worked example — XREADGROUP with > versus explicit ID

Detailed explanation. The > special ID is the "give me only never-delivered entries" cursor for XREADGROUP. Passing an explicit ID (say 0) means "give me entries in my pending list starting from that ID" — the recovery path used after a consumer restart to reprocess whatever the consumer was working on when it died. Show the difference.

• >. Live-consume mode. Delivers only entries that have never gone to any consumer in this group.
• Explicit ID (e.g., 0). Recovery mode. Delivers entries in this consumer's pending list starting after the given ID.

Question. Show the consumer boot flow: on startup, first read the consumer's PEL (recovery), then transition to live consumption with >.

Input.

Step	Cursor	Purpose
1 (boot)	0	Read this consumer's PEL
2 (steady state)	>	Read never-delivered

Code.

import redis

r = redis.Redis(decode_responses=True)

def bootstrapped_consumer(group: str, consumer: str, stream: str = "clickstream"):
    # Ensure the group exists (idempotent)
    try:
        r.xgroup_create(stream, group, id="$", mkstream=True)
    except redis.ResponseError as e:
        if "BUSYGROUP" not in str(e):
            raise

    # Step 1 — recover this consumer's PEL
    #   passing an explicit ID (0) reads entries the consumer was assigned
    #   but never XACKed (from a previous crashed run).
    while True:
        pending = r.xreadgroup(
            group, consumer,
            {stream: "0"},
            count=100,
        )
        if not pending or not pending[0][1]:
            break
        for _stream, entries in pending:
            for entry_id, fields in entries:
                if process_event(fields):
                    r.xack(stream, group, entry_id)
                # else leave in PEL for XCLAIM by another consumer

    # Step 2 — steady-state live consumption
    while True:
        batch = r.xreadgroup(
            group, consumer,
            {stream: ">"},
            count=100,
            block=5000,
        )
        if not batch:
            continue
        for _stream, entries in batch:
            for entry_id, fields in entries:
                if process_event(fields):
                    r.xack(stream, group, entry_id)

def process_event(fields: dict) -> bool:
    # returns True on success (safe to XACK), False on transient failure
    return True

Step-by-step explanation.

1. The boot flow starts by ensuring the consumer group exists via XGROUP CREATE ... "$" MKSTREAM. The "$" id means "start the group at the end of the stream" — new consumers only see entries added after the group was created. MKSTREAM creates the stream if it does not exist.
2. Step 1 is the recovery path. The consumer calls XREADGROUP ... {stream: "0"} — the explicit 0 cursor tells Redis to return entries in this consumer's pending list, starting from ID 0 (i.e., from the beginning). These are entries the consumer was assigned before its previous run crashed and never got to XACK.
3. The recovery loop drains the PEL: for each entry, re-process it (idempotently) and XACK on success. If processing fails again, leave it in the PEL and let XCLAIM (from a recovery worker) move it elsewhere later.
4. Step 2 is the steady-state path. The consumer calls XREADGROUP ... {stream: ">"} with BLOCK 5000. Redis delivers entries that no consumer in the group has ever received. This is the live-consume mode; the PEL is updated on each delivery.
5. The count=100 batch size is a throughput knob. Larger batches amortise the round-trip better but delay the first entry's processing; typical production values are 50–500 depending on per-event processing latency.

Output.

Phase	Cursor	Entries delivered	PEL effect
Boot recovery	0	This consumer's PEL entries	XACK removes them
Boot recovery	0 (loop)	Fewer entries each iteration	Drains PEL
Steady state	>	Never-delivered entries only	XACK required
Steady state (idle)	> + BLOCK 5000	None; timeout returns nil	Heartbeat

Rule of thumb. Every consumer's boot flow follows the same pattern: recover PEL with explicit ID first, then transition to > for live consumption. Skipping the recovery step means every restart leaks entries into the PEL forever — the classic "why is my XPENDING count monotonically increasing?" bug.

Senior interview question on XADD, XREAD, and XREADGROUP

A senior interviewer might ask: "You have a Redis Streams pipeline with two independent consumer groups — analytics and enrichment — reading the same stream. The enrichment group has 4 workers; analytics has 1. Walk me through what XADD/XREAD/XREADGROUP calls each side runs, the PEL bookkeeping for each group, and how a worker restart in enrichment is handled."

Solution Using per-group XREADGROUP with independent PELs plus consumer-boot recovery

import redis
from redis.exceptions import ResponseError

r = redis.Redis(host="cache.internal", decode_responses=True)
STREAM = "clickstream"

def ensure_group(group: str, start_id: str = "$"):
    try:
        r.xgroup_create(STREAM, group, id=start_id, mkstream=True)
    except ResponseError as e:
        if "BUSYGROUP" not in str(e):
            raise

# --- Producer side (one process) --------------------------------------
def producer_send(events):
    with r.pipeline(transaction=False) as pipe:
        for ev in events:
            pipe.xadd(STREAM, fields=ev, id="*",
                      maxlen=6_000_000, approximate=True)
        pipe.execute()

# --- analytics group (1 consumer) -------------------------------------
def analytics_consumer(consumer_name: str = "analytics-1"):
    ensure_group("analytics")
    # boot recovery
    while True:
        pending = r.xreadgroup("analytics", consumer_name,
                               {STREAM: "0"}, count=500)
        if not pending or not pending[0][1]:
            break
        for _, entries in pending:
            for entry_id, fields in entries:
                if handle_analytics(fields):
                    r.xack(STREAM, "analytics", entry_id)
    # live
    while True:
        batch = r.xreadgroup("analytics", consumer_name,
                             {STREAM: ">"}, count=500, block=5000)
        if not batch:
            continue
        for _, entries in batch:
            ids = []
            for entry_id, fields in entries:
                if handle_analytics(fields):
                    ids.append(entry_id)
            if ids:
                r.xack(STREAM, "analytics", *ids)

# --- enrichment group (N consumers) ----------------------------------
def enrichment_consumer(consumer_name: str):
    ensure_group("enrichment")
    # boot recovery (per-consumer PEL)
    while True:
        pending = r.xreadgroup("enrichment", consumer_name,
                               {STREAM: "0"}, count=100)
        if not pending or not pending[0][1]:
            break
        for _, entries in pending:
            for entry_id, fields in entries:
                if handle_enrichment(fields):
                    r.xack(STREAM, "enrichment", entry_id)
    # live
    while True:
        batch = r.xreadgroup("enrichment", consumer_name,
                             {STREAM: ">"}, count=100, block=5000)
        if not batch:
            continue
        for _, entries in batch:
            ids = []
            for entry_id, fields in entries:
                if handle_enrichment(fields):
                    ids.append(entry_id)
            if ids:
                r.xack(STREAM, "enrichment", *ids)

def handle_analytics(fields): return True
def handle_enrichment(fields): return True

Step-by-step trace.

Component	Command	Effect on group PEL	Effect on consumer
Producer	XADD clickstream * MAXLEN ~ 6M ...	Neither PEL touched (no delivery yet)	—
analytics-1 boots	XREADGROUP GROUP analytics analytics-1 {stream:0}	Reads existing PEL for analytics	Redelivers pending
analytics-1 live	XREADGROUP GROUP analytics analytics-1 {stream:>}	Adds new deliveries to PEL	Increments delivery-count
analytics-1 XACK	XACK clickstream analytics <id>	Removes entry from PEL	—
enrichment-2 boots	XREADGROUP GROUP enrichment enrichment-2 {stream:0}	Reads PEL entries owned by enrichment-2	Only its own pending
enrichment-2 crashes	(nothing)	PEL entries stay assigned to enrichment-2	Idle time increases
recovery worker	XAUTOCLAIM ... 30000 0 COUNT 100	Reassigns idle entries to new owner	Increments delivery-count

After the rollout, the analytics group processes every entry with delivery-count = 1 (no failures); the enrichment group processes every entry across 4 workers with load-balanced delivery. A worker restart triggers automatic recovery via XREADGROUP {stream:0} and eventual XAUTOCLAIM reassignment for stuck entries.

Output:

Metric	analytics group	enrichment group
Consumers	1	4
Delivery pattern	one-to-one with entries	round-robin across workers
Independent PEL	yes	yes
Restart behaviour	boot recovery reads own PEL	boot recovery reads own PEL
Group semantics	fully independent	fully independent

Why this works — concept by concept:

• Two groups, two PELs — each consumer group has its own pending-entry list. Redis delivers every entry independently to each group. This is the fan-out primitive that makes multi-consumer pipelines possible on a single stream.
• > for live, explicit-ID for recovery — the two XREADGROUP cursors serve two different jobs. > is the live-consume path; explicit ID (typically 0) is the "read my PEL" recovery path. Consumers alternate — recovery on boot, then > steady-state.
• PEL is per-(group, consumer) — inside a group, each consumer owns the entries it was delivered. A restart lets the same consumer name reclaim its PEL; a permanently-dead consumer needs XCLAIM / XAUTOCLAIM to reassign.
• Batch XACK — XACK stream group id1 id2 id3 ... acknowledges up to 100 entries per call. Batching XACKs is the same optimisation as batching XADDs — amortise the round-trip.
• Cost — one XADD per event on the producer side (pipelined); one XREADGROUP + one XACK per batch on each consumer. Per-entry server-side cost is O(log N) where N is the entry count, thanks to the radix-tree index. At 100k appends/sec on 4 groups × 4 workers, Redis's single thread stays under 40% CPU.

Streaming
Topic — streaming
Streaming problems on XADD, XREAD, and cursor semantics

Practice →

ETL Topic — etl ETL problems on batched real-time ingest

Practice →

---

3. Consumer groups + XACK + XPENDING

The pending-entry list is the state machine — every consumer-group interview probes what happens between XREADGROUP and XACK

The mental model in one line: a redis consumer group is a server-side state machine anchored on the pending-entry list (PEL): every XREADGROUP delivery adds (entry-id, consumer, delivery-time, delivery-count) to the PEL, every xack removes an entry, xpending reads the PEL for observability, and XCLAIM/XAUTOCLAIM reassigns entries from idle or dead consumers to healthy ones. Everything else is elaboration on this cycle.

The consumer-group state machine.

• Group creation. XGROUP CREATE <stream> <group> <start-id> [MKSTREAM]. The start-id is $ for "start at the end" or an explicit ID for "start from this point in history."
• Consumer names. Consumers are created lazily on the first XREADGROUP with a given consumer name. There is no explicit "register consumer" step; the name appears in the group's state when first used.
• Delivery. XREADGROUP GROUP <g> <c> ... {stream:>} delivers up to COUNT entries never seen by any consumer in the group. Redis picks the delivery order deterministically (by ID) and returns entries to whichever consumer issued the call.
• PEL entry. For each delivered entry, Redis appends (entry-id, consumer-name, last-delivery-ms, delivery-count) to the group's PEL.
• Acknowledgement. XACK <stream> <group> <id> [<id> ...] removes entries from the PEL. Multiple IDs can be acked in one call.
• Reassignment. XCLAIM <stream> <group> <new-consumer> <min-idle-ms> <id> [<id> ...] transfers entries from their current owner to a new consumer. Used to recover from dead consumers.
• Auto-reassignment. XAUTOCLAIM <stream> <group> <new-consumer> <min-idle-ms> <start-id> [COUNT <n>] (Redis 6.2+) scans the PEL and auto-transfers idle entries in one call.

XPENDING — the observability primitive.

• Summary form. XPENDING <stream> <group> returns (count, min-id, max-id, [(consumer, count), ...]) — the full PEL summary. Use to sanity-check group health.
• Detail form. XPENDING <stream> <group> [[IDLE <ms>] <start-id> <end-id> <count> [<consumer>]] returns per-entry details: entry ID, owner consumer, idle time, delivery count. Use to drill into stuck entries.
• Idle time. The IDLE <ms> filter returns only entries idle longer than the threshold. Set to 30–60 seconds to find entries that are stuck (consumer probably dead).
• Delivery count. The delivery-count is incremented on every XREADGROUP that redelivers this entry (via explicit ID or XCLAIM). Use as the "poison pill" detection signal — entries with delivery-count > 5 are almost certainly buggy consumers, not transient failures.

XCLAIM vs XAUTOCLAIM.

• XCLAIM. Manual reassignment of specific entry IDs. Use when you know exactly which entries are stuck (e.g., from a targeted XPENDING scan).
• XAUTOCLAIM. Introduced in Redis 6.2. Combines "scan PEL for idle entries" + "reassign" in one atomic call. The pragmatic default for recovery workers.
• Idle threshold. Both commands require a min-idle-ms argument — only entries idle longer than this are reassigned. Set to 30,000–60,000 ms; below 30 seconds you risk reassigning entries a healthy but slow consumer is still processing.
• Delivery-count increment. Both commands increment delivery-count on reassignment. Use as the DLQ trigger — reassign the entry to a :dlq worker if delivery-count > 5.

The dead-consumer failover flow.

• Detection. A recovery worker polls XPENDING <stream> <group> IDLE 30000 - + 100 every 5 seconds.
• Reassignment. Idle entries are reassigned via XAUTOCLAIM ... 30000 0 COUNT 100. Entries with delivery-count > 5 are rerouted to a DLQ worker instead.
• Consumer cleanup. A truly dead consumer (name never used again) should be removed with XGROUP DELCONSUMER <stream> <group> <consumer>. Skipping this leaves ghost consumer entries in XPENDING summary; the entries are gone (reassigned) but the row lingers.

Common interview probes on consumer groups.

• "What is the PEL and when does an entry enter it?" — required answer.
• "Walk me through what happens when a consumer dies mid-batch" — PEL retention + XCLAIM.
• "How do you detect a poison pill?" — delivery-count > threshold via XPENDING.
• "When do you use XCLAIM vs XAUTOCLAIM?" — targeted vs bulk.

Worked example — three consumers, one dying mid-batch

Detailed explanation. A three-consumer group enrichment processes clickstream entries. Consumer c2 dies mid-batch (Kubernetes evicts the pod) after being delivered five entries but before acknowledging any. A recovery worker detects the idle entries after 30 seconds and reassigns them via XAUTOCLAIM. Walk through the exact state of the PEL at each step.

• Initial state. Group has 3 consumers c1, c2, c3; each has been delivered several entries.
• Death event. c2 dies with 5 entries in its PEL, all with delivery-count = 1.
• Recovery event. After 30 s idle, XAUTOCLAIM reassigns those 5 entries to a recovery consumer c-recovery.

Question. Show the XPENDING snapshots before and after the failover, with the exact PEL state at each moment.

Input.

Consumer	PEL entries before	Idle time
c1	3	200 ms (processing)
c2	5	60 s (DEAD)
c3	4	500 ms (processing)

Code.

# 1. Snapshot before recovery
redis-cli XPENDING clickstream enrichment
# 1) (integer) 12       # total PEL entries
# 2) "1720214400000-0"  # min entry ID
# 3) "1720214460000-3"  # max entry ID
# 4) 1) 1) "c1"         # per-consumer breakdown
#       2) "3"
#    2) 1) "c2"
#       2) "5"
#    3) 1) "c3"
#       2) "4"

# 2. Detail — find entries idle > 30 s (c2's entries)
redis-cli XPENDING clickstream enrichment IDLE 30000 - + 100
# 1) 1) "1720214430000-0"
#    2) "c2"                    # owned by c2
#    3) (integer) 60000         # idle 60 seconds
#    4) (integer) 1             # delivery count
# ... 4 more entries owned by c2, all idle > 30 s ...

# 3. XAUTOCLAIM — reassign to recovery consumer
redis-cli XAUTOCLAIM clickstream enrichment c-recovery 30000 0 COUNT 100
# 1) "0-0"                              # next cursor
# 2) 1) 1) "1720214430000-0"            # entry ID
#       2) 1) "user"                    # fields
#          2) "u_42"
#          3) "page"
#          4) "/pricing"
#    ...4 more entries...
# 3) (empty array)                      # deleted-entry IDs

# 4. Snapshot after recovery
redis-cli XPENDING clickstream enrichment
# 1) (integer) 12       # PEL count unchanged (entries not acked yet)
# 2) "1720214400000-0"
# 3) "1720214460000-3"
# 4) 1) 1) "c1"
#       2) "3"
#    2) 1) "c3"
#       2) "4"
#    3) 1) "c-recovery"  # c2's entries now owned by c-recovery
#       2) "5"

Step-by-step explanation.

1. Before the failover, XPENDING shows the PEL with 12 entries total, split across three consumers. c1 owns 3 entries currently being processed (idle 200 ms); c2 owns 5 entries with idle time growing since the pod died; c3 owns 4 entries in-flight.
2. The recovery worker polls XPENDING ... IDLE 30000 - + 100 every 5 seconds. The IDLE filter narrows the result to entries that have been in the PEL longer than 30 seconds without being XACKed. In this case, c2's 5 entries qualify — all idle 60 seconds.
3. XAUTOCLAIM clickstream enrichment c-recovery 30000 0 COUNT 100 does the reassignment in one atomic call. It scans the PEL for entries idle > 30,000 ms, reassigns them from their current owner to c-recovery, increments delivery-count, and returns the entries (fields included) so the recovery worker can re-process them.
4. The delivery-count on each reassigned entry becomes 2 (was 1 on the original delivery to c2, now 2 after reassignment to c-recovery). If a subsequent XAUTOCLAIM ever needs to move them again, delivery-count grows further — the poison-pill signal.
5. After the failover, XPENDING shows the same total (12 entries) but the ownership has shifted: c1 3, c3 4, c-recovery 5. When the recovery worker processes and XACKs each entry, the PEL count drops. c2 no longer appears in the PEL summary because it has zero entries; it may still be listed by XINFO CONSUMERS until explicitly deleted with XGROUP DELCONSUMER.

Output.

Moment	c1 PEL	c2 PEL	c3 PEL	c-recovery PEL
Before crash	3	5	4	—
Crash + 60 s	3	5 (idle 60 s)	4	—
After XAUTOCLAIM	3	0	4	5
After c-recovery XACKs	3	0	4	0

Rule of thumb. Ship a recovery worker that runs XAUTOCLAIM ... 30000 0 COUNT 100 every 5–10 seconds against every consumer group. Without it, a single crashed consumer's PEL entries stall forever. The recovery worker is 20 lines of Python and is not optional in production.

Worked example — poison-pill routing to a dead-letter stream

Detailed explanation. A specific entry has a payload that causes every consumer to throw and fail to XACK. After 5 redeliveries the entry is a confirmed poison pill; the recovery worker should route it to a dead-letter stream instead of endlessly cycling it through consumers. Implement the DLQ pattern.

• Detection. XPENDING ... IDLE 30000 - + 100 returns entries with delivery-count.
• Threshold. delivery-count > 5 → poison pill.
• Action. XCLAIM to a dlq-worker consumer, XADD to a secondary :dlq stream, XACK on the original.

Question. Write the recovery worker in Python that detects poison pills and routes them to a DLQ stream while re-XCLAIMing normal transient failures.

Input.

Parameter	Value
Idle threshold	30,000 ms
Poison threshold	delivery-count > 5
DLQ stream	clickstream:dlq
Poll interval	5 s

Code.

import redis
import time
import json
import logging

log = logging.getLogger("recovery")
r = redis.Redis(host="cache-1.internal", decode_responses=True)

STREAM         = "clickstream"
GROUP          = "enrichment"
DLQ_STREAM     = "clickstream:dlq"
DLQ_CONSUMER   = "dlq-worker"
IDLE_MS        = 30_000
POISON_MAX_DEL = 5
POLL_INTERVAL  = 5.0

def recovery_loop():
    while True:
        # 1. Read all idle entries (up to 500 per pass)
        pending = r.xpending_range(
            STREAM, GROUP,
            min="-", max="+",
            count=500,
            idle=IDLE_MS,
        )
        if not pending:
            time.sleep(POLL_INTERVAL)
            continue

        # 2. Bucket poison pills vs recoverable
        poison_ids     = []
        poison_records = []
        recover_ids    = []
        for e in pending:
            if e["times_delivered"] > POISON_MAX_DEL:
                poison_ids.append(e["message_id"])
            else:
                recover_ids.append(e["message_id"])

        # 3. Route poison pills to DLQ stream + XACK original
        if poison_ids:
            claimed = r.xclaim(
                STREAM, GROUP, DLQ_CONSUMER,
                min_idle_time=IDLE_MS,
                message_ids=poison_ids,
            )
            with r.pipeline(transaction=False) as pipe:
                for entry_id, fields in claimed:
                    pipe.xadd(
                        DLQ_STREAM,
                        {
                            "original_id": entry_id,
                            "reason": "delivery_count_exceeded",
                            "payload": json.dumps(fields),
                        },
                        id="*",
                        maxlen=1_000_000,
                        approximate=True,
                    )
                    pipe.xack(STREAM, GROUP, entry_id)
                pipe.execute()
            log.warning("routed %d poison entries to %s", len(poison_ids), DLQ_STREAM)

        # 4. Reassign recoverables via XAUTOCLAIM
        if recover_ids:
            r.xclaim(
                STREAM, GROUP, "recovery-worker",
                min_idle_time=IDLE_MS,
                message_ids=recover_ids,
            )
            log.info("reassigned %d recoverable entries", len(recover_ids))

        time.sleep(POLL_INTERVAL)

if __name__ == "__main__":
    recovery_loop()

Step-by-step explanation.

1. The recovery loop polls XPENDING ... IDLE 30000 every 5 seconds. The idle filter narrows to entries that have been in the PEL longer than 30 seconds without XACK — the "consumer probably dead" signal.
2. Each returned PEL entry has times_delivered (the delivery-count). The loop buckets entries into two lists: poison_ids for entries redelivered more than 5 times, recover_ids for entries with fewer redeliveries (probably a transient consumer failure).
3. For poison pills, XCLAIM transfers ownership to the DLQ worker. The response includes the entry fields, so the recovery loop can XADD them to the clickstream:dlq stream with original_id, reason, and the serialised payload. The DLQ stream is a separate Redis Stream that a human triages later. Immediately after the XADD, the original entry is XACKed to clear it from the main group's PEL.
4. For recoverable entries, XCLAIM transfers ownership to a generic recovery-worker consumer. That consumer runs the same processing logic as the original workers; because the entries are now assigned to it, its next XREADGROUP ... {stream: 0} call will pick them up.
5. The MAXLEN ~ 1_000_000 on the DLQ stream is a safety belt — an unbounded DLQ can grow forever if the poison-pill rate is high enough. A million-entry cap is generous but bounded; the on-call reviews and truncates from there.

Output.

Entry state	delivery_count	Recovery action
Idle < 30 s	any	Skip (still in-flight)
Idle > 30 s, count ≤ 5	1–5	XCLAIM to recovery-worker
Idle > 30 s, count > 5	> 5	XADD to :dlq, XACK original
DLQ backlog	—	Human triage

Rule of thumb. Ship the DLQ pattern with every consumer group from day one. delivery_count > 5 → DLQ is the canonical poison-pill escape hatch; skipping it means one bad payload endlessly cycles through your workers, burning CPU forever.

Worked example — reading and interpreting XPENDING for on-call

Detailed explanation. A senior on-call notices the enrichment consumer group's lag is growing. The first command is XPENDING clickstream enrichment — the summary form. Walk through interpreting the output and drilling into the detail form when needed.

• Summary form. Returns (count, min-id, max-id, [(consumer, count), ...]).
• Detail form. Returns per-entry (entry-id, consumer, idle-ms, delivery-count).
• Alert threshold. PEL count > 10,000 for 2 minutes.

Question. Interpret the following XPENDING outputs. Explain what each row means and what runbook step follows.

Input.

Snapshot A — healthy
XPENDING clickstream enrichment
1) (integer) 42
2) "1720214400000-0"
3) "1720214402000-3"
4) 1) 1) "c1"
      2) "14"
   2) 1) "c2"
      2) "15"
   3) 1) "c3"
      2) "13"

Snapshot B — one consumer stuck
XPENDING clickstream enrichment
1) (integer) 15042
2) "1720214300000-0"
3) "1720214402000-3"
4) 1) 1) "c1"
      2) "14"
   2) 1) "c2"
      2) "15013"   ← the anomaly
   3) 1) "c3"
      2) "15"

Code.

# Snapshot B — drill into c2
redis-cli XPENDING clickstream enrichment IDLE 30000 - + 100 c2
# 1) 1) "1720214300500-0"
#    2) "c2"
#    3) (integer) 122340   # idle 2 minutes
#    4) (integer) 1        # delivery count still 1 → no reassign yet
# ...100 more entries owned by c2...

# Check consumer info
redis-cli XINFO CONSUMERS clickstream enrichment
# 1) 1) "name"
#    2) "c1"
#    3) "pending"
#    4) (integer) 14
#    5) "idle"
#    6) (integer) 500
#  ...
# 2) 1) "name"
#    2) "c2"
#    3) "pending"
#    4) (integer) 15013
#    5) "idle"
#    6) (integer) 130000   ← 2.2 min since last seen
#  ...

# Runbook step 1 — force-reassign c2's entries
redis-cli XAUTOCLAIM clickstream enrichment c-recovery 30000 0 COUNT 500

# Runbook step 2 — delete the dead consumer entry
redis-cli XGROUP DELCONSUMER clickstream enrichment c2

# Runbook step 3 — file a ticket, investigate why c2 died

Step-by-step explanation.

1. Snapshot A is healthy: 42 total PEL entries, roughly balanced across three consumers (14–15 each). Each consumer is processing a batch and has not yet XACKed. Nothing to do.
2. Snapshot B has 15,042 total PEL entries. The per-consumer breakdown shows c1 and c3 at ~15 entries each (normal) but c2 at 15,013 entries — a 1000× anomaly. c2 is dead or stuck; entries are accumulating because it never XACKs.
3. The drill-in query XPENDING clickstream enrichment IDLE 30000 - + 100 c2 returns per-entry detail for c2's stuck entries. Idle times of 2+ minutes confirm c2 has not been active. Delivery counts of 1 mean no recovery has run yet.
4. XINFO CONSUMERS confirms the diagnosis: c2's idle field shows 130 seconds since the consumer last called XREADGROUP. Anything above 60 seconds on a healthy consumer is suspicious; 130+ seconds is confirmed-dead.
5. The runbook: (1) XAUTOCLAIM reassigns c2's entries to a recovery consumer for reprocessing; (2) XGROUP DELCONSUMER clickstream enrichment c2 removes the ghost consumer entry from the group's state; (3) file a ticket to investigate what killed c2 (OOM, crash, deploy).

Output.

Signal	Snapshot A	Snapshot B	Interpretation
Total PEL	42	15,042	350× anomaly
Per-consumer balance	14–15 each	c2 at 15,013	Skew — c2 dead
Max PEL entry idle	~200 ms	130 s (c2)	Confirmed dead
XINFO CONSUMERS c2 idle	500 ms	130,000 ms	260× anomaly

Rule of thumb. XPENDING summary is the first command in every consumer-group incident; imbalance across consumers is the first anomaly to look for. A single consumer with 100–1000× the PEL entries of its peers is dead until proven otherwise.

Senior interview question on consumer groups, XACK, and XPENDING

A senior interviewer might ask: "You have a redis consumer group with 8 consumers. Two of them die during a Kubernetes rolling deploy. Walk me through what happens to their PEL entries, how you'd detect it in monitoring, how the recovery flow reassigns them, and what alert threshold you'd set to catch it in production."

Solution Using XAUTOCLAIM + XPENDING monitoring + PagerDuty runbook

# recovery_worker.py — runs as a sidecar or standalone process
import redis, time, json, logging, os
from redis.exceptions import ResponseError

log = logging.getLogger("recovery")
r = redis.Redis(host=os.environ["REDIS_HOST"], decode_responses=True)

STREAM         = "clickstream"
GROUP          = "enrichment"
DLQ_STREAM     = "clickstream:dlq"
DLQ_WORKER     = "dlq-worker"
RECOV_WORKER   = "recovery-worker"
IDLE_MS        = 30_000
POISON_MAX_DEL = 5

def recovery_pass():
    """Single pass — reassign idle entries; route poison pills to DLQ."""
    cursor = "0-0"
    total_moved, total_dlq = 0, 0
    while True:
        # XAUTOCLAIM returns (next_cursor, [(entry_id, {fields}), ...], [deleted])
        result = r.xautoclaim(
            STREAM, GROUP, RECOV_WORKER,
            min_idle_time=IDLE_MS,
            start_id=cursor,
            count=100,
        )
        cursor, claimed, _deleted = result
        if not claimed:
            break

        # Look up delivery-count for each claimed entry
        ids = [eid for eid, _ in claimed]
        details = r.xpending_range(STREAM, GROUP, "-", "+", 100)
        det_by_id = {d["message_id"]: d for d in details if d["message_id"] in ids}

        for entry_id, fields in claimed:
            det = det_by_id.get(entry_id)
            if det and det["times_delivered"] > POISON_MAX_DEL:
                # Poison pill → DLQ + ACK
                r.xadd(DLQ_STREAM, {
                    "original_id": entry_id,
                    "reason": "delivery_count_exceeded",
                    "delivery_count": det["times_delivered"],
                    "payload": json.dumps(fields),
                }, id="*", maxlen=1_000_000, approximate=True)
                r.xack(STREAM, GROUP, entry_id)
                total_dlq += 1
            else:
                total_moved += 1

        if cursor == "0-0":
            break

    return total_moved, total_dlq

def main_loop(poll_s: float = 5.0):
    while True:
        try:
            moved, dlq = recovery_pass()
            if moved or dlq:
                log.info("recovery pass — reassigned %d, DLQ %d", moved, dlq)
        except ResponseError as e:
            log.exception("XAUTOCLAIM error: %s", e)
        time.sleep(poll_s)

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    main_loop()

# prometheus_alerts.yml — PEL saturation
groups:
- name: redis-streams-consumer-groups
  rules:
  - alert: RedisStreamPELGrowing
    expr: redis_stream_pel_count{stream="clickstream",group="enrichment"} > 10000
    for: 2m
    labels: {severity: page}
    annotations:
      summary: enrichment PEL > 10k for 2m — likely dead consumer
      runbook: https://runbooks.internal/redis-streams-pel

  - alert: RedisStreamConsumerIdleTooLong
    expr: redis_stream_consumer_idle_ms{stream="clickstream",group="enrichment"} > 60000
    for: 1m
    labels: {severity: page}
    annotations:
      summary: consumer idle > 60s — likely dead

  - alert: RedisStreamDLQGrowing
    expr: rate(redis_stream_dlq_length[5m]) > 1
    for: 2m
    labels: {severity: warn}
    annotations:
      summary: DLQ growth rate > 1/sec — poison-pill storm

Step-by-step trace.

Layer	Behaviour
Producer	100k XADD/sec into clickstream
8 consumers	XREADGROUP + XACK; PEL rows for in-flight only
2 consumers die	~40 PEL entries per dead consumer become idle
After 30 s idle	recovery_worker's XAUTOCLAIM sees IDLE > 30000
XAUTOCLAIM	Reassigns to recovery-worker, delivery_count++
recovery-worker	XREADGROUP {stream:0} picks them up, processes, XACKs
Poison entries (delivery_count > 5)	Routed to clickstream:dlq with reason + payload
Alert threshold	PEL > 10000 for 2m, idle > 60s, DLQ growth > 1/sec

After the rollout, the recovery worker runs every 5 seconds, reassigns idle entries within 35 seconds of a consumer death, and routes poison pills to the DLQ. The three PagerDuty alerts catch anomalies before customer-visible lag.

Output:

Metric	Before recovery worker	After
Time to reassign dead-consumer PEL	∞ (never)	≤ 35 s
Poison-pill loop CPU cost	∞ (infinite retry)	0 (routed to DLQ after 5 tries)
PEL grows unbounded?	yes	no (capped by recovery + DLQ)
On-call pages per week (baseline)	3–5	≤ 1
DLQ triage cadence	none	weekly

Why this works — concept by concept:

• PEL as the state machine — every consumer group's health lives on the PEL. Delivery adds a row; XACK removes it. A dead consumer means its PEL rows never leave; the recovery worker is the only way out.
• XAUTOCLAIM as the sweep — Redis 6.2+'s XAUTOCLAIM combines "scan PEL by idle time" + "reassign" in one atomic call. Older Redis versions need XPENDING + XCLAIM, which race under load. Modern deployments should be on ≥ 6.2 for this reason alone.
• Delivery-count as the poison signal — a normal entry that gets reassigned once has delivery_count = 2. An entry with delivery_count > 5 has been reassigned 4+ times without success — a poison pill. Route to DLQ, XACK original, move on.
• DLQ as a stream, not a queue — the clickstream:dlq is another Redis Stream, XADDed by the recovery worker. Human triage reads it via XREAD or a custom UI. Do not use LPUSH into a list; you lose the ordered-log semantics.
• Cost — recovery worker runs every 5 seconds; XAUTOCLAIM is O(idle-entries) per pass. On a healthy stream with < 100 idle entries, each pass takes < 1 ms. The pattern scales to 100k events/sec streams comfortably.

Streaming
Topic — streaming
Streaming problems on consumer groups and PEL failover

Practice →

Optimization Topic — optimization Optimization problems on dead-letter and idle-recovery patterns

Practice →

---

4. Persistence — AOF, RDB, streams-in-memory

Streams live in RAM — durability is bolted on via AOF + fsync policy + periodic RDB snapshots

The mental model in one line: a Redis Stream is an in-memory radix tree; durability is not a Streams feature — it is Redis's global persistence model applied to the whole keyspace, controlled by appendonly yes (turn on AOF) + appendfsync everysec (fsync policy) + the save directive for periodic RDB snapshots — and redis streams persistence is really "AOF replay + RDB snapshot restore" applied to your XADDs and XACKs. Understanding which policy loses which milliseconds of data on crash is what senior interviewers probe.

AOF (append-only file) — the primary durability mechanism.

• What it is. Every write command Redis executes is appended to a file (appendonly.aof) on the disk. On restart, Redis replays the file to rebuild the in-memory state.
• Enable. appendonly yes in redis.conf. Off by default in some distributions; explicitly turn it on for any stream you care about.
• File format. Redis serialisation protocol (RESP) — the same wire format the client sends. Human-readable enough that you can cat appendonly.aof | grep XADD to inspect recent writes.
• File location. dir /var/lib/redis + appendfilename appendonly.aof. On a modern Redis (>= 7), the AOF is split into a base file + incremental files under appendonlydir/.
• Rewrite. The AOF grows unboundedly. auto-aof-rewrite-percentage 100 + auto-aof-rewrite-min-size 64mb triggers a background rewrite that compacts the file into an equivalent smaller version.

The three fsync policies.

• always. fsync after every write command. Safest — never lose an XADD, even on kernel panic. Slowest — roughly 3–5× throughput reduction on write-heavy workloads because every command waits for a disk flush.
• everysec. fsync once per second. The middle ground: worst-case data loss is one second of writes on kernel panic, but throughput is within 5% of the "no fsync" case. The pragmatic default every production deployment ships.
• no. Never call fsync explicitly; let the OS decide when to flush. Fastest — no synchronous disk I/O on the write path. Worst-case data loss on kernel panic is the entire kernel page cache write-back window, typically 30 seconds.

When each fsync policy is the correct choice.

• Financial ledger, order events, payment webhooks → always. You cannot lose even one XADD; pay the throughput cost.
• Clickstream, analytics events, cache-warm → everysec. One second of loss on crash is acceptable; the throughput matters more.
• Metrics, logs, telemetry → no or AOF off. Loss is fine; the workload dominates. Some deployments run pure RDB.

RDB (Redis Database) — the point-in-time snapshot.

• What it is. A binary dump of the entire keyspace at a moment in time, written to dump.rdb.
• Trigger. The save directive (save 3600 1 300 100 60 10000 means "snapshot after 1 change in 3600s or 100 changes in 300s or 10000 changes in 60s"). Also BGSAVE for manual snapshots.
• Recovery role. On restart with AOF disabled, Redis loads the RDB. With AOF enabled, Redis prefers the AOF (more recent) but keeps RDB for backup and disaster recovery.
• Performance. RDB forks a child process that writes the snapshot; the parent keeps serving traffic. Copy-on-write memory means the fork's peak memory can spike to 2× the working set briefly.

AOF + RDB — the belt-and-braces default.

• Both on. appendonly yes + save 3600 1 300 100 60 10000. AOF for durability of every write; RDB for periodic backup snapshots.
• AOF-only. Fine for many deployments — appendonly yes + save "" (disable RDB). Simpler.
• RDB-only. Only for workloads that tolerate losing minutes of writes on crash. Rare in production.

Streams-specific persistence considerations.

• Streams live in memory. The stream key holds a radix tree of entries; the radix tree is memory-resident. Persistence is the general Redis mechanism applied to that key.
• PEL persistence. Consumer-group PELs are also memory-resident and persisted via AOF/RDB. On restart, Redis replays the AOF and the PELs come back exactly as they were.
• XADD is a single write. One XADD → one AOF write. Batching XADDs via pipeline reduces the client round-trip cost but not the AOF write count.
• MAXLEN trim is a write. Every XADD with MAXLEN ~ may trim entries; the trim is a separate write to the AOF. The ~ variant amortises the trim overhead (trim happens per-page, not per-entry).

Common interview probes on persistence.

• "Walk me through what happens on a Redis crash with appendfsync everysec" — up to 1 s of writes lost.
• "When would you pick always over everysec?" — financial, order events, payments.
• "What is RDB and how does it differ from AOF?" — snapshot vs write-log.
• "How does AOF rewrite work?" — background compaction of the log into an equivalent smaller version.

Worked example — the AOF write path for a single XADD

Detailed explanation. A single XADD clickstream * MAXLEN ~ 6000000 user u_42 page /pricing command flows through the Redis event loop, the AOF buffer, and the kernel to disk. Walk through the exact write path with appendfsync everysec and quantify the durability window.

• Client side. RESP-encoded command over TCP.
• Server side. Parse, execute, append to AOF buffer, respond.
• AOF flush. Background thread flushes the buffer to disk every second.

Question. Diagram the write path and identify the exact durability guarantee at each step.

Input.

Component	State
Client	Sends XADD
Redis	Executes + writes to AOF buffer
AOF thread	Flushes buffer to disk every 1 s
Kernel	Buffers page cache; syncs on fsync()
Disk	Persists after fsync

Code.

# redis.conf — the persistence block
appendonly           yes
appendfsync          everysec       # 1s durability window
no-appendfsync-on-rewrite no        # keep fsyncing during rewrite
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

# Also keep RDB for backup
save 3600 1
save 300  100
save 60   10000

dir /var/lib/redis
appendfilename appendonly.aof

# For observability
appendonly-write-log-and-continue no

# Simulate the flow — write, verify AOF buffer, wait for flush
# 1. Send an XADD
redis-cli XADD clickstream "*" MAXLEN "~" 6000000 user u_42 page /pricing
# → 1720214400123-0

# 2. Confirm AOF contains it (may need up to 1 s under everysec)
sleep 1.1
tail -c 512 /var/lib/redis/appendonlydir/appendonly.aof.1.incr.aof

# 3. Force flush + check
redis-cli BGREWRITEAOF        # optional — background rewrite
redis-cli INFO persistence | grep aof
# aof_enabled:1
# aof_rewrite_in_progress:0
# aof_last_write_status:ok
# aof_last_bgrewrite_status:ok
# aof_pending_bio_fsync:0     # 0 = flushed
# aof_current_size:34567890

# 4. Verify recovery — kill Redis and restart
sudo kill -9 $(pidof redis-server)
sudo systemctl start redis-server
redis-cli XLEN clickstream   # should match pre-crash length within 1 s of writes

# Measure the durability window
import redis, time, subprocess
r = redis.Redis(decode_responses=True)

# Send 1000 XADDs as fast as possible
start = time.monotonic()
for i in range(1000):
    r.xadd("test:persist", {"i": i}, id="*",
           maxlen=100_000, approximate=True)
elapsed = time.monotonic() - start
print(f"1000 XADDs in {elapsed*1000:.1f} ms")

# Sample AOF state
info = r.info("persistence")
print("aof_pending_bio_fsync:", info["aof_pending_bio_fsync"])
print("aof_last_write_status:", info["aof_last_write_status"])

Step-by-step explanation.

1. The client sends XADD ... over TCP. Redis's event loop reads the RESP payload from the socket, parses the command, and dispatches to the XADD handler. Total time so far: ~10 microseconds on localhost.
2. XADD constructs the new entry (radix tree insert), applies the MAXLEN trim (may drop entries from the head), and appends the command as RESP to the AOF buffer in memory. Total time: ~20 microseconds. Redis responds to the client with the generated entry ID.
3. At this point the XADD is durable only in the AOF buffer (in Redis's process memory). A crash between "AOF buffer write" and "kernel flush" would lose the entry.
4. A background thread wakes every 1 second (with appendfsync everysec) and calls write(fd, buffer) on the AOF file descriptor + fsync(fd). After fsync returns, the entry is durably on disk. The worst-case window from "client received OK" to "durable" is roughly 1 second.
5. On restart, Redis reads the AOF file (base + incremental files in appendonlydir/) and replays every write command. The stream comes back with every XADD that had time to hit disk before the crash — everything since the last fsync is lost.

Output.

Step	Time from client send	State
Client sends XADD	0 µs	In flight
Redis parses + executes	10 µs	In memory only
Redis appends to AOF buffer	20 µs	In process memory
Redis responds OK	25 µs	Client believes durable
Background thread fsyncs	up to 1 s later	Actually durable
Crash before fsync	any point	Data loss for that XADD

Rule of thumb. With appendfsync everysec, a crash loses up to 1 second of writes. For clickstream / analytics workloads, this is fine. For financial ledgers, switch to always and accept the throughput cost. Never trust the client-side "OK" as durability without knowing the fsync policy.

Worked example — RDB snapshot triggers and forked-child cost

Detailed explanation. RDB snapshots run in a background process forked from the main Redis process. Copy-on-write means the child sees a stable snapshot of memory while the parent continues to serve traffic — but the peak memory during snapshot can briefly spike toward 2× the working set. Understand the trigger cadence and the fork cost.

• Triggers. save 3600 1 300 100 60 10000 fires a BGSAVE if any threshold is met.
• Cost. Fork is O(memory pages); on Linux with THP off, roughly 10 ms per GB of resident set. Peak memory spikes during heavy writes because copy-on-write copies modified pages.
• AOF interaction. RDB and AOF are independent; both can run at once.

Question. Configure the RDB triggers for a 10 GB stream workload with 100k appends/sec and estimate the peak memory during snapshot.

Input.

Parameter	Value
Working set (streams)	10 GB
Append rate	100k events/sec
Total instance memory	32 GB
Fork cost per GB	~10 ms (THP off)
Snapshot cadence	at least every 5 min

Code.

# redis.conf — RDB triggers tuned for 10 GB stream workload
save 300 100000         # every 5 min if > 100k changes (always fires under 100k/sec)
save 60  1000000        # every 1 min if > 1M changes  (rarely fires; safety)
save 3600 1000          # every hour if > 1000 changes (safety net for idle)

# RDB output
dbfilename dump.rdb
dir        /var/lib/redis

# Fork tuning
# Disable Transparent Huge Pages — improves fork latency
# echo never > /sys/kernel/mm/transparent_hugepage/enabled

# Copy-on-write is enabled by default; keep vm.overcommit_memory = 1 in sysctl

# Check current RDB state
redis-cli INFO persistence | grep -E 'rdb|save'
# rdb_changes_since_last_save:34521
# rdb_bgsave_in_progress:0
# rdb_last_save_time:1720214400
# rdb_last_bgsave_status:ok
# rdb_last_bgsave_time_sec:12
# rdb_current_bgsave_time_sec:-1

# Manual snapshot for backup
redis-cli BGSAVE
# Background saving started
sleep 5
redis-cli LASTSAVE
# (integer) 1720214460   # unix timestamp of last successful save

# Estimate peak memory during BGSAVE
import redis, time
r = redis.Redis(decode_responses=True)

def snapshot_cost_estimate():
    info = r.info("memory")
    rss_gb = info["used_memory_rss"] / 1e9
    write_rate = 100_000 * 250 / 1e9   # 100k evt/sec × 250 bytes = 25 MB/sec
    # BGSAVE typically takes rss_gb * 10 ms/GB * 100 (scale factor for 100 MB/s I/O)
    # ~= rss_gb seconds
    save_duration_s = rss_gb
    cow_delta_gb = write_rate * save_duration_s
    peak_gb = rss_gb + cow_delta_gb
    return {
        "current_rss_gb": rss_gb,
        "save_duration_s": save_duration_s,
        "cow_delta_gb": cow_delta_gb,
        "peak_gb": peak_gb,
    }

print(snapshot_cost_estimate())
# {'current_rss_gb': 10.0, 'save_duration_s': 10.0, 'cow_delta_gb': 0.25, 'peak_gb': 10.25}

Step-by-step explanation.

1. The save triggers define when Redis kicks off a BGSAVE. save 300 100000 fires every 5 minutes if at least 100k changes have happened — at 100k events/sec, this fires every 5 minutes on the dot. The other lines are safety nets for lower-throughput windows.
2. BGSAVE begins with a fork() syscall. The child process shares memory pages with the parent via copy-on-write. The fork itself is roughly O(memory pages); on a 10 GB working set with THP off, fork completes in ~100 ms.
3. During the snapshot, the parent keeps serving traffic. When the parent writes to a page, the kernel copies the page for the parent and leaves the child with the original — this is copy-on-write. The child's snapshot stays consistent to the fork moment.
4. If the write rate is 25 MB/sec (100k events × 250 bytes) and the snapshot takes 10 seconds, roughly 250 MB of pages get copied on write. Peak memory during snapshot ≈ 10.25 GB — only 2.5% above steady state, well within the 32 GB instance budget.
5. The gotcha: if THP (Transparent Huge Pages) is ON, fork latency scales linearly with page size (2 MB per THP vs 4 KB per normal page), and the copy-on-write overhead spikes. Disable THP on any Redis host that runs BGSAVE — this is a mandatory tuning step.

Output.

Metric	Value
Working set	10 GB
Save duration	~10 s
COW overhead	~250 MB
Peak memory during snapshot	~10.25 GB
Snapshot cadence	every 5 minutes
Instance headroom	32 - 10.25 = 21.75 GB

Rule of thumb. Size the Redis instance for 2× your working set to accommodate BGSAVE peak. Disable Transparent Huge Pages on every Redis host. Keep the RDB triggers modest — every 5 minutes is plenty for backup; more frequent just wastes I/O.

Worked example — choosing appendfsync for different workloads

Detailed explanation. The appendfsync choice is the single most-debated Redis persistence knob. Walk through three workloads and their correct fsync policy, with the throughput impact and the data-loss window for each.

• Workload A. Clickstream — 100k events/sec, loss of 1 s acceptable.
• Workload B. Payment events — 10k events/sec, no loss acceptable.
• Workload C. Metrics — 500k events/sec, minutes of loss acceptable.

Question. For each workload, pick the fsync policy, quantify the throughput impact, and state the worst-case data loss.

Input.

Workload	Rate	Loss tolerance
A: clickstream	100k/sec	1 s
B: payment events	10k/sec	0
C: metrics	500k/sec	60 s

Code.

# Workload A — clickstream
appendonly            yes
appendfsync           everysec
# throughput: within 5% of no-fsync case
# worst-case loss: up to 1 s of writes

# Workload B — payment events
appendonly            yes
appendfsync           always
# throughput: ~30% of no-fsync case (disk-bound)
# worst-case loss: 0 (each write is durable before OK returns)

# Workload C — metrics
appendonly            no
# or: appendonly yes + appendfsync no
save 3600 1 300 100000 60 5000000
# throughput: maximum (no synchronous disk I/O)
# worst-case loss: up to last RDB snapshot (5 min)

# Micro-benchmark the throughput impact
import redis, time

def bench(sync_mode: str, n: int = 10_000):
    # Assume redis.conf appendfsync is set to sync_mode before starting
    r = redis.Redis(decode_responses=True)
    start = time.monotonic()
    with r.pipeline(transaction=False) as pipe:
        for i in range(n):
            pipe.xadd(f"bench:{sync_mode}", {"i": i}, id="*",
                      maxlen=1_000_000, approximate=True)
        pipe.execute()
    elapsed = time.monotonic() - start
    return n / elapsed  # events/sec

# Observed throughput (single Redis, m5.large):
# always   : ~30_000 events/sec
# everysec : ~140_000 events/sec
# no       : ~150_000 events/sec

Step-by-step explanation.

1. Workload A (clickstream): 1 s of loss is fine because analytics is downstream and reprocesses from other sources on failure. everysec gives the pragmatic default — 1 s worst-case loss, 5% throughput cost vs no. Every clickstream deployment ships this.
2. Workload B (payment events): zero loss is required. always is the only correct choice. Every write is durable before the client sees OK; the disk is the bottleneck (roughly 30k events/sec on a modern NVMe SSD with sync writes). Below that rate, always is fine. Above that rate, batch multiple events into one XADD (or into an outer transaction) to reduce the fsync frequency.
3. Workload C (metrics): 60 s of loss is fine because metrics are inherently ephemeral. no (or AOF off entirely with periodic RDB) maximises throughput. The 500k events/sec on a single Redis is achievable only with no — everysec would still add measurable disk I/O overhead at that rate.
4. The measured throughput comes from a single-connection benchmark. Multi-connection benchmarks scale higher until the AOF write path saturates. On real workloads, the fsync policy's throughput impact is most pronounced when the writes are small (single-field XADD) and the pipeline depth is low.
5. The mixed pattern: run each workload on its own Redis instance. Do not try to share one Redis between clickstream and payment events — the fsync policy must serve the strictest workload, so the throughput cost applies to everything.

Output.

Workload	fsync policy	Throughput vs no	Data loss on crash
A: clickstream	everysec	~93%	up to 1 s
B: payments	always	~20–30%	0
C: metrics	no	100%	up to snapshot cadence

Rule of thumb. everysec is the default; move to always only for zero-loss workloads (payments, orders, ledger) and to no only for workloads where minutes of loss is acceptable (metrics, telemetry). Never share a Redis instance across workloads with different fsync requirements — segregate by instance.

Senior interview question on Redis Streams persistence

A senior interviewer might ask: "You're running Redis Streams for a payment event pipeline. The workload is 20k events per second, zero loss is required, and the SLO is p99 XADD latency under 10 ms. Walk me through the AOF + RDB configuration, the fsync policy, and how you'd monitor durability in production."

Solution Using appendonly + appendfsync always + monitored AOF write path

# /etc/redis/redis.conf — payment event pipeline
################################################################
# Persistence: zero loss required
################################################################
appendonly                 yes
appendfsync                always
no-appendfsync-on-rewrite  no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size   256mb
aof-load-truncated          yes
aof-use-rdb-preamble        yes

# RDB for periodic backup (belt-and-braces)
save 3600 1
save 300  100
save 60   10000
dbfilename dump.rdb
dir        /var/lib/redis

# Memory tuning for Streams workload
maxmemory                   16gb
maxmemory-policy            noeviction     # never evict payment data
stream-node-max-bytes       4096
stream-node-max-entries     100

# Client-facing
tcp-keepalive               60
timeout                     0

# monitor_persistence.py — Prometheus exporter for AOF durability
import redis, time
from prometheus_client import start_http_server, Gauge

r = redis.Redis(host="redis-payments.internal", decode_responses=True)

g_aof_enabled  = Gauge("redis_aof_enabled", "AOF on/off")
g_aof_pending  = Gauge("redis_aof_pending_bio_fsync", "pending fsync ops")
g_aof_last_ok  = Gauge("redis_aof_last_write_ok", "last AOF write status ok")
g_aof_size     = Gauge("redis_aof_current_size_bytes", "AOF current size")
g_rdb_last_s   = Gauge("redis_rdb_seconds_since_save", "seconds since RDB save")
g_fsync_lag_ms = Gauge("redis_aof_fsync_lag_ms", "fsync lag in ms")

def scrape():
    info = r.info("persistence")
    g_aof_enabled.set(info["aof_enabled"])
    g_aof_pending.set(info["aof_pending_bio_fsync"])
    g_aof_last_ok.set(1 if info["aof_last_write_status"] == "ok" else 0)
    g_aof_size.set(info["aof_current_size"])
    now = int(time.time())
    g_rdb_last_s.set(now - info["rdb_last_save_time"])
    # AOF fsync lag — from server info directly
    g_fsync_lag_ms.set(info.get("aof_delayed_fsync", 0))

if __name__ == "__main__":
    start_http_server(9127)
    while True:
        try:
            scrape()
        except Exception as e:
            print("scrape error:", e)
        time.sleep(10)

# prometheus_alerts.yml — durability alerts
groups:
- name: redis-payments-persistence
  rules:
  - alert: RedisAOFWriteFailed
    expr: redis_aof_last_write_ok == 0
    for: 30s
    labels: {severity: page}
    annotations:
      summary: AOF write failed — durability lost

  - alert: RedisAOFPendingFsync
    expr: redis_aof_pending_bio_fsync > 5
    for: 1m
    labels: {severity: page}
    annotations:
      summary: AOF fsync backlog > 5 — disk cannot keep up

  - alert: RedisRDBTooOld
    expr: redis_rdb_seconds_since_save > 7200
    for: 5m
    labels: {severity: warn}
    annotations:
      summary: RDB snapshot > 2h old — DR window widened

Step-by-step trace.

Layer	Choice	Reasoning
AOF	appendonly yes	Required for zero loss
fsync	appendfsync always	Every XADD durable before OK
RDB	5-min + 1-min + 1-h triggers	Backup snapshot for DR
Memory	16 GB, noeviction	Never evict payment entries
Stream tuning	stream-node-max-entries 100	Smaller radix nodes = faster fsync
Monitoring	Prometheus + 3 alerts	Detect write failures, fsync backlog, stale RDB
p99 XADD	~5–8 ms	Under 10 ms SLO on NVMe SSD
Throughput	~25k events/sec	Above 20k/sec target

After the rollout, the payment pipeline maintains zero-loss durability under 20k events/sec, p99 XADD latency of 5–8 ms (under 10 ms SLO), and the Prometheus alerts catch AOF write failures within 30 seconds. RDB snapshots run every 5 minutes for DR.

Output:

Metric	Value
Durability guarantee	Every XADD fsynced before OK
Worst-case data loss	0 events
p99 XADD latency	~5–8 ms
Throughput headroom	25k / 20k = 1.25×
RDB snapshot cadence	Every 5 min
Instance memory headroom	16 GB - ~4 GB working set = 12 GB

Why this works — concept by concept:

• appendfsync always for zero loss — every XADD is fsynced to disk before the client sees OK. This is the only correct choice for payment / order / ledger workloads. Throughput drops to ~30% of the no case, but 20k events/sec is well within a single NVMe SSD's sync-write budget.
• aof-use-rdb-preamble — the AOF is written as an RDB header + AOF tail, giving faster recovery than a pure AOF while still capturing every write. Modern Redis (>= 4) uses this by default; leaving it on is the correct choice.
• maxmemory-policy noeviction — with payments, we never want Redis to evict entries under memory pressure. noeviction returns an error to the writer instead of silently dropping data. The producer's response is to alert and shed load — never to lose events.
• Monitoring the AOF — the three alerts (write-failed, pending-fsync backlog, stale RDB) catch every durability failure mode. aof_last_write_status != ok is the direct signal; aof_pending_bio_fsync > 5 is the indirect signal that disk cannot keep up.
• Cost — appendfsync always costs roughly 70% of peak throughput on a hot workload, plus disk I/O bandwidth. For payment workloads at 20k events/sec, this is a rounding error on the infrastructure bill. The alternative — losing a payment on crash — has legal and financial consequences that dwarf the throughput cost.

Streaming
Topic — streaming
Streaming problems on Redis persistence and durability

Practice →

Optimization Topic — optimization Optimization problems on AOF fsync tuning

Practice →

---

5. Redis Streams vs Kafka + production patterns

Decide by workload — Streams for single-region sub-ms, Kafka for multi-region EOS + high throughput

The mental model in one line: redis stream vs kafka is a four-row decision matrix — latency (Streams sub-ms, Kafka ms), durability (Streams AOF, Kafka replicated log), throughput (Streams high, Kafka higher), and ops (Streams simple, Kafka complex) — with the deciding line drawn at single-region vs multi-region and at exactly-once-semantics requirements. Streams wins the "kafka lite" tier; Kafka wins everything above that scale.

The four-row decision matrix.

• Row 1 — latency. Streams: p99 XADD ~ 0.2 ms on a warm socket with everysec. Kafka: p99 produce ~ 2–5 ms with acks=all. For sub-100 ms end-to-end SLOs, Streams removes 1–5 ms of headroom.
• Row 2 — durability. Streams: AOF file on local disk, optionally replicated via Redis replication. Kafka: replicated log across N brokers; a message is durable when N replicas ack. Kafka is unambiguously more durable; Streams need external replication for the same guarantee.
• Row 3 — throughput. Streams: 100–300k events/sec per single-instance Redis, single-thread bound. Kafka: 1–5 M events/sec per broker cluster, scales linearly with brokers. At single-region 100k/sec scale, Streams is enough; above 500k/sec, Kafka is the answer.
• Row 4 — ops. Streams: one process, one AOF, one config file. Kafka: broker cluster + KRaft/ZooKeeper + Schema Registry + Kafka Connect + rebalance-aware clients. Small teams ship Streams and never see a controller-election page.

When Streams is the clear winner.

• Single-region deployment. No multi-region replication requirement.
• Sub-ms latency SLO. Clickstream, cache-warm, real-time UI events.
• Small platform team. 2–5 engineers who cannot dedicate one FTE to Kafka.
• Existing Redis stack. The team already runs Redis for caching; adding a Stream is one XADD call away.
• Moderate throughput. Up to ~300k events/sec per shard.

When Kafka is the clear winner.

• Multi-region deployment. MirrorMaker 2 for cross-region replication.
• Exactly-once semantics (EOS). Kafka transactions provide idempotent producers + consumer read-committed isolation. Streams do not.
• High throughput. Beyond ~500k events/sec — Kafka's per-partition parallelism scales linearly.
• Ecosystem depth. Kafka Connect, ksqlDB, Schema Registry, Streams DSL. Streams has none of these.
• Large platform team. 5+ engineers with Kafka expertise.

Cluster mode + hash-slotting streams.

• Redis Cluster. Redis's built-in horizontal sharding — 16384 hash slots distributed across master nodes.
• Stream keys. A stream is a single key; all its entries live on one shard. To spread load, use multiple stream keys with a hash-slot-tagged naming scheme.
• Hash tag. clickstream:{shard-0}, clickstream:{shard-1}, clickstream:{shard-2}, ... — the {...} tells Redis Cluster to hash only the tag, so all "shard-0" keys land on the same slot (useful for cross-key operations) and different tags land on different slots.
• Producer routing. Compute shard = tenant_id % N and XADD to clickstream:{shard-N}. Simple app-level sharding on top of Redis Cluster.
• Consumer routing. Each consumer subscribes to a subset of shards. Coordinate via a "shard assignment" system (Kubernetes StatefulSet ordinal, ZooKeeper watch, or a static config).

Dead-letter secondary-stream pattern.

• Primary stream. clickstream receives all live events.
• DLQ stream. clickstream:dlq receives entries that failed processing more than N times.
• Recovery worker. Polls XPENDING for entries idle > 30 s and delivery-count > 5; XADDs to :dlq with original ID + payload + reason; XACKs the original.
• Triage. A human (or another service) reads the DLQ periodically, inspects the payload, and either replays back to the main stream (after fixing the bug) or acknowledges permanent failure.

Senior interview signals — the four production patterns.

• XPENDING drain. The runbook step when a consumer dies mid-batch.
• MAXLEN sizing. Every producer sets it; wrong sizing OOMs the box.
• Memory-vs-durability trade-off. Named as the axis every persistence decision moves along.
• Cluster hash-slotting. Ship multi-key streams from day one if the throughput might exceed a single-shard ceiling.

Common interview probes on Streams-vs-Kafka.

• "Walk me through when you'd pick Streams over Kafka" — required answer.
• "How do you shard a Redis Stream?" — hash tag + tenant-hash routing.
• "What's the dead-letter pattern?" — secondary stream + delivery-count threshold.
• "Does Redis Streams support exactly-once?" — no; Kafka transactions are the only EOS story.

Worked example — the decision matrix for a real workload

Detailed explanation. A team is designing an event pipeline for a new product launch. The workload is 80k events/sec at peak, single region (us-east-1 only), no exactly-once requirement (idempotent consumers make retries safe), and the team of three engineers already runs Redis for caching. Walk through the four-row matrix and land on the decision.

• Latency requirement. p99 end-to-end under 100 ms.
• Durability requirement. Up to 1 s of loss on crash acceptable.
• Team. 3 backend engineers, familiar with Redis.
• Roadmap. No multi-region plans; no EOS requirement.

Question. Fill in the decision matrix. Land on Streams or Kafka. Justify each row.

Input.

Row	Streams	Kafka	Requirement
Latency	sub-ms	2–5 ms	< 100 ms end-to-end
Durability	AOF everysec	replicated log	1 s loss OK
Throughput	up to 300k/sec	1M+/sec	80k/sec
Ops	1 process	broker cluster	3-engineer team

Code.

Decision walkthrough — 80k events/sec, single-region, 3-engineer team
=====================================================================

Row 1 — latency
  Requirement: p99 end-to-end < 100 ms
  Streams: p99 XADD ~ 0.2 ms → plenty of budget
  Kafka:   p99 produce ~ 2–5 ms → also fine, but wastes 2–5 ms
  Winner:  Streams (or tie)

Row 2 — durability
  Requirement: 1 s of loss acceptable
  Streams: appendfsync everysec → matches spec exactly
  Kafka:   replicated log → over-engineered for this spec
  Winner:  Streams

Row 3 — throughput
  Requirement: 80k events/sec
  Streams: up to 300k/sec on a single shard → 3.75× headroom
  Kafka:   1M+/sec → 12× headroom (also fine)
  Winner:  Streams (both work; Streams is simpler)

Row 4 — ops
  Requirement: 3-engineer team, already runs Redis
  Streams: 1 process + 1 config file → 0 new ops surface
  Kafka:   broker cluster + KRaft + registry → 1 FTE minimum
  Winner:  Streams (unambiguously)

DECISION: Redis Streams

# The full Streams stack config
# redis.conf
appendonly            yes
appendfsync           everysec
maxmemory             16gb
maxmemory-policy      noeviction
stream-node-max-entries 100

# Producer (per app pod)
# pipeline every 10 ms, batch up to 1000 XADDs
# XADD clickstream "*" MAXLEN "~" 4800000 ...
# 80k events/sec × 60s retention = 4.8M entries

# Consumers (4-worker group)
XGROUP CREATE clickstream analytics "$" MKSTREAM
# each worker:
XREADGROUP GROUP analytics worker-N COUNT 200 BLOCK 5000 STREAMS clickstream ">"

Step-by-step explanation.

1. Row 1 — latency — falls decisively to Streams because sub-ms p99 XADD gives 100 ms of runway to the downstream consumer. Kafka would also meet the SLO but wastes 2–5 ms of headroom that could go into the consumer's work.
2. Row 2 — durability — the spec explicitly says "1 s of loss acceptable." appendfsync everysec matches this exactly; Kafka's replicated log is over-engineered for the spec. Do not pay for durability guarantees you do not need.
3. Row 3 — throughput — 80k events/sec is well within a single-Redis budget (300k/sec ceiling). No sharding required. Kafka's linear scaling to millions of events/sec is unused here.
4. Row 4 — ops — the deciding row. A 3-engineer team cannot support Kafka without hiring a platform engineer. Streams is one process + one config file + the existing Redis operational knowledge the team already has. Ops cost is the single biggest driver of the decision.
5. All four rows point to Streams; three of four unambiguously, one is a tie. Ship Streams. Revisit if the workload grows past 300k/sec (shard with hash tags) or if a multi-region requirement lands (then migrate to Kafka).

Output.

Row	Streams score	Kafka score	Winner
Latency	10	8	Streams (edge)
Durability	10	10	Tie
Throughput	9	10	Kafka (unused)
Ops	10	5	Streams (decisive)
Total	39/40	33/40	Streams

Rule of thumb. The four-row matrix decides most Streams-vs-Kafka questions. The ops row is the deciding factor for teams under 5 engineers; latency + team size = Streams. Kafka wins when multi-region, EOS, or > 500k events/sec lands on the roadmap.

Worked example — sharding streams across a Redis Cluster

Detailed explanation. A workload grows past a single-Redis ceiling (say, 400k events/sec). The scaling answer is to spread the stream across N shards via hash-tagged stream keys. Producers compute the shard from a tenant/user hash and XADD to the matching key. Consumers subscribe to their assigned shard. Walk through the design for 4 shards.

• Cluster. 4-shard Redis Cluster (each shard = 1 master + 1 replica).
• Stream keys. clickstream:{shard-0} through clickstream:{shard-3}.
• Producer routing. shard = hash(user_id) % 4.
• Consumer group. Per-shard consumer group (analytics-{shard-N}), 2 workers each.

Question. Write the producer helper and the consumer helper for 4-shard hash-tagged Streams.

Input.

Component	Value
Shards	4
Producer key template	clickstream:{shard-N}
Consumer group per shard	analytics-{shard-N}
Workers per shard	2
Total workers	8

Code.

import redis
import hashlib
from redis.cluster import RedisCluster, ClusterNode

# Connect to the Redis Cluster
rc = RedisCluster(
    startup_nodes=[
        ClusterNode("redis-0.internal", 6379),
        ClusterNode("redis-1.internal", 6379),
        ClusterNode("redis-2.internal", 6379),
        ClusterNode("redis-3.internal", 6379),
    ],
    decode_responses=True,
)

NUM_SHARDS = 4

def shard_for(user_id: str) -> int:
    h = hashlib.blake2b(user_id.encode(), digest_size=8).hexdigest()
    return int(h, 16) % NUM_SHARDS

def stream_key(shard: int) -> str:
    return f"clickstream:{{shard-{shard}}}"

def group_name(shard: int) -> str:
    return f"analytics-{{shard-{shard}}}"

# Producer helper
def emit(user_id: str, event: dict, per_shard_maxlen: int = 1_200_000):
    shard = shard_for(user_id)
    rc.xadd(
        stream_key(shard),
        fields=event,
        id="*",
        maxlen=per_shard_maxlen,
        approximate=True,
    )

# Consumer helper (each worker is assigned one shard)
def consumer_loop(assigned_shard: int, worker_name: str):
    key   = stream_key(assigned_shard)
    group = group_name(assigned_shard)
    try:
        rc.xgroup_create(key, group, id="$", mkstream=True)
    except redis.ResponseError as e:
        if "BUSYGROUP" not in str(e):
            raise

    # boot recovery
    while True:
        pending = rc.xreadgroup(group, worker_name, {key: "0"}, count=200)
        if not pending or not pending[0][1]:
            break
        for _s, entries in pending:
            for entry_id, fields in entries:
                if process(fields):
                    rc.xack(key, group, entry_id)

    # live
    while True:
        batch = rc.xreadgroup(group, worker_name, {key: ">"},
                               count=200, block=5000)
        if not batch:
            continue
        for _s, entries in batch:
            ids = []
            for entry_id, fields in entries:
                if process(fields):
                    ids.append(entry_id)
            if ids:
                rc.xack(key, group, *ids)

def process(fields): return True

# Kubernetes assigns each pod ordinal 0..7; workers 0,1 → shard 0; 2,3 → shard 1; ...
if __name__ == "__main__":
    import os
    ordinal = int(os.environ["POD_ORDINAL"])
    assigned = ordinal // 2
    consumer_loop(assigned, worker_name=f"worker-{ordinal}")

# Verify each shard key routes to a different slot
redis-cli --cluster keyslot 'clickstream:{shard-0}'
# (integer) 5060
redis-cli --cluster keyslot 'clickstream:{shard-1}'
# (integer) 12539
redis-cli --cluster keyslot 'clickstream:{shard-2}'
# (integer) 4363
redis-cli --cluster keyslot 'clickstream:{shard-3}'
# (integer) 2044
# → all different slots → distributed across the cluster

Step-by-step explanation.

1. The producer's shard_for(user_id) hashes the user ID (blake2b for speed and determinism) and modulos to the shard count. Consistent hashing across producers means every event for a given user lands on the same shard.
2. The stream_key uses the {shard-N} hash-tag syntax. Redis Cluster's slot calculation hashes only the substring inside {}, so clickstream:{shard-0}, enrichment:{shard-0}, and sessions:{shard-0} all land on the same shard — useful when consumers need to read multiple streams for the same shard atomically. Different {shard-N} values land on different slots, distributing load across the cluster.
3. The consumer's assignment strategy uses the Kubernetes StatefulSet pod ordinal to derive the shard. Pod 0 and 1 handle shard 0, pods 2 and 3 handle shard 1, etc. This is a static assignment; more sophisticated setups use a consensus service (Kubernetes lease, Consul, etcd) for dynamic rebalancing.
4. Each consumer runs its own consumer group per shard (analytics-{shard-N}). PELs are per-group, so a worker dying in shard 0 does not affect shard 1's PEL. This gives per-shard fault isolation.
5. The per_shard_maxlen = 1_200_000 bounds each shard's memory. 4 shards × 1.2M entries × 250 bytes ≈ 1.2 GB total — well within a 4-node cluster's budget. The total capacity is 4× a single-shard deployment.

Output.

Aspect	Single-shard	4-shard cluster
Throughput ceiling	~300k/sec	~1.2M/sec
Memory footprint	1× per stream	4× (spread)
Fault isolation	shared	per-shard
Consumer complexity	1 group	N groups, static assignment
Cross-shard queries	N/A	app-side fan-out

Rule of thumb. For workloads that will grow past ~300k events/sec, design the stream key with a {shard-N} hash tag from day one. Migrating a single-key stream to a hash-tagged multi-key stream after launch is significantly harder than starting with the tag in place — even if you only use one shard initially.

Worked example — the dead-letter secondary-stream pattern

Detailed explanation. A senior engineer's canonical DLQ pattern with Redis Streams is a secondary stream (<name>:dlq) that the recovery worker XADDs poison-pill entries to. The DLQ stream is a full-featured Redis Stream — not a Redis List — so consumers can XREADGROUP it, replay from historical IDs, and inspect PEL health. Walk through the full pattern with an example poison entry.

• Trigger. Recovery worker sees delivery_count > 5.
• Action. XADD to :dlq with original_id + reason + serialised payload.
• Consumer. A DLQ triage service XREADGROUPs from :dlq to email humans or write a Jira ticket.

Question. Show the recovery worker code that routes a poison entry to :dlq, plus the DLQ triage service that reads from it.

Input.

Component	Value
Primary stream	clickstream
Group	enrichment
DLQ stream	clickstream:dlq
DLQ triage group	dlq-triage
Poison threshold	delivery_count > 5

Code.

import redis
import json
from datetime import datetime

r = redis.Redis(host="cache-1.internal", decode_responses=True)

STREAM        = "clickstream"
GROUP         = "enrichment"
DLQ_STREAM    = "clickstream:dlq"

def route_to_dlq(entry_id: str, fields: dict, delivery_count: int, reason: str):
    """XADD to :dlq stream with full context, then XACK the original."""
    dlq_id = r.xadd(
        DLQ_STREAM,
        {
            "original_id":     entry_id,
            "original_stream": STREAM,
            "original_group":  GROUP,
            "delivery_count":  delivery_count,
            "reason":          reason,
            "routed_at":       datetime.utcnow().isoformat() + "Z",
            "payload":         json.dumps(fields),
        },
        id="*",
        maxlen=1_000_000,
        approximate=True,
    )
    r.xack(STREAM, GROUP, entry_id)
    return dlq_id

# DLQ triage service — 1 worker
def dlq_triage_loop():
    r.xgroup_create(DLQ_STREAM, "dlq-triage", id="$", mkstream=True)
    while True:
        batch = r.xreadgroup(
            "dlq-triage", "triage-1",
            {DLQ_STREAM: ">"},
            count=10,
            block=10_000,
        )
        if not batch:
            continue
        for _s, entries in batch:
            for entry_id, dlq_fields in entries:
                triage_notify(dlq_fields)     # email / Jira
                r.xack(DLQ_STREAM, "dlq-triage", entry_id)

def triage_notify(dlq_fields: dict):
    print(f"DLQ entry: {dlq_fields['original_id']} "
          f"reason={dlq_fields['reason']} "
          f"delivery_count={dlq_fields['delivery_count']}")
    # send to PagerDuty / Slack / Jira

# Inspect DLQ backlog
redis-cli XLEN clickstream:dlq
# (integer) 47      # 47 entries currently in DLQ

# Read most recent DLQ entries
redis-cli XREVRANGE clickstream:dlq + - COUNT 5
# Shows the 5 most recent poison entries

# Replay one DLQ entry back to primary (after fixing the bug)
redis-cli XADD clickstream "*" $(redis-cli XREVRANGE clickstream:dlq + - COUNT 1 | tail -n +2 | sed 's/^\s*//')
# then XACK the DLQ entry:
redis-cli XACK clickstream:dlq dlq-triage <dlq_entry_id>

Step-by-step explanation.

1. When the recovery worker detects a poison entry (delivery_count > 5), it calls route_to_dlq(entry_id, fields, delivery_count, reason). The function XADDs the full context — original ID, reason, delivery count, timestamp, serialised payload — to the :dlq stream.
2. The DLQ entry gets its own auto-generated ID. The original entry ID is preserved in the original_id field so triage can trace back to the primary stream if needed. The payload field holds the JSON-serialised original fields, so the entry is self-contained even if the primary stream is trimmed.
3. Immediately after the DLQ XADD, the recovery worker XACKs the original entry. This removes it from the primary group's PEL and stops the redelivery loop. The entry is now "handled" from the primary consumers' perspective — the DLQ is the record of what happened.
4. The DLQ triage service is a separate small consumer that XREADGROUPs from :dlq and does whatever the ops policy requires — email PagerDuty, open a Jira ticket, log to a data-warehouse table for post-mortem analysis. In simple deployments the "triage" is just a Slack channel notification.
5. Replay is optional. After the bug that caused the poison is fixed, an operator can XADD the payload back to the primary stream (a fresh XADD, not a resurrection of the old entry) and XACK the DLQ entry. In practice, most DLQ entries are simply reviewed and discarded — the bug is the fix, not the replay.

Output.

Step	Action	Effect
Poison detected	recovery worker XPENDING scan	delivery_count > 5 identified
Route to DLQ	route_to_dlq(...)	XADD :dlq + XACK primary
Triage read	dlq_triage_loop XREADGROUP	Entry visible to humans
Notify	PagerDuty / Slack / Jira	On-call sees poison entry
Optional replay	XADD primary from payload	New XADD; original stays in DLQ
Close	XACK DLQ entry	DLQ PEL clears

Rule of thumb. The DLQ is another Redis Stream, not a list — you get all the XREADGROUP / XPENDING / PEL semantics for free. Every consumer group in production has a :dlq counterpart. Interviewers will ask for this pattern by name; being able to sketch it in 30 lines is a strong senior signal.

Senior interview question on Streams vs Kafka + production patterns

A senior interviewer might ask: "You have a real-time clickstream at 400k events/sec across two regions, an existing Redis stack, and a fairly small platform team of four. Walk me through whether you'd start with Redis Streams (sharded via cluster) or Kafka, and how the decision changes if the throughput doubles or if a strict exactly-once requirement lands."

Solution Using a hybrid — Streams for single-region hot path, Kafka for cross-region backbone

Decision — 400k evt/sec, 2 regions, 4-engineer team
====================================================

Step 1 — decompose the workload
   Hot path (per-region ingest → per-region consumers): 400k/sec, sub-ms
   Cold path (per-region → cross-region durable log):   400k/sec, minutes lag OK

Step 2 — match tool to path
   Hot path → Redis Streams sharded 2x (per region)
              200k/sec per shard is well under 300k ceiling
              consumer groups + AOF everysec + DLQ pattern
   Cold path → Kafka (single global cluster)
              MirrorMaker 2 not needed; a single global cluster is simpler
              exactly-once for cross-region ETL if needed

Step 3 — bridge the two
   Region A Streams → Kafka connector process (per region)
     - XREADGROUP from clickstream:{shard-0} + clickstream:{shard-1}
     - produce to Kafka topic `clickstream` with acks=all
     - XACK only after Kafka produce confirmed
   Region B Streams → same bridge, different consumer names

Step 4 — throughput sensitivity
   IF workload doubles to 800k/sec per region:
      shard 4x per region → 200k/sec per shard, still OK
      Kafka bridge scales linearly with partitions
   IF EOS required end-to-end:
      Streams cannot provide EOS → move hot path to Kafka
      lose the sub-ms latency (accept 2–5 ms) but gain EOS

# Bridge — Streams → Kafka
import redis
from confluent_kafka import Producer

r = redis.Redis(host="redis-region-a.internal", decode_responses=True)
kafka = Producer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "acks": "all",
    "enable.idempotence": True,   # dedupe within a producer session
    "compression.type": "lz4",
    "linger.ms": 20,
})

STREAM = "clickstream:{shard-0}"
GROUP  = "kafka-bridge"

def bridge():
    r.xgroup_create(STREAM, GROUP, id="$", mkstream=True)
    while True:
        batch = r.xreadgroup(GROUP, "bridge-a", {STREAM: ">"},
                             count=500, block=5000)
        if not batch:
            continue
        acked = []
        for _s, entries in batch:
            for entry_id, fields in entries:
                # produce to Kafka; ack Streams only on delivery
                def on_delivery(err, msg, eid=entry_id):
                    if err is None:
                        acked.append(eid)
                kafka.produce(
                    "clickstream",
                    key=fields.get("user_id"),
                    value=b"".join(f"{k}={v};".encode() for k, v in fields.items()),
                    callback=on_delivery,
                )
        kafka.flush(timeout=5.0)
        if acked:
            r.xack(STREAM, GROUP, *acked)

Step-by-step trace.

Layer	Choice	Reasoning
Per-region hot path	Redis Streams (2 shards each)	sub-ms latency; 200k/sec/shard
Cross-region backbone	Kafka global cluster	replicated log; EOS optional
Bridge	Per-region Streams→Kafka process	XREADGROUP + kafka.produce + XACK on delivery
Producer regions	2 regions × 2 shards = 4 producer paths	independent PELs
Consumer regions	Per-region analytics + global Kafka consumers	mix real-time and batch
DLQ	Per-region Streams :dlq + Kafka DLT	dead-letter at both layers

After the rollout, the hot path retains sub-ms XADD latency in each region while the cold path gains cross-region durability via Kafka. The bridge is roughly 100 lines of Python per region; the total operational overhead is one Redis Cluster + one Kafka cluster + two bridge processes.

Output:

Metric	Streams-only	Kafka-only	Hybrid (chosen)
Hot-path p99 latency	0.2 ms	3 ms	0.2 ms
Cross-region durability	manual	native	native (via bridge)
Ops surface	2 Redis clusters + bridges	Kafka + MirrorMaker	Redis + Kafka + bridges
Team size to operate	2–3	4–5	3–4
Fits 4-engineer team?	yes	tight	yes

Why this works — concept by concept:

• Decompose by latency requirement — the hot path (per-region, sub-ms) is a different workload than the cold path (cross-region, minutes acceptable). Force-fitting both into one tool sacrifices the strengths of the other. Streams for hot, Kafka for cold.
• Bridge on XACK-after-produce — the bridge only XACKs the Redis entry after Kafka has confirmed the produce (acks=all + enable.idempotence=true). This is at-least-once delivery from Streams into Kafka; the idempotent producer + Kafka transactional consumer can achieve EOS downstream if needed.
• Shard early — starting at 2 shards per region even before hitting the single-shard ceiling means the throughput-doubling scenario is a NUM_SHARDS = 4 config change, not a re-architecture. Sharding after the fact is significantly more expensive.
• DLQ at both layers — Streams :dlq handles per-region poison pills; Kafka's dead-letter topic handles bridge failures. Two layers of DLQ = two layers of triage; the on-call runbook covers both.
• Cost — one Redis Cluster (~4 nodes) per region + one global Kafka cluster (~6 nodes) + 2 bridge processes. Infrastructure cost is roughly 2× Streams-only but 0.6× Kafka-only. The engineering cost is comparable to Kafka-only; the operational cost sits between. The result is the right latency at each hop and the right durability everywhere.

Streaming
Topic — streaming
Streaming problems on Streams-vs-Kafka decisions

Practice →

Optimization
Topic — optimization
Optimization problems on sharded stream throughput

Practice →

---

Cheat sheet — Redis Streams recipes

• XADD + MAXLEN ~ template. XADD <stream> "*" MAXLEN "~" <cap> <field> <value> .... Always include MAXLEN with the ~ (approximate) modifier for O(1) amortised trim. Compute cap = retention_seconds × events_per_second; add 20% headroom. Pipeline in batches of 500–1000 for high-throughput producers; single per-event XADDs cap at ~20k/sec over TCP due to round-trip cost.

• XREADGROUP consumer skeleton. Boot with XGROUP CREATE <stream> <group> "$" MKSTREAM. Recovery loop: XREADGROUP GROUP <g> <c> COUNT 100 STREAMS <stream> "0" drains this consumer's PEL. Steady state: XREADGROUP GROUP <g> <c> COUNT 100 BLOCK 5000 STREAMS <stream> ">" — > means never-delivered, BLOCK 5000 gives sub-ms wake latency plus a 5-second heartbeat. XACK on success in batches: XACK <stream> <group> <id1> <id2> ....

• XPENDING + XCLAIM idle-recovery. Recovery worker runs every 5 seconds: XAUTOCLAIM <stream> <group> <recovery-worker> 30000 0 COUNT 100. Idle threshold of 30 seconds catches dead consumers without stealing from healthy but slow ones. Delivery-count > 5 → route to :dlq stream with XADD <stream>:dlq "*" original_id <id> reason <reason> payload <json> then XACK the original.

• Cluster-mode stream layout. Hash-tag stream keys as <name>:{shard-N} so different N values map to different slots. Producer: shard = hash(user_id) % NUM_SHARDS. Consumer group per shard: <group>:{shard-N}. Static assignment via Kubernetes StatefulSet ordinal or dynamic via Consul/etcd. Cross-shard queries are app-layer fan-out; native cross-shard is not supported.

• appendonly / fsync config. appendonly yes + appendfsync everysec is the pragmatic default — 1 s of loss on crash, 5% throughput cost vs no. Switch to always for payment/order/ledger workloads (zero loss, ~30% throughput cost). Never use no for anything you cannot lose. Keep RDB too — save 3600 1 300 100 60 10000 for periodic backup snapshots.

• Consumer boot flow. Every consumer boot follows: (1) XGROUP CREATE (idempotent — swallow BUSYGROUP error), (2) drain PEL via XREADGROUP {stream: 0} until empty, (3) transition to steady-state XREADGROUP {stream: >} with BLOCK. Skipping step 2 leaks entries into the PEL forever.

• Dead-letter secondary stream. <stream>:dlq receives poison entries via XADD <stream>:dlq "*" original_id <id> original_stream <stream> reason <r> delivery_count <n> payload <json>. Cap with MAXLEN ~ 1_000_000. A separate triage consumer group reads it, notifies humans (PagerDuty / Slack / Jira), and optionally replays entries back to the main stream after the bug is fixed.

• Stream memory estimation. Roughly 250 bytes × entry_count for typical clickstream payloads (~180 byte payload + radix-tree overhead). For a 60-second window at 100k events/sec: 100_000 × 60 × 250 = 1.5 GB. Cap MAXLEN accordingly; verify with MEMORY USAGE <stream-key> after the first hour.

• Prometheus alerts. (a) redis_stream_pel_count > 10000 for 2m — likely dead consumer. (b) redis_stream_consumer_idle_ms > 60000 for 1m — confirmed dead. (c) rate(redis_stream_dlq_length[5m]) > 1 for 2m — poison-pill storm. (d) redis_aof_last_write_ok == 0 for 30s — durability lost. (e) redis_aof_pending_bio_fsync > 5 for 1m — disk saturating.

• BLOCK never = 0. XREAD BLOCK 0 blocks forever; the process becomes hard to stop cleanly. Always use BLOCK 1000 to BLOCK 5000 — sub-ms wake latency plus periodic heartbeat to emit metrics and check shutdown flags.

• Streams-vs-Kafka four-row matrix. Streams wins on latency + ops; Kafka wins on multi-region + EOS + throughput above 500k/sec. For single-region under 300k/sec on a small team, Streams is unambiguously the right choice. For multi-region or EOS, Kafka is unambiguously the right choice. The middle is a hybrid — Streams for hot path, Kafka for cross-region backbone.

• stream-node-max-entries tuning. Redis stores stream entries in radix-tree nodes; each node holds up to stream-node-max-entries (default 100) or stream-node-max-bytes (default 4096) entries. Larger nodes = better compression + fewer nodes; smaller nodes = faster fsync. Payment workloads: 100 entries. Metrics workloads: 500 entries.

• Idempotent consumers. Because Streams is at-least-once, every consumer must be idempotent — either dedupe by event ID before applying side effects, or design side effects to be safe on replay. Non-idempotent consumers are the #1 cause of "why did we bill twice?" incidents.

• NOMKSTREAM for prod producers. XADD <stream> NOMKSTREAM "*" ... fails the append if the stream key does not exist. Useful for pipelines where the consumer registers the stream first; catches "producer typo'd the stream name" bugs immediately instead of silently creating a phantom stream.

Frequently asked questions

What is Redis Streams and how is it different from Redis Pub/Sub?

redis streams is an ordered, append-only log data structure introduced in Redis 5.0 — every entry is a small hash of field/value pairs identified by a <ms>-<seq> ID that is strictly monotonically increasing, stored in memory as a radix tree, and optionally persisted via AOF/RDB. Every XADD is durable (in memory), every XREAD is a cursor-based read that returns historical entries, and consumer groups add server-side fan-out with per-entry acknowledgement via XACK. Pub/Sub is fundamentally different: it is a fire-and-forget broadcast bus with in-flight-only semantics — a PUBLISH goes to every currently-subscribed client and is not stored anywhere. If no client is subscribed at the exact moment of publish, the message is lost forever. For ingest pipelines, real-time ETL, cache-warm queues, or anything where losing an event is a bug, use Streams. Pub/Sub is only appropriate for websocket-style fan-out to connected clients (cache-invalidation broadcasts, live dashboard updates) where in-flight-only semantics are acceptable.

redis stream vs kafka — which should I pick for a real-time pipeline in 2026?

Decide by the four-row matrix — latency, durability, throughput, ops. Redis Streams wins on latency (p99 XADD ~0.2 ms) and ops (one process, existing Redis stack, no ZooKeeper / KRaft / Schema Registry). Kafka wins on multi-region durability (replicated log across brokers + MirrorMaker 2), exactly-once semantics (Kafka transactions), and throughput above ~500k events/sec (per-partition parallelism). The 2026 default: for single-region deployments under ~300k events/sec on a team of fewer than 5 engineers, Streams is unambiguously the right choice. For multi-region deployments, exactly-once requirements (payment settlements, order-book replicas), or throughput above 500k/sec, Kafka is unambiguously the right choice. Between those extremes, a hybrid — Streams for the per-region hot path, Kafka as a global backbone via a per-region bridge — captures the best of both. Do not force-fit either tool into the other's shape; each is optimised for a different physical model.

What is XPENDING and what do the delivery-count and idle-time columns mean?

XPENDING <stream> <group> reads the group's pending-entry list (PEL) — the server-side record of entries delivered to consumers via XREADGROUP but not yet acknowledged via XACK. The summary form returns (total_pel_count, min_id, max_id, [(consumer_name, count), ...]); the detail form (XPENDING <stream> <group> [IDLE <ms>] <start> <end> <count> [<consumer>]) returns per-entry (entry_id, owner_consumer, idle_ms, delivery_count). idle_ms is the milliseconds since the entry was last delivered — high values (> 30 s) mean the owner consumer has probably died or stalled. delivery_count is the total number of times the entry has been delivered (initial delivery + every subsequent XCLAIM / XAUTOCLAIM / explicit-ID redelivery); values above 5 signal a poison pill that should be routed to a :dlq stream instead of endlessly cycling. The runbook is standard: XPENDING summary → per-consumer breakdown → per-entry detail with IDLE filter → XAUTOCLAIM for reassignment or XADD :dlq + XACK for poison routing.

Are Redis Streams durable, and what does appendfsync everysec really guarantee?

Yes — with appendonly yes (AOF enabled), every XADD is durable within the fsync window defined by appendfsync. redis appendonly with appendfsync always fsyncs the AOF after every write command — zero loss on crash but roughly 30% of peak throughput. appendfsync everysec fsyncs the AOF once per second — worst-case loss of 1 second of writes on kernel panic, but throughput within 5% of the "no fsync" case. appendfsync no never explicitly fsyncs; the OS decides when to flush, giving maximum throughput but up to 30 seconds of loss on crash. The pragmatic default for clickstream, analytics, and cache-warm workloads is everysec; for payment / order / ledger workloads switch to always; for metrics / telemetry where minutes of loss is fine, no or AOF-off + periodic RDB is acceptable. On top of AOF, keep RDB snapshots enabled via save 3600 1 300 100 60 10000 for point-in-time backup and disaster recovery — the two mechanisms complement each other.

How do I handle a slow consumer without letting its lag stall the whole consumer group?

A slow consumer in a redis consumer group accumulates entries in its PEL — the entries are delivered but never acknowledged. Because Redis delivers each entry to exactly one consumer in the group (not all consumers), slow-consumer lag is contained to that consumer's PEL and does not directly block the others. However, if left unbounded, the growing PEL wastes memory and hides real dead-consumer signals. The pattern: (1) monitor per-consumer PEL count via XPENDING <stream> <group> — alert on any consumer whose PEL is > 10× the group median; (2) run a recovery worker that XAUTOCLAIMs entries idle > 30 seconds so a dying consumer's backlog reassigns to healthy peers; (3) if a specific consumer name is chronically slow, delete it with XGROUP DELCONSUMER and rely on the recovery worker to reassign its PEL; (4) design consumers to be idempotent so reassignment via XCLAIM does not double-apply side effects. Never raise the group's XREADGROUP COUNT to "let it catch up" — that just gives the slow consumer more entries to fail on.

Can I shard a Redis stream across a Redis Cluster and still get consumer-group semantics?

Yes, but with the constraint that a single stream key lives on a single Redis Cluster shard — you cannot spread one stream key across multiple shards. The pattern is to use multiple stream keys with hash-tagged names: clickstream:{shard-0}, clickstream:{shard-1}, ..., where the {shard-N} hash tag tells Redis Cluster to route by the tag alone. Different N values land on different slots (distributing load), while related keys with the same tag (e.g., clickstream:{shard-0} and enrichment:{shard-0}) land on the same slot (enabling cross-key operations on the same shard). Producers compute shard = hash(user_id) % NUM_SHARDS and XADD to the matching key. Consumers run per-shard consumer groups (analytics:{shard-N}) with their own PELs and their own recovery workers — fault isolation is per-shard. Cross-shard queries (e.g., aggregating events across all shards) are app-layer fan-out: read from each shard, merge in the application, no native cross-shard primitive. This is the same architectural trade-off Kafka partitioning makes; the difference is that Redis Cluster's hash-tag mechanism is more explicit and the app-side routing code is a few lines instead of an entire Kafka consumer library.

Practice on PipeCode

• Drill the streaming practice library → for the XADD, XREADGROUP, PEL failover, and dead-letter problems senior interviewers love.
• Rehearse on the ETL practice library → for the real-time ingest, batching, and persistence-tuning problems that motivate Streams in the first place.
• Sharpen the tuning axis with the optimization practice library → for the MAXLEN sizing, fsync-policy, and sharded-throughput problems.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the Streams-vs-Kafka intuition against real graded inputs.

Lock in Redis Streams muscle memory

Redis docs explain commands. PipeCode drills explain the decision — when XREADGROUP beats XREAD, when appendfsync always beats everysec, when Streams beats Kafka. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice streaming problems →
Practice optimization problems →