Architectural Trade-Offs
When designing high-throughput event streaming architectures in Python, software engineers typically evaluate two leading storage engines: Apache Kafka and Redis Streams. While both support append-only logs and consumer groups, their fundamental concurrency models and memory bounds differ significantly.
Kafka vs Redis Streams Core Matrix
| Feature | Apache Kafka | Redis Streams (XADD/XREADGROUP) |
|---|---|---|
| Storage Architecture | Disk-first distributed commit log (PageCache) | In-memory radix tree with optional disk persistence (RDB/AOF) |
| Throughput / Latency | Millions msgs/sec, 5-15ms P99 latency | 100k-500k msgs/sec, sub-millisecond P99 latency (<1ms) |
| Partitioning | Distributed partitions per topic across brokers | Single key log (partitioning requires multiple Redis keys) |
| Retention Policy | Days/Months via segment rollover & compaction | Memory-bounded with MAXLEN ~ eviction |
| Python Client Eco |
aiokafka, confluent-kafka
|
redis.asyncio |
Production Architecture
+--------------------+ PubSub / Produce +------------------------+
| Python Microservice| =============================> | Stream Message Broker |
| (FastAPI Worker) | | (Kafka or Redis Streams)|
+--------------------+ +------------------------+
|
Consumer Group Partitioning
v
+--------------------+ +------------------------+
| Storage / Analytics| <============================= | Competing Async Workers|
| (TimescaleDB / S3) | Ack / Commit | (Consumer Group XREAD) |
+--------------------+ +------------------------+
Production Implementation: Redis Streams Async Consumer
The following working code demonstrates a fault-tolerant Python consumer utilizing redis.asyncio with explicit consumer group acknowledgement and graceful shutdown handling:
from __future__ import annotations
import asyncio
import logging
import signal
from typing import Any
import redis.asyncio as aioredis
logger = logging.getLogger("stream_consumer")
logging.basicConfig(level=logging.INFO)
STREAM_NAME = "events:pipeline:v1"
GROUP_NAME = "analytics_processors"
CONSUMER_NAME = "worker_node_01"
class RedisStreamConsumer:
def __init__(self, redis_url: str = "redis://localhost:6379/0"):
self.redis_url = redis_url
self.client: aioredis.Redis | None = None
self._running = False
async def connect(self) -> None:
self.client = aioredis.from_url(
self.redis_url,
encoding="utf-8",
decode_responses=True,
)
# Ensure consumer group exists
try:
await self.client.xgroup_create(
STREAM_NAME, GROUP_NAME, id="0", mkstream=True
)
logger.info("Created consumer group '%s' on stream '%s'", GROUP_NAME, STREAM_NAME)
except aioredis.ResponseError as exc:
if "BUSYGROUP" not in str(exc):
raise
logger.info("Consumer group '%s' already registered", GROUP_NAME)
async def process_message(self, message_id: str, fields: dict[str, Any]) -> None:
"""Simulates idempotent processing of the ingested stream payload."""
event_type = fields.get("type", "unknown")
user_id = fields.get("user_id", "anonymous")
logger.debug("Processing event %s for user %s (ID: %s)", event_type, user_id, message_id)
# Simulated async I/O operation
await asyncio.sleep(0.01)
async def run_loop(self) -> None:
self._running = True
logger.info("Starting stream consumer loop for %s...", CONSUMER_NAME)
while self._running:
try:
# Read new unread messages with 2000ms block
entries = await self.client.xreadgroup(
groupname=GROUP_NAME,
consumername=CONSUMER_NAME,
streams={STREAM_NAME: ">"},
count=50,
block=2000,
)
if not entries:
continue
for stream_key, messages in entries:
for msg_id, data in messages:
try:
await self.process_message(msg_id, data)
# Explicit acknowledgement
await self.client.xack(STREAM_NAME, GROUP_NAME, msg_id)
except Exception as proc_err:
logger.error("Failed processing message %s: %s", msg_id, proc_err)
# Message remains in Pending Entries List (PEL) for retry
except asyncio.CancelledError:
break
except Exception as loop_err:
logger.error("Unexpected stream consumer loop error: %s", loop_err)
await asyncio.sleep(1.0)
async def shutdown(self) -> None:
logger.info("Shutting down consumer...")
self._running = False
if self.client:
await self.client.aclose()
async def main() -> None:
consumer = RedisStreamConsumer()
await consumer.connect()
loop = asyncio.get_running_loop()
for sig in (signal.SIGINT, signal.SIGTERM):
loop.add_signal_handler(sig, lambda: asyncio.create_task(consumer.shutdown()))
await consumer.run_loop()
if __name__ == "__main__":
asyncio.run(main())
When to Choose Which?
-
Choose Redis Streams when latency is critical (sub-millisecond), dataset fits in RAM or is bounded with
MAXLEN, and your stack is already centered around Redis. - Choose Apache Kafka when retaining terabytes of historical logs, requiring strict partition order across multiple brokers, or building large-scale CDC (Change Data Capture) pipelines.
GitHub Repository: github.com/aman179102
Top comments (0)