DEV Community

Aman Kumar
Aman Kumar

Posted on Originally published at dev.to

Event Streaming at Scale: Kafka vs Redis Streams for High-Throughput Python Backends

Architectural Trade-Offs

When designing high-throughput event streaming architectures in Python, software engineers typically evaluate two leading storage engines: Apache Kafka and Redis Streams. While both support append-only logs and consumer groups, their fundamental concurrency models and memory bounds differ significantly.

Kafka vs Redis Streams Core Matrix

Feature Apache Kafka Redis Streams (XADD/XREADGROUP)
Storage Architecture Disk-first distributed commit log (PageCache) In-memory radix tree with optional disk persistence (RDB/AOF)
Throughput / Latency Millions msgs/sec, 5-15ms P99 latency 100k-500k msgs/sec, sub-millisecond P99 latency (<1ms)
Partitioning Distributed partitions per topic across brokers Single key log (partitioning requires multiple Redis keys)
Retention Policy Days/Months via segment rollover & compaction Memory-bounded with MAXLEN ~ eviction
Python Client Eco aiokafka, confluent-kafka redis.asyncio

Production Architecture

+--------------------+        PubSub / Produce         +------------------------+
| Python Microservice| =============================> | Stream Message Broker  |
| (FastAPI Worker)   |                                | (Kafka or Redis Streams)|
+--------------------+                                +------------------------+
                                                                  |
                                                     Consumer Group Partitioning
                                                                  v
+--------------------+                                +------------------------+
| Storage / Analytics| <============================= | Competing Async Workers|
| (TimescaleDB / S3) |         Ack / Commit           | (Consumer Group XREAD) |
+--------------------+                                +------------------------+
Enter fullscreen mode Exit fullscreen mode

Production Implementation: Redis Streams Async Consumer

The following working code demonstrates a fault-tolerant Python consumer utilizing redis.asyncio with explicit consumer group acknowledgement and graceful shutdown handling:

from __future__ import annotations

import asyncio
import logging
import signal
from typing import Any

import redis.asyncio as aioredis

logger = logging.getLogger("stream_consumer")
logging.basicConfig(level=logging.INFO)

STREAM_NAME = "events:pipeline:v1"
GROUP_NAME = "analytics_processors"
CONSUMER_NAME = "worker_node_01"


class RedisStreamConsumer:
    def __init__(self, redis_url: str = "redis://localhost:6379/0"):
        self.redis_url = redis_url
        self.client: aioredis.Redis | None = None
        self._running = False

    async def connect(self) -> None:
        self.client = aioredis.from_url(
            self.redis_url,
            encoding="utf-8",
            decode_responses=True,
        )
        # Ensure consumer group exists
        try:
            await self.client.xgroup_create(
                STREAM_NAME, GROUP_NAME, id="0", mkstream=True
            )
            logger.info("Created consumer group '%s' on stream '%s'", GROUP_NAME, STREAM_NAME)
        except aioredis.ResponseError as exc:
            if "BUSYGROUP" not in str(exc):
                raise
            logger.info("Consumer group '%s' already registered", GROUP_NAME)

    async def process_message(self, message_id: str, fields: dict[str, Any]) -> None:
        """Simulates idempotent processing of the ingested stream payload."""
        event_type = fields.get("type", "unknown")
        user_id = fields.get("user_id", "anonymous")
        logger.debug("Processing event %s for user %s (ID: %s)", event_type, user_id, message_id)
        # Simulated async I/O operation
        await asyncio.sleep(0.01)

    async def run_loop(self) -> None:
        self._running = True
        logger.info("Starting stream consumer loop for %s...", CONSUMER_NAME)

        while self._running:
            try:
                # Read new unread messages with 2000ms block
                entries = await self.client.xreadgroup(
                    groupname=GROUP_NAME,
                    consumername=CONSUMER_NAME,
                    streams={STREAM_NAME: ">"},
                    count=50,
                    block=2000,
                )

                if not entries:
                    continue

                for stream_key, messages in entries:
                    for msg_id, data in messages:
                        try:
                            await self.process_message(msg_id, data)
                            # Explicit acknowledgement
                            await self.client.xack(STREAM_NAME, GROUP_NAME, msg_id)
                        except Exception as proc_err:
                            logger.error("Failed processing message %s: %s", msg_id, proc_err)
                            # Message remains in Pending Entries List (PEL) for retry

            except asyncio.CancelledError:
                break
            except Exception as loop_err:
                logger.error("Unexpected stream consumer loop error: %s", loop_err)
                await asyncio.sleep(1.0)

    async def shutdown(self) -> None:
        logger.info("Shutting down consumer...")
        self._running = False
        if self.client:
            await self.client.aclose()


async def main() -> None:
    consumer = RedisStreamConsumer()
    await consumer.connect()

    loop = asyncio.get_running_loop()
    for sig in (signal.SIGINT, signal.SIGTERM):
        loop.add_signal_handler(sig, lambda: asyncio.create_task(consumer.shutdown()))

    await consumer.run_loop()


if __name__ == "__main__":
    asyncio.run(main())
Enter fullscreen mode Exit fullscreen mode

When to Choose Which?

  1. Choose Redis Streams when latency is critical (sub-millisecond), dataset fits in RAM or is bounded with MAXLEN, and your stack is already centered around Redis.
  2. Choose Apache Kafka when retaining terabytes of historical logs, requiring strict partition order across multiple brokers, or building large-scale CDC (Change Data Capture) pipelines.

GitHub Repository: github.com/aman179102

Top comments (0)