DEV Community

Ayi NEDJIMI
Ayi NEDJIMI

Posted on

How to Build an Event-Driven Architecture with Minimal Infrastructure

Most teams reach for Kafka or RabbitMQ the moment "event-driven" appears in a design meeting. That is often premature. For services handling tens of thousands of events per day — not millions — a leaner stack delivers the same decoupling benefits with far less operational burden.

This article walks through building an event-driven system incrementally: starting with an in-process bus, graduating to Redis Streams, and landing on the patterns that make the whole thing reliable.

Why Event-Driven Matters (and When It Does Not)

Event-driven architecture solves a real problem: service coupling. When service A calls service B directly, A must know B's interface, its availability, and its latency budget. When A emits an event instead, it knows nothing about its consumers.

The trap is infrastructure inflation. Teams spin up Kafka clusters — ZooKeeper, Schema Registry, Kafka Connect — to handle workloads that would run fine on a single Redis instance or a Postgres table. Before adding infrastructure, ask: how many events per second at peak? If the answer is below ~5,000 RPS, Redis Streams will serve you well. Below ~500 RPS, Postgres LISTEN/NOTIFY or a simple polling loop is enough.

Over-engineering the event layer early locks you into operational complexity before you understand your actual access patterns.

A Minimal In-Process Event Bus in Python

Start simple: an in-process event bus is the right choice for a monolith, or for integration tests that need to run without external dependencies.

from collections import defaultdict
from typing import Callable, Any

class EventBus:
    def __init__(self):
        self._handlers: dict[str, list[Callable]] = defaultdict(list)

    def subscribe(self, event_type: str, handler: Callable[[dict], Any]) -> None:
        self._handlers[event_type].append(handler)

    def publish(self, event_type: str, payload: dict) -> None:
        for handler in self._handlers.get(event_type, []):
            handler(payload)

bus = EventBus()

def on_user_created(event: dict) -> None:
    print(f"Sending welcome email to {event['email']}")

def on_user_created_audit(event: dict) -> None:
    print(f"Writing audit log for user {event['user_id']}")

bus.subscribe("user.created", on_user_created)
bus.subscribe("user.created", on_user_created_audit)
bus.publish("user.created", {"user_id": 42, "email": "alice@example.com"})
Enter fullscreen mode Exit fullscreen mode

This is synchronous and in-memory — no durability, no retry, no cross-process fan-out. For internal workflows within a single service, that trade-off is usually correct. The handlers are testable, the bus has no moving parts, and you can replace it with a real broker later without changing your domain code.

Scaling Out: Redis Streams as a Lightweight Message Bus

When you need durability, fan-out to multiple services, or cross-process communication, Redis Streams is the natural next step. It is already in most stacks, requires no additional broker, and supports consumer groups natively.

The model: each service writes to a stream with XADD; consumer groups track per-consumer offsets, giving you at-least-once delivery with acknowledgment.

import redis
import json
import time

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

def publish_event(stream: str, event_type: str, payload: dict) -> str:
    entry = {
        "id": f"{event_type}-{int(time.time() * 1000)}",
        "type": event_type,
        "data": json.dumps(payload),
        "ts": str(int(time.time())),
    }
    return r.xadd(stream, entry)

def consume_events(stream: str, group: str, consumer: str, batch: int = 10):
    try:
        r.xgroup_create(stream, group, id="0", mkstream=True)
    except redis.exceptions.ResponseError:
        pass  # Group already exists

    while True:
        messages = r.xreadgroup(group, consumer, {stream: ">"}, count=batch, block=5000)
        if not messages:
            continue
        for _, entries in messages:
            for msg_id, fields in entries:
                try:
                    handle_event(fields)
                    r.xack(stream, group, msg_id)
                except Exception as exc:
                    print(f"Error processing {msg_id}: {exc}")

def handle_event(fields: dict) -> None:
    event_type = fields.get("type")
    data = json.loads(fields.get("data", "{}"))
    print(f"Processing {event_type}: {data}")

publish_event("events", "order.placed", {"order_id": 101, "amount": 49.99})
Enter fullscreen mode Exit fullscreen mode

Key details: xack after successful processing implements the "claim and acknowledge" pattern. If the consumer crashes before acking, the message stays in the Pending Entries List and can be reclaimed. Consumer groups also let you run multiple workers on the same stream, distributing load automatically. Trim the stream regularly with r.xtrim("events", maxlen=100_000, approximate=True) to avoid unbounded memory growth.

Idempotency: The Pattern You Cannot Skip

At-least-once delivery means duplicates will arrive. Your handlers must be idempotent — processing the same event twice must produce the same result as processing it once. Retrofitting this after a production incident is painful.

The standard approach: deduplicate on a stable event ID before doing any work.

import redis
import hashlib

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
DEDUP_TTL = 86400  # 24 hours

def idempotent_handler(event_id: str, process_fn, payload: dict) -> bool:
    key = f"processed:{hashlib.sha256(event_id.encode()).hexdigest()}"
    # SET NX returns True only on first acquisition
    acquired = r.set(key, "1", ex=DEDUP_TTL, nx=True)
    if not acquired:
        return False  # Already processed — safe to skip
    process_fn(payload)
    return True
Enter fullscreen mode Exit fullscreen mode

Your publisher should embed a stable ID in every event — a UUID generated at publish time. Consumers use it as the dedup key. If Redis is not an option, a Postgres table with a unique constraint on event_id works just as well and gives you a durable dedup log you can audit.

Dead Letter Queues: Handle Poison Messages Before They Handle You

Some events will always fail — malformed payloads, permanent downstream outages, schema mismatches after a deploy. Without a dead letter queue, these messages either block consumption or silently disappear.

With Redis Streams, a DLQ is just a second stream:

MAX_RETRIES = 3

def consume_with_dlq(stream: str, dlq_stream: str, group: str, consumer: str):
    try:
        r.xgroup_create(stream, group, id="0", mkstream=True)
    except redis.exceptions.ResponseError:
        pass

    while True:
        messages = r.xreadgroup(group, consumer, {stream: ">"}, count=10, block=5000)
        if not messages:
            continue
        for _, entries in messages:
            for msg_id, fields in entries:
                retries = int(fields.get("_retries", 0))
                try:
                    handle_event(fields)
                    r.xack(stream, group, msg_id)
                except Exception as exc:
                    if retries >= MAX_RETRIES:
                        fields["_error"] = str(exc)
                        fields["_failed_at"] = str(int(time.time()))
                        r.xadd(dlq_stream, fields)
                        r.xack(stream, group, msg_id)
                    else:
                        fields["_retries"] = str(retries + 1)
                        r.xadd(stream, fields)
                        r.xack(stream, group, msg_id)
Enter fullscreen mode Exit fullscreen mode

This keeps the main stream clean, makes failures visible, and allows manual replay from the DLQ once the root cause is fixed.

When to Graduate to Kafka or a Managed Broker

Redis Streams covers the majority of real-world use cases, but there are genuine limits:

  • Volume above ~100k events/sec: Redis is single-threaded per shard; Kafka's partitioned log scales horizontally
  • Long retention at scale: Redis is RAM-bound; keeping 30 days of high-volume events is expensive
  • Cross-team schema contracts: Kafka's Schema Registry enforces compatibility between producers and consumers — worth the overhead when multiple teams share a stream

For audit and security event pipelines, the durability and ordering guarantees of a managed broker are often worth it even at moderate volume. Losing an authentication event or a configuration change log has compliance consequences that make the operational cost of a real broker worthwhile.

The Takeaway

Event-driven architecture does not require a Kafka cluster from day one. A practical progression:

  1. In-process bus for intra-service decoupling — zero dependencies, fully testable
  2. Redis Streams when you need durability, cross-service fan-out, or cross-process workers
  3. Idempotency keys from the start — embed a stable event ID in every message at publish time
  4. DLQ before you need it — implement it on day one, not after losing events in production
  5. Managed broker when throughput, retention, or schema governance actually demand it

The complexity of your infrastructure should match the complexity of your problem, not the ambition of the architecture diagram.


I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Top comments (0)