Most discussions about message brokers stay at a high level, placing generic broker boxes on a system architecture diagram. In practice, building or operating a distributed message broker requires understanding specific mechanics around storage, replication, and consumer coordination.
I wrote a detailed walkthrough analyzing how a fault-tolerant message broker works under the hood. It focuses on implementation details, edge cases, and failure modes rather than surface-level component diagrams. It is structured to be useful for engineers at any experience level.
Full article: https://engineering.by-tech.workers.dev/notes/distributed-message-broker-system-design/
Key Concepts Covered
- Ordering vs Throughput: Using partition-based append-only logs to achieve horizontal scaling while maintaining strict ordering per key.
- Replication Semantics: What acks=all and min.insync.replicas actually protect against during leader failures, and where failure windows still exist.
- Duplicate Processing: Why consumer crashes lead to duplicate downstream side effects, and how generation fencing, idempotency keys, and transa ctional outbox patterns prevent state corruption.
- Operational Boundaries: Managing backpressure, slow consumers falling behind retention windows, and broker disk exhaustion.
Architecture and Parameters Evaluated
- Storage Engine: Partitioned, append-only retained logs optimized for sequential disk I/O.
- Replication Factor: 3 replicas with min.insync.replicas = 2.
- Consumer Coordination: Generation-fenced offset commits to reject stale consumer writes after rebalances.
- Reliability Features: Retry topics, Dead Letter Queues, quotas, and retention cleanup policies.
The goal was to connect every architectural decision directly to a specific failure mode and detail the trade-offs involved.
Feedback, questions, and discussion are welcome in the comments!
Top comments (0)