Under the Hood: Distributed Message Broker Design, Storage, and Failure Modes

Subham Ghosh — Sun, 27 Sep 2026 12:30:11 +0000

Most discussions about message brokers stay at a high level, placing generic broker boxes on a system architecture diagram. In practice, building or operating a distributed message broker requires understanding specific mechanics around storage, replication, and consumer coordination.

I wrote a detailed walkthrough analyzing how a fault-tolerant message broker works under the hood. It focuses on implementation details, edge cases, and failure modes rather than surface-level component diagrams. It is structured to be useful for engineers at any experience level.

Full article: https://engineering.by-tech.workers.dev/notes/distributed-message-broker-system-design/

Key Concepts Covered

Ordering vs Throughput: Using partition-based append-only logs to achieve horizontal scaling while maintaining strict ordering per key.
Replication Semantics: What acks=all and min.insync.replicas actually protect against during leader failures, and where failure windows still exist.
Duplicate Processing: Why consumer crashes lead to duplicate downstream side effects, and how generation fencing, idempotency keys, and transa ctional outbox patterns prevent state corruption.
Operational Boundaries: Managing backpressure, slow consumers falling behind retention windows, and broker disk exhaustion.

Architecture and Parameters Evaluated

Storage Engine: Partitioned, append-only retained logs optimized for sequential disk I/O.
Replication Factor: 3 replicas with min.insync.replicas = 2.
Consumer Coordination: Generation-fenced offset commits to reject stale consumer writes after rebalances.
Reliability Features: Retry topics, Dead Letter Queues, quotas, and retention cleanup policies.

The goal was to connect every architectural decision directly to a specific failure mode and detail the trade-offs involved.

Feedback, questions, and discussion are welcome in the comments!

How Google Drive Syncs Files and Resolves Concurrent Edits

Subham Ghosh — Sun, 20 Sep 2026 04:41:28 +0000

File synchronization looks simple until devices go offline, files become large, and multiple users edit the same content.
This article examines a Google Drive–style sync architecture and the problems behind reliable file updates:

Offline edits reconnecting at the same time
Large files with only small changes
Out-of-order retries, metadata updates, and notifications
Concurrent edits and version conflicts
The accompanying HLD diagram maps the flow across the client, API gateway, metadata server, block server, metadata database, and object storage. The article explains how these components interact during synchronization.
Read the full article: https://engineering.by-tech.workers.dev
Any feedback, questions, or discussion are welcome.

DEV Community: Subham Ghosh

Under the Hood: Distributed Message Broker Design, Storage, and Failure Modes

How Google Drive Syncs Files and Resolves Concurrent Edits