DEV Community: Subham Ghosh

Designing an Email Service at Scale

Subham Ghosh — Sun, 18 Oct 2026 02:48:57 +0000

How do you build a distributed platform that handles 1 billion users, 50 billion daily messages, and petabytes of storage with zero data loss?

Dive into the complete engineering breakdown here:
https://engineering.by-tech.workers.dev/articles/designing-an-email-service/

At a high level, modern email architecture operates as two independent, one-way pipelines bridging legacy protocols and cloud-native systems:

Outbound Delivery: Users submit messages via HTTPS REST APIs, where payloads are immediately offloaded to S3. Delivery tasks are queued in Kafka, and outbound workers manage DKIM signing and DNS MX resolution before delivering over SMTP.
Inbound Ingestion: Edge gateways terminate global Anycast connections on port 25 and perform lightweight SPF/DMARC checks. Heavy message bodies stream to S3, while a Kafka claim-check pointer triggers deep scanning, Cassandra metadata writes, and real-time WebSocket pushes to active devices.

Key technical constraints discussed in the article:

The Custody Contract: Why returning an SMTP 250 OK is an irreversible legal commitment that must wait for multi-AZ Kafka replication.
The Dual-Write Trap: How using a single-partition Cassandra atomic batch for transactional outboxes guarantees emails are never saved without also being queued for delivery.
Backscatter Prevention: How sending an automated bounce to a forged address can instantly get your outbound IP blacklisted by major ISPs.
Data Modeling: Sharding Cassandra by user_id and monthly time buckets to keep read latency under 200ms without unbounded partition growth.

Design a metrics monitoring system

Subham Ghosh — Sun, 11 Oct 2026 05:15:08 +0000

How do you design a metrics monitoring system for 100,000 servers?

I’ve shared my approach here: https://engineering.by-tech.workers.dev/articles/metrics-monitoring-system-design/

Here’s the high-level flow:

Agents collect metrics; a durable message queue buffers them.
A time-series database stores metrics for dashboard queries.
A stream processor evaluates alert rules independently of dashboard storage.
Notification workers deliver alerts and retry failed sends.
Older measurements become summaries to keep storage manageable.

I also cover capacity planning, push vs. pull collection, data retention, and recovery from failures - along with alert rules for sustained spikes and missing data, since a brief CPU spike needs a different response from prolonged overload.

What Does It Take to Build a Correct Real-Time Leaderboard?

Subham Ghosh — Sat, 03 Oct 2026 15:14:45 +0000

What happens when a gaming leaderboard must stay fast, accurate, retry-safe, and recoverable—even while millions of players are competing?

The article below captures my thought process and understanding of how to design this system:
https://engineering.by-tech.workers.dev/articles/real-time-gaming-leaderboard-hld/

The challenge isn’t simply displaying the Top 10. It’s preventing duplicate score updates, handling out-of-order events, returning exact player ranks, recovering after data loss, and safely finalizing tournament results.

The design covers idempotent score processing, Redis sorted sets, live updates, deterministic tie-breaking, sharding, and multi-region consistency.

SystemDesign #DistributedSystems #BackendEngineering #Redis

Under the Hood: Distributed Message Broker Design, Storage, and Failure Modes

Subham Ghosh — Sun, 27 Sep 2026 12:30:11 +0000

Most discussions about message brokers stay at a high level, placing generic broker boxes on a system architecture diagram. In practice, building or operating a distributed message broker requires understanding specific mechanics around storage, replication, and consumer coordination.

I wrote a detailed walkthrough analyzing how a fault-tolerant message broker works under the hood. It focuses on implementation details, edge cases, and failure modes rather than surface-level component diagrams. It is structured to be useful for engineers at any experience level.

Full article: https://engineering.by-tech.workers.dev/articles/distributed-message-broker-system-design/

Key Concepts Covered

Ordering vs Throughput: Using partition-based append-only logs to achieve horizontal scaling while maintaining strict ordering per key.
Replication Semantics: What acks=all and min.insync.replicas actually protect against during leader failures, and where failure windows still exist.
Duplicate Processing: Why consumer crashes lead to duplicate downstream side effects, and how generation fencing, idempotency keys, and transa ctional outbox patterns prevent state corruption.
Operational Boundaries: Managing backpressure, slow consumers falling behind retention windows, and broker disk exhaustion.

Architecture and Parameters Evaluated

Storage Engine: Partitioned, append-only retained logs optimized for sequential disk I/O.
Replication Factor: 3 replicas with min.insync.replicas = 2.
Consumer Coordination: Generation-fenced offset commits to reject stale consumer writes after rebalances.
Reliability Features: Retry topics, Dead Letter Queues, quotas, and retention cleanup policies.

The goal was to connect every architectural decision directly to a specific failure mode and detail the trade-offs involved.

Feedback, questions, and discussion are welcome in the comments!

How Google Drive Syncs Files and Resolves Concurrent Edits

Subham Ghosh — Sun, 20 Sep 2026 04:41:28 +0000

File synchronization looks simple until devices go offline, files become large, and multiple users edit the same content.
This article examines a Google Drive–style sync architecture and the problems behind reliable file updates:

Offline edits reconnecting at the same time
Large files with only small changes
Out-of-order retries, metadata updates, and notifications
Concurrent edits and version conflicts
The accompanying HLD diagram maps the flow across the client, API gateway, metadata server, block server, metadata database, and object storage. The article explains how these components interact during synchronization.
Read the full article: https://engineering.by-tech.workers.dev
Any feedback, questions, or discussion are welcome.