Designing a Notification System for an E-commerce App

Every e-commerce app sends notifications: your order shipped, your payment failed, a flash sale started. This post designs one end to end, with a focus on the architecture.

The problem

An online shopping app has about 10 million customers. When something happens (an order is placed or shipped, a payment fails, a price drops), the customer gets a notification by push, email or SMS. Some are urgent (a login code), some are marketing (a flash sale), and users choose which kinds and channels they want.

Requirements

Question	Answer
Types	Critical (OTP, password reset, login alert, payment failed), order updates (confirmed, shipped, delivered, cancelled, refund), marketing (flash sale, price drop, back in stock, abandoned cart)
User choice	Users choose kinds and channels. Critical cannot be turned off. At least one channel is always on
"Delivered"	The provider confirms the notification reached the device or inbox
History	Users see their last 90 days
Speed	OTP within 5 s; payment failed and order updates within 30 s; marketing within minutes (a 5M-user campaign finishes within 30 min)
Failures	Critical messages must not be lost or sent twice; a stale marketing offer is worse than a missed one

Out of scope: the services that trigger notifications, and the providers (APNs/FCM, an email provider, an SMS gateway).

Capacity estimation

Inputs: 10M users; about 50% have everything on (about 4 notifications a day), 30% transactional only (1), 15% custom (2), 5% almost muted (0.2); a flash-sale campaign of 5M users within 30 minutes; peak is about 10x the average; channel opt-in is push 60%, email 90%, SMS 25%, so each notification becomes about 1.75 provider calls.

Step	Calculation	Result
Per day	5M x 4 + 3M x 1 + 1.5M x 2 + 0.5M x 0.2	26.1M notifications/day
Average	26.1M / 86,400	about 300 notifications/s
Normal peak	300 x 10	about 3,000 notifications/s
Campaign	5M / (30 x 60 s)	about 2,800 notifications/s
Combined peak	3,000 + 2,800	about 5,800 notifications/s
Provider calls	5,800 x 1.75	about 10,150 calls/s (push 3,480, email 5,220, SMS 1,450)
DB writes	5,800 x (1 insert + 1.75 status updates)	about 16,000 writes/s (average about 830)
Storage, 90 days	26.1M x 300 B x 90	about 0.7 TB

Notifications, provider calls and database writes are different numbers, and the campaign is added on top of the normal peak.

What this tells us: 0.7 TB fits on one machine, but 16,000 writes/s at peak does not suit a single relational database. The write path drives the architecture.

Architecture

Calling services (orders, payments, marketing)
        |  POST /notifications  (request_id, user_id, type, expires_at, ...)
        v
+-----------------------------+
| Notification API            |  1. duplicate check  --> Redis (request_id, NX EX)
|  validate, reply 202        |  2. settings lookup  --> settings cache / table
|                             |  3. save "accepted"  --> Notifications store
|                             |  4. enqueue          --> topic below
+-----------------------------+
        |
        v
+--------------------------------------------------------------+
| Queues: one topic per channel x priority (9 topics)          |
|            critical        orders         marketing          |
|   push     push-critical   push-orders    push-marketing     |
|   email    email-critical  email-orders   email-marketing    |
|   sms      sms-critical    sms-orders     sms-marketing      |
+--------------------------------------------------------------+
        |
        v
+--------------------------------------------------------------+
| Worker groups (their own workers per topic)                  |
|   1. drop if past expires_at                                 |
|   2. own rate limiter (token bucket) toward the provider     |
|   3. call the provider; retry with backoff                   |
+--------------------------------------------------------------+
        |                    |                      |
        v                    v                      v
   APNs / FCM          Email provider          SMS gateway
   (a backup provider is kept for critical messages)
        |
        +--> status: sent / delivered / failed --> Notifications store
        +--> retries exhausted ------------------> Dead-letter topic

Recovery job: re-queues records still "accepted" after about a minute

The front door

1. Validate the request.
2. Duplicate check: SET request:<id> NX EX 86400 in Redis. A repeat gets nil and is dropped.
3. Settings lookup: which channels the user has on, and whether marketing is allowed (cached).
4. Save the record with status accepted.
5. Enqueue on the right topic.
6. Reply 202 Accepted.

We reply "accepted", not "sent". Sending happens in the background, so the caller is never slowed by a slow or failing provider.

Why save before queueing?

Order	Crash in the gap	Result
Queue, then save	Message queued, no record	It is sent but untracked
Save, then queue	Record exists, nothing queued	Never sent, unless something notices

So we save first, and a recovery job re-queues any record still accepted after about a minute. The worst case is a duplicate, never a silent loss.

Idempotency

The caller creates a request_id once per event ("order 9001 shipped") and reuses it on every retry. Redis is the fast check, and the same request_id is stored on the record as the durable safety net, because Redis can lose keys on a restart.

Data model

Store	Holds	Why
Notifications (Cassandra or DynamoDB)	id, user_id, type, template, status, timestamps, expires_at, request_id	High write rate. Partition by user_id, sorted by created_at, so "this user's latest 50" is one partition read. 90-day TTL
Request IDs (Redis)	request:<id> with an expiry	Fast duplicate check
User settings (small table, cached)	channels on, marketing on	About 10M rows, read often, changed rarely

Why not just Postgres? The average (about 830 writes/s) is easy, but the peak and campaign burst (about 16,000/s) would need sharding and careful tuning. Cassandra and DynamoDB are built for heavy writes, and our reads are a simple key lookup. We give up joins, flexible queries and multi-row transactions. TTL deletion isn't instant (DynamoDB can take days), so the history query also filters by date.

API

POST /notifications
{
  "request_id": "...",     // idempotency key, same on every retry of an event
  "user_id": "...",
  "type": "otp | order_update | marketing",
  "order_id": "...",
  "template_id": "...",
  "expires_at": "..."      // never send late
}

202 Accepted
{ "notification_id": "...", "status": "accepted" }

Plus GET and PUT /users/{id}/settings, and GET /users/{id}/notifications?limit=50&cursor=....

Queues and workers

• One topic per channel and priority. Kafka has no priority queue, and neither does standard SQS, so priority means separate topics with separate workers.
• Partitions give parallelism, topics give priority. Separate partitions inside one topic do not protect an OTP from a campaign, because the same worker group serves them. Critical topics need their own workers.
• Partition sizing: partitions >= calls/s / calls per worker, rounded up (you can add partitions later but never remove them). With an assumed 100 calls/s per worker, email at about 5,220 calls/s needs about 53 workers, so use 64 partitions on the marketing topic.
• Partition key: user_id, where one user's messages must stay in order.
• Provider limits: if the email provider allows 4,000 calls/s and we need 5,220, workers run their own token bucket and send at most 4,000/s. The extras wait in the queue; they are not dropped. Part of the allowed rate is reserved for critical messages.
• Expiry: a worker checks expires_at before sending. A stale OTP or an old offer is dropped and marked expired.
• Delivery guarantee: commit the offset after the provider call, which gives at-least-once. Workers must tolerate a rare duplicate.

Failure handling

	One call times out	Whole provider is down	Expired	Retries exhausted
OTP	Retry quickly, until expires_at	Backup provider, else another enabled channel	Drop	Mark failed, alert
Payment failed / order updates	Retry with backoff	Backup provider or another channel	Rarely applies	Dead-letter topic
Marketing	Give up (at most one retry)	Wait in the queue until recovery	Drop	Drop

For critical messages a loss hurts and a duplicate is harmless; for marketing a miss is cheap and a duplicate annoys at scale.

• Backoff and dead letters: wait longer between retries (1 s, 2 s, 4 s), cap the attempts, then park the message in a dead-letter topic.
• Circuit breaker: after many failures in a row, stop calling a provider for a short while and test it occasionally.
• A timeout is ambiguous: we don't know if the provider got the call. Push services have collapse ids and keys that replace an earlier message, but support varies, so we don't rely on them.

Trade-offs

Decision	Chose	Gave up
Database	Cassandra / DynamoDB	Joins, flexible queries, multi-row transactions
Duplicates vs loss	OTP and orders may duplicate; marketing may be missed	A perfect "exactly once"
Queues	Topics per channel x priority, reserved capacity for critical	More queues and workers; idle reserved capacity
Save before queue	A possible duplicate after a crash	Simplicity (needs a recovery job)
Provider pacing	Our own limiter, extras wait	Marketing can be delayed