DEV Community: Rohit Kori

Designing an E-Commerce System

Rohit Kori — Mon, 10 Aug 2026 17:28:08 +0000

Imagine opening Amazon or Flipkart during a major sale.

You search for a product, open its page, add it to your cart, choose an address, pay for the order, and eventually receive the package.

From the customer's perspective, the experience is simple:

Search
  ↓
Product
  ↓
Cart
  ↓
Checkout
  ↓
Payment
  ↓
Order
  ↓
Delivery

Behind the scenes, however, many independent systems are working together.

The platform has to search through hundreds of millions of products, serve millions of product-page requests, keep carts available across devices, maintain accurate inventory, process payments, create orders, communicate with warehouses, and survive huge traffic spikes during events such as Black Friday or Prime Day.

The most interesting part of the design is not the product catalog itself. The difficult problems appear when large traffic, concurrency, inventory, payment, and distributed services meet each other.

This article walks through the system from the beginning and gradually introduces those problems.

1. Understanding the Requirements

Let's start with the customer experience.

A customer should be able to browse the product catalog and search for products using text and filters.

For example:

"wireless headphones"

The customer might then filter by:

Brand = Sony
Price = ₹5,000–₹30,000
Rating > 4
Category = Headphones

After finding a product, the customer should be able to view its details, add it to a cart, and proceed to checkout.

During checkout, the system needs to:

Validate the cart
       ↓
Check the current price
       ↓
Check inventory
       ↓
Reserve inventory
       ↓
Calculate the final amount
       ↓
Authorize payment
       ↓
Create the order

After the order is created, other systems take over:

Order
  ↓
Fulfillment
  ↓
Shipping
  ↓
Delivery

The platform also needs to support sellers, inventory management, notifications, recommendations, reviews, and analytics.

For this design, the most important parts are:

Product catalog
Search
Cart
Inventory
Checkout
Payment
Order management
Fulfillment
Notifications

The internals of warehouse management, advanced fraud detection, and recommendation algorithms can be treated as separate systems.

2. The First Important Observation

Not every part of an e-commerce platform has the same consistency requirements.

Consider a product search.

A customer searches:

"iPhone"

It is perfectly acceptable if a newly added product takes a few seconds to appear in search.

Now consider inventory.

Suppose only one phone remains:

Stock = 1

Two customers attempt to buy it at nearly the same time.

The system must not sell two phones.

This gives us an important distinction:

Browsing / Search
        ↓
Very high traffic
Eventual consistency is often acceptable

Checkout / Inventory / Payment
        ↓
Correctness is critical
Strong consistency is required

This distinction will influence almost every architectural decision we make.

3. How Much Traffic Are We Designing For?

Before choosing databases and services, we need some scale assumptions.

Suppose our platform has:

100M+ registered users
10M+ daily active users
100M–500M products
Millions of orders per day
Tens or hundreds of thousands of search requests per second

The exact numbers aren't important. They are assumptions that help us reason about the architecture.

Suppose we process:

10M orders/day

The average is only around:

10,000,000 / 86,400
≈ 116 orders/sec

That number might look manageable.

But designing for 116 requests/sec would be a mistake.

Traffic is not evenly distributed.

During a large sale, the system might suddenly receive:

10K+ checkout attempts/sec

and search traffic could be much higher.

So we design for peak traffic, not average traffic.

This immediately tells us that different workloads need different scaling strategies.

4. The High-Level Architecture

A useful first version of the architecture looks like this:

                         USERS
                           |
                           ▼
                    CDN / API Gateway
                           |
              ┌────────────┼────────────┐
              ↓            ↓            ↓
           Product       Search        Cart
           Service       Service       Service
              |            |            |
              ↓            ↓            ↓
          Product DB   Elasticsearch   Redis
              |
              ↓
             Kafka
              |
       ┌──────┼───────────────┐
       ↓      ↓               ↓
   Inventory  Order        Other Events
       |       |
       ↓       ↓
  Inventory   Order DB
     DB
       |
       ↓
    Checkout
       |
   ┌───┼────────────┐
   ↓   ↓            ↓
Inventory Pricing  Payment
                    |
                    ▼
             Payment Gateway
                    |
                    ▼
                  Order
                    |
                    ▼
                  Kafka
                    |
        ┌───────────┼───────────┐
        ↓           ↓           ↓
   Notification  Warehouse   Analytics

This diagram looks complicated, but the reasoning is straightforward.

We separate the system according to the workload:

Product browsing → cache / CDN / search index

Search           → Elasticsearch

Cart             → Redis

Inventory        → transactional database

Orders           → transactional database

Events           → Kafka

Payment          → payment provider

Flash sales      → Redis + queues

Let's understand why each piece exists.

5. Building the Product Catalog

The product catalog is the foundation of the platform.

A product might contain:

productId
name
description
brand
category
price
images
attributes
rating

The difficult part is that different categories have different attributes.

A T-shirt might have:

size
color
material
fit

A laptop might have:

CPU
RAM
storage
screen size
GPU

A book might have:

ISBN
author
publisher
language

Because the structure isn't identical for every product, a document-oriented database such as MongoDB or DynamoDB can be a reasonable choice for the catalog.

A simplified document could look like:

{
  "productId": "P123",
  "name": "Sony Headphones",
  "brand": "Sony",
  "category": "Electronics",
  "price": 34999,
  "attributes": {
    "color": "Black",
    "connectivity": "Bluetooth"
  }
}

The catalog database is the source of truth for product information.

But we shouldn't use this database directly for every product search.

6. Why Search Needs Its Own System

Searching a large catalog is a different problem from storing products.

Suppose the customer searches:

wireless headphones

and wants:

Brand = Sony
Price = ₹5,000–₹30,000
Rating >= 4
Availability = In stock

A relational or document database can perform simple queries, but at very large scale we want a system specifically optimized for search.

This is where Elasticsearch comes in.

The architecture becomes:

Product Database
       |
       ↓
Product Changed Event
       |
       ↓
     Kafka
       |
       ↓
 Search Indexer
       |
       ↓
 Elasticsearch

Elasticsearch can provide:

Full-text search
Filtering
Faceted navigation
Autocomplete
Fuzzy matching
Relevance ranking

The customer request then becomes:

User
 ↓
Search API
 ↓
Elasticsearch
 ↓
Search results

This keeps heavy search traffic away from the transactional product database.

7. Why Search Can Be Eventually Consistent

Suppose a seller creates a new product.

The product is immediately saved:

Product DB
    ↓
Product exists

Then an event is published:

Product Created
    ↓
Kafka
    ↓
Search Indexer
    ↓
Elasticsearch

There may be a short delay.

For a few seconds:

Product DB       → new product exists
Elasticsearch    → product not indexed yet

That's usually fine.

Search doesn't have to be perfectly synchronized every millisecond.

But checkout is different.

Suppose Elasticsearch says:

Price = ₹50,000

while the product database says:

Price = ₹55,000

We cannot let the customer pay based only on the search result.

Therefore, during checkout:

Search result
      ↓
Re-fetch authoritative data
      ↓
Validate current price
      ↓
Validate current inventory

The search index helps us find the product.

It does not become the source of truth for money or inventory.

8. Making Product Reads Fast

Product pages can receive enormous traffic.

A popular product might be viewed millions of times.

We don't want:

1M requests
     ↓
Product Database

Instead, we can cache frequently accessed product information in Redis.

User
 ↓
Product Service
 ↓
Redis

For example:

product:P123

name = iPhone
price = ₹80,000
brand = Apple
...

If the value exists in Redis:

Redis → return immediately

If it doesn't:

Redis miss
    ↓
Product DB
    ↓
Store in Redis
    ↓
Return

This is a classic cache-aside pattern.

The database remains authoritative.

Redis exists primarily to make common reads fast and protect the database from excessive read traffic.

9. Designing the Shopping Cart

Now the customer adds a product to the cart.

A cart might look like:

Cart
-------------------------
iPhone       × 1
Headphones   × 2

For a logged-in customer, Redis is a good fit because cart operations are frequent and latency-sensitive.

For example:

cart:user123

iPhone       → 1
Headphones   → 2

A TTL can eventually remove abandoned carts.

Guest users are slightly different.

A guest cart can initially live in the browser. When the user logs in, the guest cart can be merged with the user's server-side cart.

The important thing is that the cart should not be treated as the final source of truth for price or inventory.

10. The Cart Price Problem

Suppose a customer adds headphones for:

₹10,000

to the cart.

Two hours later, the seller changes the price:

₹12,000

What should happen?

The cart might still display the old price.

But when checkout begins, we must revalidate it.

The flow becomes:

Cart
 ↓
Fetch current product price
 ↓
Fetch current promotions
 ↓
Calculate current total

The server, not the browser, determines the final amount.

A customer should never be able to send:

{
  "price": 1
}

and expect the server to charge ₹1 for a ₹10,000 product.

11. Inventory Is Where the Design Gets Interesting

Inventory is one of the hardest parts of an e-commerce system because of concurrency.

Suppose:

iPhone stock = 1

At exactly the same time:

User A → BUY
User B → BUY

Both requests could read:

stock = 1

If we simply perform:

Read stock
 ↓
Check stock
 ↓
Decrease stock

both requests could succeed.

The result would be:

Two customers
      ↓
Two orders
      ↓
One physical phone

That's overselling.

We need an atomic operation.

12. A Simple Inventory Solution: Database Locking

One approach is a database transaction with a row lock.

Conceptually:

BEGIN;

SELECT quantity
FROM inventory
WHERE product_id = ?
FOR UPDATE;

-- check quantity

UPDATE inventory
SET quantity = quantity - 1
WHERE product_id = ?;

COMMIT;

The first transaction locks the inventory row.

If stock is:

the first buyer reserves it:

1 → 0

When the second buyer gets the lock:

0 → reject

This is easy to reason about and gives strong consistency.

But there is a problem.

13. Why Database Locking Isn't Enough for Flash Sales

Imagine a flash sale:

Stock = 1,000

and:

1,000,000 users

all trying to buy the same product.

Now one database row becomes extremely hot.

Thousands of requests may compete for the same lock.

We can get:

High lock contention
Slow transactions
Connection pool exhaustion
Database CPU pressure

So we need a different strategy for extreme traffic.

14. Atomic Inventory Updates

For normal high-volume traffic, we can often avoid explicit locks by making the database update itself atomic.

For example:

UPDATE inventory
SET quantity = quantity - 1
WHERE product_id = ?
AND quantity >= 1;

If:

rows affected = 1

the reservation succeeded.

If:

rows affected = 0

there was no inventory available.

The important part is that the database performs:

Check quantity
+
Decrease quantity

as one atomic operation.

This is often a better starting point than pessimistic locking.

15. Inventory Reservations

There is another important concept: we shouldn't necessarily permanently consume inventory as soon as a customer starts checkout.

Suppose:

Stock = 10

A customer starts checkout.

We reserve one item:

Available = 9
Reserved = 1

The customer now has, for example, 15 minutes to complete payment.

If payment succeeds:

Reserved → Sold

If the customer abandons checkout:

Reservation expires
      ↓
Reserved → Available

This prevents customers from holding inventory forever.

The reservation lifecycle can be represented as:

AVAILABLE
    ↓
RESERVED
    ↓
    ├── PAYMENT SUCCESS → SOLD
    |
    └── TIMEOUT → AVAILABLE

16. Flash Sales Require a Different Approach

Now consider a limited product:

1,000 units

and:

1,000,000 users

We cannot allow all million requests to hit the inventory database.

Instead, Redis can act as a very fast admission-control layer.

Before the sale:

flash:stock:P123 = 1000

A purchase attempts an atomic decrement:

DECR flash:stock:P123

Redis can process these operations extremely quickly.

Once the counter reaches zero:

Sold out

But Redis should not become our only permanent source of truth.

A better architecture is:

Millions of users
       ↓
Rate Limiter
       ↓
Queue
       ↓
Redis atomic stock counter
       ↓
Successful buyers
       ↓
Order workers
       ↓
Durable inventory database

Redis protects the database from the enormous burst.

The database remains the durable record.

A reconciliation process can compare Redis and the database and repair any inconsistencies.

17. Protecting the System With a Queue

Even Redis can receive a huge number of requests.

A queue gives the system backpressure.

Instead of:

1M users
   ↓
1M synchronous requests
   ↓
Database

we can do:

1M users
   ↓
Queue
   ↓
Controlled number of workers
   ↓
Inventory / Order services

The customer might see:

"You are in the queue."

This allows the system to process successful purchase attempts at a rate the downstream services can handle.

18. Checkout Is a Distributed Workflow

Now let's put the major pieces together.

Suppose the customer has:

iPhone       ₹80,000
Headphones   ₹10,000

and clicks:

Place Order

The checkout workflow might be:

Validate Cart
      ↓
Re-check Prices
      ↓
Reserve Inventory
      ↓
Calculate Final Amount
      ↓
Authorize Payment
      ↓
Create Order
      ↓
Confirm Inventory
      ↓
Publish Order Event

Notice that this involves multiple services:

Cart
Inventory
Pricing
Payment
Order

Each may have its own database.

That creates a distributed transaction problem.

19. Why Not Use One Giant Transaction?

We might imagine:

BEGIN

Reserve Inventory
Charge Payment
Create Order

COMMIT

But inventory, payment, and orders may be separate services.

Traditional database transactions don't naturally span all of them.

Two-Phase Commit is one possible solution, but it introduces:

Blocking
High latency
Tight coupling
Complex failure handling

For a microservice architecture, a Saga is usually more practical.

20. The Saga Pattern

A Saga breaks the checkout workflow into smaller operations.

For example:

1. Reserve Inventory
2. Authorize Payment
3. Create Order

Suppose:

Reserve Inventory → SUCCESS
Authorize Payment  → SUCCESS
Create Order       → FAILURE

We need compensating actions.

Create Order → FAILED
       ↓
Void Payment Authorization
       ↓
Release Inventory

The important idea is:

A distributed transaction is replaced by a sequence of local transactions plus compensating actions.

21. Orchestrating the Checkout

There are two common ways to implement a Saga.

With choreography, services communicate through events:

Inventory Reserved
       ↓
Payment Service
       ↓
Payment Completed
       ↓
Order Service

This is loosely coupled, but large workflows can become difficult to understand.

With orchestration, one service coordinates the workflow:

                 Order Orchestrator
                    |
       ┌────────────┼────────────┐
       ↓            ↓            ↓
  Inventory       Payment       Order

The orchestrator knows:

What step comes next?
What happens when a step fails?
What compensation is required?

For checkout, orchestration is often easier to reason about and debug.

22. Payment Is Another Reliability Problem

Now suppose the payment service calls an external payment provider.

The request looks like:

Our Payment Service
        ↓
Payment Gateway
        ↓
Bank / Card Network

Imagine the bank successfully charges:

₹10,000

but our connection times out before we receive the response.

Our service doesn't know whether the payment succeeded.

If we simply retry:

Retry
  ↓
Another ₹10,000 charge

the customer could be charged twice.

This is why payment APIs must be idempotent.

23. Idempotency

Every logical payment attempt gets an idempotency key:

PAYMENT-123

The first request:

PAYMENT-123
     ↓
SUCCESS

A retry with the same key:

PAYMENT-123
     ↓
Already processed
     ↓
Return previous result

The payment provider and our own payment service should both support this concept.

The payment table can store:

paymentId
orderId
amount
status
provider
idempotencyKey
createdAt

and enforce uniqueness on the idempotency key.

24. Authorize First, Capture Later

A useful payment flow is:

Order placement
      ↓
Payment authorization
      ↓
Order processing
      ↓
Product shipped
      ↓
Payment capture

Authorization means the payment method has enough funds and the amount is reserved.

If the order is cancelled before shipment:

Cancel order
      ↓
Void authorization

instead of charging the customer and then issuing a refund.

This is particularly useful for physical goods because payment capture can be tied to fulfillment.

25. The Order State Machine

An order should have explicit states.

For example:

PLACED
   ↓
PAYMENT_AUTHORIZED
   ↓
CONFIRMED
   ↓
PROCESSING
   ↓
SHIPPED
   ↓
DELIVERED

Other possible states include:

CANCELLED
RETURNED
REFUNDED

The state machine becomes very useful when failures happen.

For example:

Payment = AUTHORIZED
Order = PAYMENT_AUTHORIZED
Inventory = RESERVED

If the order service crashes, we know where the workflow stopped.

We can resume or compensate from that state.

26. Keeping Services in Sync With Kafka

Once an order is successfully created, many other systems need to know about it.

For example:

Notification
Warehouse
Analytics
Recommendations
Seller dashboard
Fraud detection

We don't want checkout to wait for all of them.

Instead:

Create Order
     ↓
Publish OrderCreated
     ↓
Return response

Consumers process the event asynchronously:

                    OrderCreated
                         |
          ┌──────────────┼──────────────┐
          ↓              ↓              ↓
    Notification      Warehouse      Analytics

Kafka is useful as the event backbone.

Typical event streams might include:

product-events
inventory-events
order-events
user-events
click-events

This keeps downstream systems decoupled from the checkout path.

27. What If Kafka Is Down?

Suppose:

Order saved successfully
        ↓
Kafka unavailable

We cannot afford to lose the OrderCreated event.

A common solution is the transactional outbox pattern.

Inside the same database transaction:

BEGIN

Save Order
Save Outbox Event

COMMIT

Now both the order and the event are durable.

A background worker later publishes:

Outbox
   ↓
Kafka

If Kafka is temporarily unavailable, the worker retries.

This creates a reliable bridge between the transactional database and the asynchronous event system.

28. Making Kafka Processing Reliable

Distributed messaging usually works with at-least-once delivery.

That means a consumer may receive the same event more than once.

For example:

OrderCreated
    ↓
Consumer processes event
    ↓
Consumer crashes before acknowledging
    ↓
Kafka delivers event again

Therefore consumers should be idempotent.

A common pattern is to give each event a unique:

eventId

and record processed event IDs when necessary.

We also need to monitor:

Kafka consumer lag

If consumers cannot keep up with producers, lag grows.

Messages that repeatedly fail can be moved to a dead-letter queue for investigation.

29. API Design

Once the core architecture is clear, the major APIs become straightforward.

Search:

GET /api/v1/products/search?q=wireless+headphones&category=electronics&minPrice=5000&maxPrice=30000

Product:

GET /api/v1/products/{productId}

Add to cart:

POST /api/v1/carts/{cartId}/items

Get cart:

GET /api/v1/carts/{cartId}

Create order:

POST /api/v1/orders
Idempotency-Key: checkout-123

Get order:

GET /api/v1/orders/{orderId}

Payment:

POST /api/v1/payments
Idempotency-Key: payment-123

The important API design principle is that operations which can be retried should be made idempotent.

30. Data Model

The catalog can use a document model because product attributes vary.

For example:

Product
-------------------------
productId
sellerId
name
category
price
attributes
images
rating
createdAt
updatedAt

Inventory is more transactional:

Inventory
-------------------------
productId
warehouseId
availableQuantity
reservedQuantity
updatedAt

Primary Key:
(productId, warehouseId)

The cart can contain:

Cart
-------------------------
cartId
userId
status
createdAt
updatedAt
expiresAt

and:

CartItem
-------------------------
cartId
productId
quantity
priceAtAddTime

Orders should preserve the actual price agreed during checkout:

Order
-------------------------
orderId
userId
status
subtotal
totalAmount
currency
createdAt
updatedAt

Order items:

OrderItem
-------------------------
orderId
productId
sellerId
quantity
unitPrice

Payment:

Payment
-------------------------
paymentId
orderId
amount
currency
status
provider
idempotencyKey
createdAt
updatedAt

An order should never recalculate its historical price from the current product price.

31. Choosing the Databases

There is no requirement to use one database for everything.

A reasonable large-scale architecture might use:

Catalog
   → MongoDB / DynamoDB

Search
   → Elasticsearch

Cart
   → Redis

Orders
   → PostgreSQL / MySQL

Inventory
   → Strongly consistent relational DB

Events
   → Kafka

The important principle is:

Choose storage according to the access pattern and consistency requirement.

For a smaller system, PostgreSQL could handle much more of the platform.

The more specialized architecture becomes useful as traffic and scale increase.

32. Partitioning and Sharding

Eventually, an order database may become too large for one machine or one database cluster.

Orders can be partitioned using a key such as:

userId

if the common query is:

Show all orders for this user

For example:

Shard 1 → users A–F
Shard 2 → users G–M
Shard 3 → users N–S
Shard 4 → users T–Z

The exact partitioning strategy depends on the workload.

We must also watch for hot partitions.

A very large seller or customer can generate disproportionately high traffic.

33. Multi-Warehouse Inventory

Large marketplaces rarely have one warehouse.

Imagine:

Bangalore → 5 iPhones
Mumbai    → 10 iPhones
Delhi     → 2 iPhones

Inventory therefore becomes:

Product
   |
   ├── Bangalore → 5
   ├── Mumbai    → 10
   └── Delhi     → 2

When a customer places an order, the system needs to decide where the order should be fulfilled.

Possible factors include:

Availability
Customer location
Shipping cost
Delivery time
Warehouse capacity

Ideally, we ship from a nearby warehouse that has inventory.

34. Multi-Seller Orders

A marketplace can have products from several sellers in one cart:

iPhone → Seller A
Shoes   → Seller B
Book    → Seller C

The customer experiences one checkout, but internally the platform may create separate fulfillment orders:

Customer Order
      |
      ├── Seller A Order
      ├── Seller B Order
      └── Seller C Order

Each seller may ship independently.

Payment may also need to be split into:

Customer payment
      ↓
Platform commission
      ↓
Seller payouts

This is one reason marketplace systems are considerably more complicated than a simple online store.

35. Pricing and Promotions

The final checkout price may involve several steps:

Base price
   ↓
Discount
   ↓
Coupon
   ↓
Tax
   ↓
Shipping
   ↓
Final amount

A promotion might have:

Expiry
Usage limit
Eligible products
Eligible users
Minimum order value

The pricing service must validate all of these.

The client must never be trusted to supply the final price.

36. Recommendations and Reviews

Recommendations don't need to block checkout.

A recommendation system might use:

Purchase history
Browsing history
Product similarity
Co-purchase patterns

For example:

Customers who bought A
also bought B

The recommendation results can be precomputed and stored in Redis.

Reviews and ratings can also be handled asynchronously.

A new review taking a few seconds to appear in an aggregate rating is generally acceptable.

37. Rate Limiting

Rate limiting protects the system from abuse and sudden bursts.

For example:

Search
→ relatively high limit

Add to Cart
→ lower limit

Checkout
→ much lower limit

A distributed rate limiter can use Redis counters.

The flow is:

Client
  ↓
API Gateway
  ↓
Rate Limiter
  ↓
Application Service

During a flash sale, rate limiting becomes especially important because the goal is not merely to accept traffic but to protect downstream services from being overwhelmed.

38. CDN

Product images and static assets are excellent CDN candidates.

Instead of:

User
 ↓
Application server
 ↓
Object storage

we can use:

User
 ↓
CDN
 ↓
Cache hit

For a cache miss:

CDN
 ↓
Origin
 ↓
Cache
 ↓
User

This keeps huge amounts of image and static-content traffic away from application servers.

39. Observability

At this scale, knowing that a service is "up" isn't enough.

We need to know whether the entire customer journey is healthy.

Useful metrics include:

Search p95 / p99 latency
Product API latency
Checkout latency
Checkout success rate
Inventory reservation failures
Payment failure rate
Order creation failures
Redis hit ratio
Database latency
Database connection usage
Kafka consumer lag
Flash-sale queue depth

Distributed tracing can follow:

Request
  ↓
API Gateway
  ↓
Order Service
  ↓
Inventory Service
  ↓
Payment Service
  ↓
Database

A correlation ID makes it possible to trace one checkout across multiple services.

40. Failure Handling

Distributed systems fail.

The architecture should assume this rather than treating failure as an exceptional event.

If the inventory service is unavailable:

Checkout
   ↓
Inventory unavailable
   ↓
Do not create a confirmed order
   ↓
Retry or fail gracefully

If the payment gateway is unavailable:

Payment unavailable
   ↓
Do not mark payment as successful

If Kafka is unavailable:

Persist event in outbox
   ↓
Publish later

If Redis is unavailable:

Use the durable database where safe

However, for flash-sale admission, Redis may be part of the high-throughput control path, so the system should have an explicit degraded mode rather than blindly bypassing it.

41. Inventory Reconciliation

Because systems such as Redis, Kafka consumers, and databases can fail independently, inventory can temporarily drift.

For example:

Redis says:
10 available

Database says:
8 available

A reconciliation process can periodically compare authoritative inventory with derived state.

Inventory DB
     ↓
Compare
     ↓
Redis
     ↓
Repair mismatch

This is especially important after flash sales.

The database remains the durable source of truth.

42. CQRS and Read Models

At very large scale, read traffic can be separated from write traffic.

The write path is:

Command
   ↓
Transactional DB

Events are then published:

Transactional DB
       ↓
Kafka
       ↓
Read model builders

Those consumers update:

Elasticsearch
Redis
Other read models

Conceptually:

                Commands
                   |
                   ▼
              Write Model
                   |
                   ▼
              PostgreSQL
                   |
                 Kafka
                   |
          ┌────────┴────────┐
          ↓                 ↓
    Elasticsearch         Redis
      Read Model         Read Model

This is essentially CQRS.

The trade-off is eventual consistency between the write model and read models.

That is acceptable for search and browsing, but not for the final inventory or payment decision.

43. Security

E-commerce systems handle sensitive customer and payment information.

Important controls include:

Authentication
Authorization
TLS
Rate limiting
Input validation
Audit logging
Fraud detection

Payment card information should normally be handled by a PCI-compliant payment provider rather than stored directly in the application database.

The server must also validate:

Price
Quantity
Seller
Product
Order ownership
Coupon

A client should never be able to manipulate these values simply by changing the request.

44. Availability and Disaster Recovery

A production platform should have multiple instances of critical services.

For example:

Load Balancer
     |
 ┌───┼───┐
 ↓   ↓   ↓
S1  S2  S3

Databases should use:

Replication
Backups
Automated failover

Kafka should use replication.

Redis should have appropriate replication and recovery depending on how it is being used.

For disaster recovery, backups should be tested rather than merely configured.

A recovery plan should define:

RPO
How much data can we lose?

RTO
How quickly must we recover?

45. Multi-Region Architecture

As the platform becomes global, one region may no longer be enough.

We might have:

India
Europe
US

with regional services:

India Region
   ↓
India Inventory
India Orders

Europe Region
   ↓
Europe Inventory
Europe Orders

US Region
   ↓
US Inventory
US Orders

Inventory is particularly difficult because it is mutable state.

A product can exist in multiple warehouses:

India → 100
Europe → 50
US → 200

We should avoid making every inventory update depend on one globally shared database if possible.

Instead, inventory can be partitioned by fulfillment region or warehouse.

Catalog and search data can be replicated more freely because those workloads tolerate eventual consistency.

46. Incident: Elasticsearch Goes Down

Suppose Elasticsearch becomes unhealthy.

Search requests start failing or becoming slow.

The important thing is that checkout should still work.

A customer who already knows the product ID should still be able to open the product page through:

Product DB / Redis

The system can:

Route to healthy search replicas
       ↓
Reduce non-critical search features
       ↓
Recover Elasticsearch

Search degradation should not bring down the entire commerce platform.

47. Incident: Inventory Reservation Leak

Suppose customers complain:

"The product says sold out, but nobody can buy it."

One possible cause is that reservations were never released.

For example:

Customer starts checkout
       ↓
Inventory reserved
       ↓
Payment fails
       ↓
Reservation never released

Over time:

Available inventory → artificially decreases

A reservation expiry mechanism and reconciliation job prevent this.

For example:

Reservation created
       ↓
15-minute TTL
       ↓
Payment not completed
       ↓
Release inventory

48. Incident: Duplicate Payment

Suppose a customer sees two payment authorizations.

We investigate using:

orderId
paymentId
idempotencyKey
provider transaction ID

The system should:

Identify duplicate
       ↓
Void/refund duplicate
       ↓
Fix retry/idempotency behavior

The important lesson is that payment APIs should be designed for retries from the beginning.

49. Cost and Operational Trade-offs

Every additional distributed system has an operational cost.

Our architecture now contains:

Redis
Elasticsearch
Kafka
Multiple databases
Payment provider
CDN

Each adds:

Infrastructure cost
Monitoring
On-call responsibility
Failure modes
Operational complexity

For a small e-commerce application, this would be overengineering.

A much smaller system could start with:

Monolith
   ↓
PostgreSQL
   ↓
Redis

and introduce Elasticsearch, Kafka, and separate services only when the scale justifies them.

This is an important architectural principle:

Don't introduce distributed complexity until the workload requires it.

50. How the Architecture Evolves

A sensible system does not need to start with twenty services.

A small platform might begin as:

Monolith
   |
PostgreSQL
   |
Redis

The product catalog, cart, orders, and inventory can initially live in one application.

As traffic grows, search can be separated:

Application
    |
    ├── PostgreSQL
    ├── Redis
    └── Elasticsearch

As asynchronous workloads grow:

Application
    |
    └── Kafka
          |
          ├── Notifications
          ├── Analytics
          └── Search indexing

As checkout becomes more complex:

Inventory Service
Payment Service
Order Service

can be separated.

During flash sales:

Redis atomic counters
+
Queue
+
Dedicated workers

can protect the core transactional systems.

Finally, global expansion can introduce:

Multi-region services
Multi-warehouse inventory
Regional order processing
Disaster recovery

The architecture evolves because the problems evolve.

51. The Most Important Trade-offs

There is no universally correct choice.

PostgreSQL vs MongoDB

PostgreSQL is excellent for:

Orders
Payments
Inventory

because transactions matter.

MongoDB/DynamoDB can be attractive for:

Flexible product attributes

Database Lock vs Atomic Update

Row locks are easy to understand but can create contention.

Atomic updates usually provide better throughput for simple inventory decrements.

Redis vs Database

Redis provides:

Speed
High throughput
Atomic counters

The database provides:

Durability
Strong source of truth

For flash sales:

Redis → absorb the burst
Database → durable truth

Saga vs 2PC

2PC gives stronger transactional coordination but introduces blocking and coupling.

Saga uses:

Local transactions
+
Compensation

and fits microservice workflows better.

Synchronous vs Asynchronous

Keep critical checkout operations synchronous.

Move non-critical work to Kafka:

Email
Analytics
Recommendations
Seller notifications

Search Index vs Primary Database

Elasticsearch is excellent for search but is not the source of truth.

The primary catalog database remains authoritative.

52. The Final Architecture

After all the deeper decisions, the architecture can be summarized as:

                           USERS
                             |
                             ▼
                       CDN / Gateway
                             |
            ┌────────────────┼─────────────────┐
            ↓                ↓                 ↓
         Product           Search              Cart
         Service           Service            Service
            |                |                 |
            ↓                ↓                 ↓
        Product DB     Elasticsearch          Redis
            |
            ↓
           Kafka
            |
      ┌─────┴──────────┐
      ↓                ↓
  Inventory          Other
   Service           Consumers
      |
      ↓
Strong Inventory DB
      |
      ↓
   Checkout
      |
 ┌────┼───────────────┐
 ↓    ↓               ↓
Inv  Pricing        Payment
                      |
                      ↓
               Payment Gateway
                      |
                      ↓
                    Order
                      |
                      ↓
                    Kafka
                      |
          ┌───────────┼───────────┐
          ↓           ↓           ↓
     Notification  Warehouse   Analytics

During a flash sale, the inventory path changes:

Millions of Users
       ↓
Rate Limiter
       ↓
Queue
       ↓
Redis Atomic Counter
       ↓
Order Workers
       ↓
Inventory DB

The architecture therefore has two very different paths.

Read path

User
 ↓
CDN / Cache
 ↓
Search / Product DB

Optimized for:

Very high throughput
Low latency
Eventual consistency

Transaction path

User
 ↓
Checkout
 ↓
Inventory
 ↓
Payment
 ↓
Order

Optimized for:

Correctness
Strong consistency
Idempotency
Failure recovery

53. The Core Ideas to Remember

The entire design becomes much easier to remember if we reduce it to a few principles.

First, separate read-heavy workloads from write-critical workloads.

Read-heavy
→ CDN
→ Redis
→ Elasticsearch

Write-critical
→ Transactional database

Second, inventory is the main concurrency problem.

Many buyers
     ↓
Limited stock
     ↓
Exactly one valid reservation

Third, checkout is a distributed transaction.

Inventory
   +
Payment
   +
Order

Use:

Saga
+
Compensation
+
Idempotency

Fourth, flash sales require traffic absorption.

Rate limiter
   ↓
Queue
   ↓
Redis
   ↓
Workers
   ↓
Durable DB

Fifth, Kafka moves non-critical work away from the request path.

Order Created
      ↓
Kafka
      ↓
Notifications
Analytics
Warehouse
Recommendations

Finally:

The database should remain the source of truth for business-critical state, while caches, search indexes, queues, and event streams help the system scale.

54. How to Explain This in a System Design Interview

A clean explanation can naturally progress like this:

Start by saying:

"I'd separate the system into a read-heavy browsing path and a consistency-critical transaction path."

Then explain the read path:

Catalog
   ↓
Elasticsearch
   ↓
Redis / CDN

Then move to checkout:

Cart
 ↓
Inventory
 ↓
Payment
 ↓
Order

Then explain the hardest problem:

"The biggest correctness issue is preventing overselling when multiple users attempt to buy the last item concurrently."

Explain the normal solution:

Atomic database update

Then the flash-sale solution:

Redis atomic counter
+
Queue
+
Durable inventory database

Then explain checkout consistency:

Saga
+
Compensation

Then payment:

Idempotency
+
Authorization / Capture

Then asynchronous processing:

Kafka

Finally, discuss:

Failure recovery
Observability
Sharding
Multi-warehouse
Multi-region
Trade-offs

That gives you a natural conversation instead of a list of technologies.

55. Final Mental Model

If you remember only one picture, remember this:

                         E-COMMERCE
                             |
             ┌───────────────┼────────────────┐
             ↓               ↓                ↓
          CATALOG          SEARCH             CART
             |               |                |
          Product DB     Elasticsearch       Redis
             |               |                |
             └───────────────┼────────────────┘
                             |
                          CHECKOUT
                             |
              ┌──────────────┼──────────────┐
              ↓              ↓              ↓
          INVENTORY        PRICING        PAYMENT
              |                             |
         Strong DB                    Payment Gateway
              |                             |
              └──────────────┬──────────────┘
                             ↓
                           ORDER
                             |
                             ↓
                           Kafka
                             |
              ┌──────────────┼──────────────┐
              ↓              ↓              ↓
        Notification     Warehouse       Analytics

And during extreme traffic:

Millions of users
       ↓
Rate Limiter
       ↓
Queue
       ↓
Redis
       ↓
Workers
       ↓
Transactional Systems

The key lesson is not that every e-commerce system must use MongoDB, Redis, Elasticsearch, Kafka, and microservices.

The key lesson is why each component exists:

Elasticsearch → scalable search
Redis         → fast reads / hot state / atomic counters
Kafka         → asynchronous event processing
Database      → durable business truth
Queue         → backpressure
Saga          → distributed workflow
Idempotency   → safe retries
CDN           → offload static traffic

Once those relationships are clear, the architecture becomes much easier to design from scratch in an interview.

Designing an Online Auction / Bidding System

Rohit Kori — Mon, 10 Aug 2026 16:36:17 +0000

Imagine an auction website similar to eBay.

A seller lists an item:

Vintage Camera
Starting bid: $500

Other users can open the auction page and place bids:

User A → $550
User B → $600
User C → $700

Everyone currently watching the auction should see the highest bid change in real time.

Eventually, the auction ends. The highest valid bidder becomes the winner, receives a notification, and gets a limited amount of time to complete payment.

That sounds simple.

But once we have millions of users, thousands of active auctions, real-time bid updates, concurrent bids, auction expiration, and payment failures, the design becomes an interesting distributed-systems problem.

This article builds the system from the ground up and gradually addresses those problems.

1. Understanding the Auction

Let's first define what an auction actually means in our system.

A seller creates an auction for an item.

Other users can:

View auction
   ↓
See current highest bid
   ↓
Place a higher bid

The important rule is that an auction does not necessarily end at a fixed clock time.

Instead, in this design:

The auction closes when there has been no higher bid for one hour.

For example:

10:00 → User A bids $500
10:20 → User B bids $550
10:45 → User C bids $600

The one-hour timer is effectively extended by the latest bid.

If no higher bid arrives after:

10:45 + 1 hour

the auction can be closed and User C becomes the winner.

After that:

Winner
   ↓
Payment notification
   ↓
10-minute payment window
   ↓
Payment succeeds → Auction succeeds
Payment fails/expires → Auction fails

2. Requirements

Before designing the system, let's make the rules explicit.

A user should be able to:

Create an auction
View an active auction
Place a bid
See the current highest bid in real time

The system should:

Close an auction after one hour without a higher bid
Determine the winner
Notify the winner
Give the winner 10 minutes to pay

There are also a few important business rules.

If two users submit the same bid amount, the first bid wins.

A bidder can only have one active bid in a particular auction.

However, the bidder can increase that bid later.

For example:

User A → $500
User A → $600
User A → $700

The latest higher bid becomes the user's current bid.

At the same time, we still keep the complete bid history.

So the system remembers:

$500
$600
$700

rather than keeping only $700.

For simplicity, we won't require a separate user-provided TTL for auctions that never receive a bid.

Search, payment processing, and inventory are treated as separate systems. Our focus is the auction service itself.

3. What Does the System Need to Guarantee?

The system has two very different kinds of data.

The first is the live bidding experience.

A customer watching an auction wants to see:

Current highest bid: $700

If the UI briefly shows $650 while another bid of $700 is being processed, that isn't necessarily catastrophic.

The live-bidding path can therefore tolerate some eventual consistency.

The second is winner selection.

When the auction closes, we must be absolutely certain who won.

We cannot have:

Client A → Winner: User A
Client B → Winner: User B

for the same auction.

Therefore:

Live bid display
    ↓
Eventual consistency acceptable

Winner determination
    ↓
Strong consistency required

This distinction is one of the most important ideas in the design.

4. Estimating the Scale

Let's use the assumptions from the original design.

Suppose the platform has:

1 billion daily active users
100,000 auctions created per day
10% of users place one bid per day

That gives roughly:

100M bids/day

The average bid rate is approximately:

100M / 86,400
≈ 1,157 bids/sec

But average traffic isn't enough.

Traffic will be bursty.

Some auctions will attract almost no attention:

Auction A → 3 bidders

while a popular auction might attract enormous traffic:

Auction B → millions of viewers
Auction B → thousands of concurrent bids

The design therefore needs to handle both:

Large overall traffic
+
Hot individual auctions

The original design also assumes roughly a:

10:1 read-to-write ratio

That means viewing auction state is much more common than placing bids.

This becomes important when we design the real-time update path.

5. The Basic Architecture

At a high level, the system looks like this:

                           USERS
                             |
                             ▼
                       Load Balancer
                             |
             ┌───────────────┼────────────────┐
             ↓               ↓                ↓
       Auction Service   Bid Update       Fulfillment
             |             Service           Service
             |                |                |
             ↓                ↓                ↓
        Auction DB       Dispatcher       Auction DB
             |
             ↓
           Cache
             |
             ↓
            Kafka
             |
       ┌─────┴──────┐
       ↓            ↓
 Notifications   Reconciliation

There are several important components:

Auction Service
    ↓
Creates auctions and accepts bids

Auction DB
    ↓
Durable source of truth

Cache
    ↓
Fast access to current auction state

Bid Update Service
    ↓
Maintains real-time connections to viewers

Dispatcher
    ↓
Routes bid updates to the correct Bid Update Service

Fulfillment Service
    ↓
Detects auctions that should end and processes winners

Notification Service
    ↓
Notifies winners

Reconciliation Service
    ↓
Detects and repairs abnormal states

Let's build these pieces one by one.

6. Creating an Auction

Creating an auction is relatively straightforward.

The client sends:

POST /api/v1/auctions

with information such as:

{
  "itemId": "item-123"
}

The Auction Service creates a row in the auction database.

A simplified model is:

Auction
-------------------------
auction_id
owner_id
item_id
status
created_at
updated_at
expire_at
winner_id
winner_bid_id
winner_price
payment_expire_at

The important fields are:

status
expire_at

because they control the auction lifecycle.

A newly created auction starts as:

ACTIVE

and its initial expiration time is established according to the auction's bidding rules.

The service also places the auction state into the cache.

7. Why Make the Auction Service Stateless?

The Auction Service does not need to remember auction state inside its own process.

Instead:

Auction Service
      ↓
Cache / DB

Any Auction Service instance can process a request.

For example:

User A
  ↓
Auction Service #1

User B
  ↓
Auction Service #7

User C
  ↓
Auction Service #12

All of them can access the same external state.

This is what makes the stateless design easy to scale horizontally.

If one instance fails:

Request
   ↓
Another Auction Service instance

The auction data is still available.

8. The Auction Database and Cache

We need durable storage and fast access.

The database contains the complete auction state and bid history.

The cache contains the information we need frequently.

A useful cache entry is:

auction:{auction_id}

{
    status,
    highest_bid,
    highest_bidder_id,
    updated_at,
    expire_at
}

For example:

auction:A123

status            = ACTIVE
highest_bid       = $700
highest_bidder_id = U456
updated_at        = 10:45:12
expire_at         = 11:45:12

The cache is useful because thousands or millions of users may repeatedly ask:

"What is the current highest bid?"

We don't want every one of those reads to hit the database.

But an important rule is:

The cache is not the ultimate source of truth.

The Auction DB contains the durable record.

9. The First Consistency Problem

Suppose we successfully write a bid to the database:

DB write → SUCCESS

but then the cache update fails:

Cache update → FAILURE

Now we have:

Database → $700
Cache    → $650

This is a cache inconsistency.

We can retry the cache update.

But what if the retry also fails?

We therefore need mechanisms to detect stale cache entries.

That is why the cache stores:

updated_at

The system can use that timestamp to determine whether cached information is sufficiently fresh.

When necessary, it can read the database and repair the cache.

This is essentially a form of read repair.

10. How Do Users Receive Live Bid Updates?

Now we reach one of the most interesting parts.

Imagine 50,000 people are watching the same auction.

When somebody bids:

User A → $700

we need to push the update to all those viewers.

We have several possible technologies.

The main choices are:

HTTP polling
Long polling
WebSocket
Server-Sent Events (SSE)

Polling would mean:

Client
 ↓
"Any new bid?"
 ↓
Server
 ↓
Client
 ↓
"Any new bid?"
 ↓
Server

This creates unnecessary traffic.

Long polling is better, but still requires repeated HTTP requests.

WebSocket provides a persistent bidirectional connection.

SSE provides a persistent one-way connection:

Server
  ↓
Client

For this auction system, the client mainly needs to receive updates.

The bid itself can still be sent through a normal HTTP request.

Therefore SSE is a natural fit.

11. Why SSE Works Well Here

The two directions are different.

When a user places a bid:

Client → Server

we can use:

POST /api/v1/auctions/{auctionId}/bids

When the server tells users that somebody else has placed a higher bid:

Server → Client

we can use SSE.

So:

Bid placement
    ↓
HTTP

Live updates
    ↓
SSE

WebSocket would also work, especially if the product later requires richer bidirectional real-time communication.

But for simple one-way live updates, SSE is less complex.

12. Connecting a User to a Bid Update Service

When a user opens an active auction page, the client first gets the auction details:

GET /api/v1/auctions/{auctionId}

If the auction is still active, the browser opens an SSE connection.

A load balancer may route the connection to any Bid Update Service instance.

For example:

User U1 → bus1
User U2 → bus1
User U3 → bus2
User U4 → bus3

Each Bid Update Service keeps an in-memory subscription table.

For example:

bus1

auction A1 → [U1, U2]
auction A2 → [U5]

This tells the service:

These users are currently watching these auctions.

13. Why Do We Need a Dispatcher?

Suppose a bid arrives for auction A1.

The Auction Service knows:

Auction = A1
New highest bid = $700

But which Bid Update Service has the viewers?

It might be:

bus1

The Auction Service should not need to know the internal connection state of every Bid Update Service.

So we introduce a Dispatcher.

The Dispatcher maintains another subscription table:

Dispatcher

A1 → bus1
A2 → bus2
A3 → bus1

Now the flow becomes:

User places bid
       ↓
Auction Service
       ↓
Dispatcher
       ↓
Correct Bid Update Service
       ↓
Connected viewers

This separates responsibilities:

Auction Service
    → process business logic

Dispatcher
    → route update

Bid Update Service
    → maintain client connections

14. The Full Bid Update Flow

Suppose:

Auction A1
Current bid = $600

User U10 places:

$700

The request goes:

U10
 ↓
Auction Service

The Auction Service:

Checks auction status
       ↓
Writes bid to DB
       ↓
Updates highest bid in cache
       ↓
Sends update to Dispatcher

The Dispatcher checks:

A1 → bus1

and forwards the update:

Dispatcher
    ↓
bus1

The Bid Update Service checks:

A1 → [U1, U2, U3, U4]

and sends:

New highest bid = $700

to those SSE connections.

The complete flow is:

Bidder
  ↓
Auction Service
  ↓
Auction DB + Cache
  ↓
Dispatcher
  ↓
Bid Update Service
  ↓
SSE
  ↓
All viewers

15. Why Not Just Poll the Database?

A naive design would be:

Bid Update Service
       ↓
Poll Auction DB
       ↓
Find new bids
       ↓
Push to users

This sounds simple.

But imagine:

100,000 active auctions

and each auction is being polled every few seconds.

The database would receive enormous numbers of unnecessary queries.

Most queries would return:

Nothing changed.

Instead of constantly asking the database:

"Did something happen?"

we push the event when something actually happens.

That is much more efficient.

16. Making the Dispatcher Highly Available

The Dispatcher is stateful because it maintains:

auction → Bid Update Service

If the Dispatcher fails, bid updates cannot be routed to viewers.

The actual auction may still work, but the live experience breaks.

There are several ways to make it resilient.

One option is to maintain a write-ahead log and snapshots:

Subscription changes
        ↓
WAL
        ↓
Snapshot

If the Dispatcher crashes:

Snapshot
   +
WAL
   ↓
Rebuild subscription table

Another option is to replicate the state into an external key-value store.

A third option is an active-standby design:

Primary Dispatcher
       ↓
Standby Dispatcher

If the primary fails:

Standby → becomes primary

17. Could We Remove the Dispatcher?

Yes.

Instead of:

Auction Service
      ↓
Dispatcher
      ↓
Bid Update Service

we could maintain the subscription mapping in a distributed key-value or coordination service.

Then:

Bid Update Service
      ↓
Coordination Store

and the Auction Service can look up which Bid Update Service is responsible for an auction.

There is a trade-off.

With a Dispatcher:

Pros:
- Auction Service has less responsibility
- Dispatcher can scale independently
- Retry logic is centralized

But:

Cons:
- Additional component
- More operational complexity

Without a Dispatcher:

Pros:
- Simpler architecture
- Fewer components

But:

Cons:
- Auction Service handles forwarding
- Retry logic becomes its responsibility

The right choice depends on how much complexity the system can justify.

18. What Happens If a Bid Update Is Lost?

Real-time systems sometimes lose messages.

Suppose:

$700 bid happens

but one client never receives the update.

Is the auction broken?

Not necessarily.

During an active auction, another bid may soon arrive:

$700
 ↓
$750
 ↓
$800

The missing $700 event becomes less important because newer updates overwrite the displayed state.

The dangerous case is the last bid.

Suppose:

$700

is the final bid, and the client never receives it.

A useful recovery mechanism is to have the client periodically check for stale updates.

For example:

No bid update for a while
        ↓
Hard pull
        ↓
GET /auctions/{id}
        ↓
Retrieve current authoritative state

This combines:

Fast push
+
Occasional authoritative pull

and makes the system resilient to lost live events.

19. Placing a Bid

Now let's look more closely at the actual bid request.

The client sends:

POST /api/v1/auctions/{auctionId}/bids

with:

{
  "bidAmount": 700,
  "requestId": "req-123"
}

The Auction Service first checks:

Does auction exist?
Is status ACTIVE?

It can check the cache first:

Cache
 ↓
ACTIVE?

If the cache doesn't contain the auction, it can fall back to the database.

This cache miss should normally be a corner case.

20. Recording Bid History

Once the auction is confirmed to be active, the bid is written to the bid table.

A useful schema is:

Bid
-------------------------
bid_id
auction_id
bidder_id
amount
request_id
created_at

The original design uses an append-only pattern.

That means:

User A → $500
User A → $600
User A → $700

creates three records.

We don't overwrite the previous rows.

This gives us a complete audit trail.

The latest valid bid for a bidder can be treated as their current bid.

The request ID or insertion timestamp can help determine ordering more robustly than relying only on client timestamps.

21. Updating the Highest Bid

After the bid is persisted, the service checks whether it is higher than the current cached bid.

Suppose:

Cache:
highest_bid = $600

and the new bid is:

$700

Then:

highest_bid
    ↓
$700

highest_bidder_id
    ↓
U10

The cache is updated.

Then the Dispatcher is notified so that live viewers receive the new value.

If the new bid is lower than the current highest bid, it is still stored in the bid history but does not change the current highest-bid state.

Under our business rule, a bidder can only increase their own bid.

22. Why the Append-Only Bid Table Is Useful

An append-only design gives us several advantages.

It provides:

Complete history
Auditability
High write throughput
Simple writes

For example:

Bid 101 → U1 → $500
Bid 102 → U2 → $550
Bid 103 → U1 → $600
Bid 104 → U3 → $700

We can later reconstruct what happened.

It also avoids repeatedly modifying one large bid record.

23. A Hot Auction Creates a Hot Key

There is an important scaling problem.

Most auctions may have:

5–10 bidders

but one extremely popular auction could have:

Millions of viewers
Thousands of concurrent bids

All of these operations revolve around:

auction_id = A123

If our cache partitions by auction ID, all updates may land on the same partition.

This is a classic hot-key problem.

A single auction can become a bottleneck even when the overall system has plenty of capacity.

24. Handling Hot Auctions

There are several approaches.

One option is to use a lease or lock mechanism to coordinate concurrent updates.

Conceptually:

Bid request
    ↓
Acquire lease for auction A123
    ↓
Update highest bid
    ↓
Release lease

The advantage is that concurrent writers are coordinated.

The disadvantage is that a request may have to retry if another writer currently owns the lease.

Another approach is replicated storage with quorum-style behavior.

If the system is designed so that:

Higher bid always wins

then conflict resolution becomes relatively simple.

For example:

Replica A → $700
Replica B → $750

The conflict resolver can choose:

max($700, $750)
= $750

This works particularly well because the business rule does not allow a bidder to reduce their own bid.

25. Auction Expiration Is a Scheduling Problem

Eventually, the auction must end.

We don't want every Auction Service instance constantly scanning every auction.

Instead, we can use a Fulfillment Service.

Its job is similar to a scheduler.

It periodically looks for auctions whose:

status = ACTIVE

and:

expire_at <= now

The cache can make this check efficient because it already contains:

status
expire_at

26. Determining the Winner

The Fulfillment Service finds an auction that appears ready to close.

But we should not blindly trust the cache.

The cache might be stale.

So the service asks the Auction DB to verify the winner.

Conceptually:

Fulfillment Service
        ↓
"Is this really the current winning bid?"
        ↓
Auction DB

If the cache was stale:

Cache → $700
DB    → $750

the Fulfillment Service can repair the cache.

This is another example of read repair.

27. Moving the Auction to Payment

Once the winner is confirmed, the auction transitions from:

ACTIVE

to:

PAYMENT_PENDING

The database records:

winner_id
winner_bid_id
winner_price
payment_expire_at

These values should be updated together as one logical state transition.

The cache is updated as well.

Then the notification system sends a message to the winner:

Congratulations!

You won the auction for $750.

Please complete payment within 10 minutes.

The live viewers can also receive an auction-closed update through the Dispatcher.

28. Payment Completion

Payment itself is handled by a separate payment system.

The auction does not need to own the payment implementation.

The flow is:

Auction
   ↓
PAYMENT_PENDING
   ↓
Payment Service
   ↓
Payment succeeds
   ↓
Auction → SUCCEEDED

The payment service should be idempotent so retries don't accidentally create duplicate charges.

The auction system only needs to reliably react to the final payment result.

29. What If the Winner Doesn't Pay?

The winner has:

10 minutes

to complete payment.

The Fulfillment Service periodically checks auctions in:

PAYMENT_PENDING

If:

payment_expire_at < now

and the payment hasn't succeeded:

PAYMENT_PENDING
        ↓
FAILED

The item can then be handled according to the broader marketplace policy.

The important point is that the auction system has another timed state transition:

ACTIVE
   ↓
PAYMENT_PENDING
   ↓
SUCCEEDED / FAILED

30. Why the Fulfillment Service Reads the Cache

There is a deliberate trade-off here.

The Fulfillment Service could query the database directly:

Find every ACTIVE auction
   ↓
Check expire_at
   ↓
Find highest bid
   ↓
Execute expired auctions

The problem is that this can require expensive queries over the Auction DB.

Instead, the cache already contains:

status
highest_bid
highest_bidder_id
expire_at

So the Fulfillment Service can use the cache to find candidates quickly.

The trade-off is:

Cache approach
    ↓
Lower latency
Less DB load
But possible stale data

Database approach
    ↓
More accurate
No cache dependency
But more DB load and more expensive queries

The final design can combine both:

Cache → find candidate
   ↓
DB → verify

This gives us both performance and correctness.

31. The Reconciliation Service

Distributed systems can end up in abnormal states.

For example:

Payment succeeded
       ↓
Auction DB was not updated

Now we have:

Payment = SUCCESS
Auction  = PAYMENT_PENDING

A Reconciliation Service periodically searches for these inconsistencies.

It can compare:

Auction state
Payment state
Cache state

and repair the auction.

For example:

Payment says SUCCESS
Auction says PAYMENT_PENDING
       ↓
Reconciliation
       ↓
Auction → SUCCEEDED

This gives the system a recovery mechanism instead of relying only on the happy path.

32. The Complete Stateless Design

Putting everything together:

                           USER
                            |
                            ▼
                       Load Balancer
                            |
                 ┌──────────┴──────────┐
                 ↓                     ↓
          Auction Service       Bid Update Service
                 |                     |
          ┌──────┴──────┐              |
          ↓             ↓              |
      Auction DB      Cache            |
          |             |              |
          └──────┬──────┘              |
                 ↓                     |
               Kafka                   |
                 |                     |
                 ↓                     |
          Async Consumers              |
                                       |
User SSE ←─────────────────────────────┘

Auction Service
      |
      ↓
Dispatcher
      |
      ↓
Bid Update Service
      |
      ↓
SSE → viewers

Cache
      |
      ↓
Fulfillment Service
      |
      ↓
Auction DB
      |
      ↓
Winner
      |
      ↓
Notification
      |
      ↓
Payment Service
      |
      ↓
SUCCEEDED / FAILED

Reconciliation Service
      |
      └── checks abnormal states

The Auction Service itself remains stateless.

The state lives in:

Auction DB
Cache

and temporary connection state lives in:

Bid Update Service
Dispatcher

33. Stateful Auction Service

So far we've used a stateless architecture.

There is another interesting option.

We could make the Auction Service itself stateful.

When an auction is created:

Auction A123
      ↓
Assigned to Auction Service #5

All bids for that auction are then routed to the same server:

Auction A123
      ↓
Auction Service #5
      ↓
All bids

The server could keep the current auction state in memory.

This reduces the need for multiple servers to coordinate on the same auction.

34. Routing Requests to the Correct Server

With a stateful design, the load balancer needs to know:

Auction A123 → Server #5
Auction A456 → Server #8

This requires service discovery or a consistent routing mechanism.

The request:

POST /auctions/A123/bids

must always reach the instance responsible for:

A123

This can make per-auction ordering and consistency easier.

35. The Problem With Stateful Servers

The stateful design introduces a new problem.

Suppose:

Auction A123
      ↓
Server #5
      ↓
In-memory state

and Server #5 crashes.

We lose the in-memory state.

Therefore, the server needs recovery mechanisms such as:

Write-ahead log
+
Snapshots

or rebuilding state from the Auction DB.

We may also replicate the state:

Primary
   ↓
Follower

so that the follower can take over after failure.

This makes the architecture more complicated.

36. Stateless vs Stateful

The two approaches have different strengths.

Area	Stateless	Stateful
Consistency	More coordination required	Easier per-auction ordering
Availability	Easier	Harder because state must be recovered
Scaling	Easier to add nodes	More difficult due to routing
Hot auctions	Can be challenging	One server can become a hotspot
Failure recovery	External state survives node loss	In-memory state needs recovery
Operational complexity	Generally simpler	More complex

The stateless approach is usually more common.

The stateful approach is still useful when processing a stream of events belonging to the same entity.

37. High Availability

Let's examine what happens when individual components fail.

The stateless Auction Service is relatively easy to make highly available.

If one node fails:

Client
   ↓
Retry
   ↓
Another Auction Service

Because the state is external, the new node can continue processing.

Duplicate requests are possible.

For auction creation, we can use an idempotent request or an upsert-like operation to prevent duplicate auctions.

For bids, the append-only model makes duplicate writes easier to handle, especially when requests have unique IDs.

38. Dispatcher Availability

The Dispatcher is different because it maintains state.

If it fails, the routing table disappears.

Possible solutions include:

WAL + snapshots
External replicated KV store
Active / standby

The important idea is:

Stateful components need a recovery story.

39. Bid Update Service Availability

The Bid Update Service maintains live client connections.

Its state is:

User connection
Auction subscription

But this state is tied to the lifetime of the connection.

If the server crashes:

SSE connection dies
       ↓
Client reconnects
       ↓
Another Bid Update Service

We don't necessarily need to persist every connection in a durable database.

The connection state is temporary.

This is different from:

Auction state
Bid history
Winner
Payment state

which must survive server failures.

40. Cache and Database Replication

The cache and Auction DB are both important infrastructure.

Different replication strategies are possible:

Single leader
Multi-leader
Quorum

A single leader is simpler.

A replicated cache/database can improve availability.

Quorum replication can improve durability and consistency at the cost of more coordination.

The right strategy depends on the underlying technology and the consistency guarantees we need.

For the auction's authoritative state, correctness during winner selection is more important than squeezing out the last bit of write latency.

41. Scaling the Auction Service

In the stateless architecture, scaling is straightforward:

More traffic
    ↓
Add more Auction Service instances

A load balancer distributes requests.

In a stateful design, scaling is harder because auctions need to be assigned to specific servers.

We can shard auctions using a key such as:

auction_id

or:

owner_id

The original design notes that auction_id gives good co-location, but can create hot partitions.

Partitioning by owner_id may distribute traffic differently.

The correct partition key depends on the actual workload.

42. Scaling the Dispatcher

The Dispatcher keeps a table such as:

auction → Bid Update Service

The memory footprint can be manageable.

But memory size isn't the only concern.

The Dispatcher may receive a very high number of requests.

Therefore, we can scale it using:

Read replicas
Sharding
Partitioning by auction_id

Replication can be:

Synchronous

for stronger consistency, or:

Asynchronous

when eventual consistency is acceptable.

43. Scaling the Cache and Auction Database

There are several possible partitioning strategies.

Partition by:

auction_id

This has a useful property:

Auction data
+
Bid data

can be co-located.

But a popular auction can become a hot partition.

Another option is partitioning by:

user_id

This can distribute writes more evenly because an individual user is less likely to become a massive hotspot.

Rate limiting can further protect the system from unusually active users.

There is no universally correct partition key.

We choose based on the workload.

44. Scaling Bid Update Services

Bid Update Services are relatively easy to scale.

Each node maintains its own in-memory connections:

bus1 → users
bus2 → users
bus3 → users

When the number of connections grows:

Add more Bid Update Service instances

The load balancer distributes new SSE connections across them.

The Dispatcher keeps track of which service owns the subscriptions.

45. Scaling Fulfillment

The Fulfillment Service can also be distributed.

Auctions can be partitioned by:

auction_id

and different workers can process different partitions.

For example:

Worker 1 → A–F
Worker 2 → G–M
Worker 3 → N–S
Worker 4 → T–Z

The important requirement is to prevent two workers from closing the same auction simultaneously.

The final winner transition must therefore be protected by an atomic database update or equivalent concurrency control.

46. Cache and Auction DB Consistency

Let's revisit one of the most subtle problems.

Suppose the system does:

1. Write bid to DB
2. Update cache

The DB write succeeds:

DB = $700

but the cache update fails:

Cache = $650

A retry can fix it.

But retries can also fail.

Therefore, the cache entry includes:

updated_at

When the system detects that the cached state is stale:

Cache
  ↓
Stale?
  ↓
Read DB
  ↓
Repair cache

This can happen when:

Serving a read

or:

Executing an auction

This is why the cache is treated as a fast representation of the state rather than the final authority.

47. Write-Through vs Write-Back

There are two broad ways to synchronize cache and database.

With write-through-style behavior:

Write DB
   ↓
Update Cache

The database is updated immediately.

With write-back:

Update Cache
   ↓
Persist to DB later

Write-back can reduce database writes in some workloads.

For example, if we wanted to update the winning bid in the auction table on every bid, write-back could reduce the number of direct database writes.

But it also makes durability and failure recovery more complicated.

For the auction design, keeping the bid history durably in the database and using the cache for the current highest-bid state is a safer model.

48. SSE vs WebSocket

Both technologies can provide real-time communication.

	SSE	WebSocket
Direction	Server → Client	Bidirectional
Protocol style	HTTP	WebSocket
Data	Text/event stream	Text + binary
Reconnection	Built in	Application typically handles it
Best suited for	One-way live updates	Interactive two-way communication

For our auction:

Bid placement
→ HTTP

Bid updates
→ SSE

This is simple because the server mainly pushes state to viewers.

WebSocket becomes more attractive if the product eventually needs richer bidirectional interaction.

49. Another Real-Time Design

There is another possible connection strategy.

Instead of opening an SSE connection every time a user navigates to an auction, the application could maintain one long-lived WebSocket connection after login.

For example:

User logs in
     ↓
WebSocket established
     ↓
User opens Auction A
     ↓
Subscribe to A
     ↓
User opens Auction B
     ↓
Unsubscribe A
     ↓
Subscribe B

This may be useful if users frequently move between auctions.

The right choice depends on how users interact with the product.

50. Reliability of Live Updates

We don't necessarily need exactly-once delivery for every live bid update.

Suppose a client receives:

$700
$700

twice.

The UI can simply keep:

max(currentBid, receivedBid)

and display:

$700

Similarly, if the client receives:

$700
$750
$800

and $750 is duplicated, there is no business impact.

This makes the real-time layer easier to design.

The important part is that the final authoritative winner comes from the database.

51. API Design

The core API surface can be kept simple.

Create an auction:

POST /api/v1/auctions

Get an auction:

GET /api/v1/auctions/{auctionId}

Place a bid:

POST /api/v1/auctions/{auctionId}/bids
Idempotency-Key: bid-123

Example:

{
  "amount": 700
}

Get bid history:

GET /api/v1/auctions/{auctionId}/bids

Open live updates:

GET /api/v1/auctions/{auctionId}/events

implemented as an SSE stream.

The API should return conflicts such as:

Auction not found
Auction already closed
Bid is not higher than current bid
Duplicate request

with appropriate HTTP status codes.

52. Data Model

A simplified auction table:

Auction
--------------------------------
auction_id
owner_id
item_id
status
created_at
updated_at
expire_at
winner_id
winner_bid_id
winner_price
payment_expire_at

Possible statuses:

ACTIVE
PAYMENT_PENDING
SUCCEEDED
FAILED

The bid table:

Bid
--------------------------------
bid_id
auction_id
bidder_id
amount
request_id
created_at

Indexes should support common access patterns such as:

auction_id + created_at
auction_id + amount
auction_id + bidder_id

The cache stores:

auction_id
status
highest_bid
highest_bidder_id
updated_at
expire_at

The important distinction is:

Bid table
→ complete history

Cache
→ current hot state

53. Auction State Transitions

The lifecycle can be visualized as:

                 ┌──────────────┐
                 │    ACTIVE    │
                 └──────┬───────┘
                        │
                 no higher bid
                    for 1 hour
                        │
                        ▼
              ┌───────────────────┐
              │ PAYMENT_PENDING   │
              └─────────┬─────────┘
                        │
                 ┌──────┴───────┐
                 │              │
             payment         timeout
              success            │
                 │              │
                 ▼              ▼
          ┌───────────┐    ┌────────┐
          │ SUCCEEDED │    │ FAILED │
          └───────────┘    └────────┘

Making the states explicit makes recovery much easier.

54. The Hardest Race: Bid vs Auction Expiration

There is a subtle race condition.

Suppose the auction expires at:

11:00:00

At almost exactly the same time:

User A submits a $900 bid

and:

Fulfillment Service tries to close the auction

Which one wins?

We need a clearly defined ordering rule.

A robust approach is to make the final transition conditional in the database.

For example:

Close auction only if:

status = ACTIVE
AND expire_at <= now

A bid should similarly be accepted only if:

status = ACTIVE
AND current time < expire_at

The database transaction / compare-and-set operation determines which state transition wins.

This is an important example of why final winner selection cannot rely only on cache state.

55. Idempotency and Retries

Distributed systems retry requests.

For example:

Client → Bid Service
       ↓
Request succeeds
       ↓
Network response lost
       ↓
Client retries

Without idempotency:

Same logical bid
   ↓
Two database records

This is why a client request can include:

requestId

or:

Idempotency-Key

The server can detect that the same logical operation has already been processed.

This is especially important for:

Auction creation
Bid placement
Payment
Auction state transitions

56. What Happens If the Auction Service Crashes?

Suppose:

User sends $700 bid

The Auction Service writes the bid successfully:

DB → SUCCESS

but crashes before updating the cache.

After recovery:

DB → $700
Cache → $650

The reconciliation/read-repair mechanism can detect the discrepancy.

The important design principle is:

The durable operation should be recoverable even if the process dies immediately afterward.

57. What Happens If Fulfillment Crashes?

Suppose Fulfillment decides:

Auction A123 should close

but crashes before completing the transition.

Another Fulfillment worker can pick it up.

The final database operation should be conditional:

UPDATE auction
SET status = PAYMENT_PENDING
WHERE auction_id = ?
AND status = ACTIVE
AND expire_at <= now

Only one worker will successfully transition the row.

This makes the operation idempotent and safe to retry.

58. What Happens If Notification Fails?

Suppose:

Auction → PAYMENT_PENDING

but notification delivery fails.

The auction should not remain stuck simply because an email or push notification failed.

Instead:

Auction state
   ↓
Persisted successfully

Notification
   ↓
Async retry

The notification system can retry independently.

This is another reason not to put non-critical side effects directly inside the critical transaction.

59. What Happens If Payment Succeeds but Auction Isn't Updated?

This is one of the most important recovery cases.

Suppose:

Payment
   ↓
SUCCESS

but:

Auction DB
   ↓
Still PAYMENT_PENDING

The Reconciliation Service can detect:

Payment = SUCCESS
Auction = PAYMENT_PENDING

and correct the auction:

Auction → SUCCEEDED

This is why reconciliation is not an optional afterthought in distributed systems.

60. Final Stateless Architecture

The complete stateless design can now be summarized as:

                              USERS
                                |
                                ▼
                         Load Balancer
                                |
                   ┌────────────┴────────────┐
                   ↓                         ↓
            Auction Service           Bid Update Service
                   |                         |
             ┌─────┴─────┐             SSE Connections
             ↓           ↓                   |
        Auction DB     Cache                 |
             |           |                   |
             |           └──────┐            |
             |                  ↓            |
             |             Fulfillment       |
             |                  |            |
             |                  ↓            |
             |             Auction DB        |
             |                               |
             └──────────────┐                |
                            ↓                |
                         Dispatcher ─────────┘
                            |
                            ↓
                    Bid Update Services
                            |
                            ↓
                         Viewers

Auction Service
       |
       ↓
     Kafka
       |
   ┌───┴──────────────┐
   ↓                  ↓
Notification     Reconciliation
   |
   ↓
Winner

Fulfillment
   |
   ↓
Payment Service
   |
   ├── SUCCESS → SUCCEEDED
   |
   └── TIMEOUT → FAILED

The key property is that the Auction Service itself does not keep auction state in memory.

61. When Would We Choose the Stateful Design?

The stateless architecture is usually the better default.

It is easier to:

Scale
Recover
Deploy
Load balance

The stateful design becomes attractive when we need extremely efficient per-auction processing and want all events for one auction handled by the same process.

For example:

Auction A123
   ↓
Stateful server #5
   ↓
All bids for A123

This can make ordering easier.

But now:

Server #5 fails

and we need:

Replica
WAL
Snapshot
Recovery
Routing

Therefore, stateful systems trade simpler per-entity processing for more difficult availability and scaling.

62. The Most Important Trade-offs

There is no single perfect architecture.

The major decisions are:

Stateless vs Stateful

Stateless
→ easier scaling and availability

Stateful
→ easier per-auction ordering

Cache vs Database for Current Bid

Cache
→ fast

Database
→ authoritative

The practical design uses both.

SSE vs WebSocket

SSE
→ simple one-way updates

WebSocket
→ richer bidirectional communication

Dispatcher vs Direct Coordination Store

Dispatcher
→ cleaner separation

Direct coordination
→ fewer components

Cache-driven vs DB-driven Fulfillment

Cache
→ fast, lower DB pressure

DB
→ authoritative, more expensive

The hybrid approach is:

Cache → identify candidate
DB → verify

63. The Design in One Mental Model

The entire system can be remembered through four responsibilities.

First:

Auction Service

handles the business operation:

Create auction
Place bid
Update current bid

Second:

Auction DB

keeps the durable truth:

Auction
Bid history
Winner
Payment state

Third:

Bid Update Service + Dispatcher

handles the real-time experience:

New bid
   ↓
Dispatcher
   ↓
Bid Update Service
   ↓
SSE
   ↓
Viewers

Fourth:

Fulfillment + Reconciliation

handles time and recovery:

Auction expires
     ↓
Determine winner
     ↓
Payment
     ↓
Success / Failure

Abnormal state
     ↓
Reconcile

64. A Natural Interview Walkthrough

If you were explaining this system in an interview, the conversation can naturally progress like this.

Start with the requirements:

"Users can create auctions, view active auctions, place bids, and receive real-time updates. An auction closes after one hour without a higher bid. The winner gets ten minutes to pay."

Then establish the scale:

1B DAU
100K auctions/day
~100M bids/day
10:1 read/write

Then identify the core challenge:

"The most interesting problem is handling real-time updates while maintaining correct winner selection."

Then introduce the architecture:

Auction Service
Auction DB
Cache
Bid Update Service
Dispatcher
Fulfillment

Explain the live path:

HTTP bid
   ↓
DB
   ↓
Cache
   ↓
Dispatcher
   ↓
SSE

Then explain expiration:

Cache
   ↓
Fulfillment
   ↓
DB verification
   ↓
Winner

Then discuss failure:

Retry
Idempotency
Read repair
Reconciliation

Finally compare:

Stateless vs Stateful
SSE vs WebSocket
Cache vs DB
Dispatcher vs coordination store

That gives you a coherent system-design discussion rather than a list of technologies.

65. Final Takeaways

The auction system looks simple from the outside:

Seller lists item
     ↓
Users bid
     ↓
Highest bidder wins
     ↓
Winner pays

The distributed system underneath is much more interesting.

The most important ideas are:

1. Keep bid history durable and append-only.

2. Keep the current highest bid in a fast cache.

3. Use SSE for efficient one-way live updates.

4. Use a Dispatcher to route updates to the right
   Bid Update Service.

5. Treat the Auction DB as the authoritative source.

6. Allow eventual consistency for live display,
   but use strong consistency when selecting the winner.

7. Use a Fulfillment Service to process auction expiration.

8. Verify the winner against the database before closing
   the auction.

9. Use idempotency and conditional state transitions
   so retries are safe.

10. Use reconciliation to repair abnormal distributed states.

11. Watch for hot auctions and hot cache keys.

12. Stateless architecture is easier to scale and recover;
    stateful architecture can simplify per-auction ordering
    but makes failure recovery harder.

13. SSE is sufficient when the main real-time requirement
    is server-to-client updates; WebSocket is useful when
    richer bidirectional communication is needed.

The deepest system-design lesson is this:

The live bidding experience can tolerate some temporary inconsistency, but the final winner cannot.

That single distinction explains why the system combines:

Cache
+
Real-time events
+
SSE
+
Database
+
Scheduler
+
Reconciliation

Each component solves a different part of the problem.

66. Reference

The design and assumptions in this article are based on the free Coding Monkey article:

How to Design Auction System

https://pyemma.github.io/How-to-design-auction-system/

The original article discusses the stateless and stateful designs, live bid routing, SSE, Dispatcher, cache/database consistency, Fulfillment Service, scalability, availability, and SSE/WebSocket trade-offs.