DEV Community: MANGESH MANDLIK

Google Docs System Design: Real-Time Editing, OT, CRDT & Scale

MANGESH MANDLIK — Wed, 28 Oct 2026 03:26:57 +0000

Two people open the same document. Alice inserts a sentence near the top. Bob deletes a word in the same paragraph, before Alice's change reaches his laptop. Both see their own edits immediately. A few moments later, both should see the same document, without silently losing either edit.

That sounds like a text editor with WebSockets. It is actually a distributed synchronization problem.

In this guide, we'll design a Google Docs-style real-time collaborative editor from a single-user autosave application to a production-oriented architecture. We'll examine concurrent edits, Operational Transform (OT), CRDTs, authoritative document ownership, durability, offline recovery, presence, version history, and scaling. This is an illustrative system design—not a description of Google's private implementation.

1. What are we designing?

Users should be able to create and open documents, edit simultaneously, see other collaborators' changes and cursors, reconnect after losing connectivity, review earlier versions, and share documents with appropriate permissions.

The essential engineering requirements are distinct:

Local responsiveness: typing should appear immediately, even when the network is slow.
Convergence: healthy clients that have processed the same accepted history should arrive at the same document state.
Durability: an edit reported as saved must survive failures covered by the stated durability policy.
Authorization: a client must not continue reading or writing after access is revoked.
Scalability: millions of open connections and occasional extremely popular documents must not overload one global server.

Convergence, durability, and preserving a user's intended meaning are not interchangeable guarantees. A synchronization algorithm may converge perfectly while combining two semantically conflicting edits into awkward text.

2. Start simple: one user, one document

For a single-user editor, the client can maintain a local buffer and periodically save the latest body to storage:

Editor → Autosave API → Document database

This breaks under concurrent writers. Suppose both Alice and Bob read version 10. Alice saves a modified body as version 11. Bob then saves his independently modified version 10 body. If the server simply overwrites the entire document, Bob's save can erase Alice's change.

A conditional write that rejects Bob's stale version prevents silent overwrite, but still does not provide a good collaborative experience: Bob needs to reconcile his edits with Alice's. We therefore need operation-based synchronization, not blind whole-document replacement.

3. Evolve to real-time collaboration

The client applies edits optimistically, then sends operations to a backend over a persistent WebSocket connection. A document owner reconciles concurrent operations and assigns authoritative versions.

                         ┌───────────────────────┐
Editor A ── WebSocket ────┤                       │
                         │   WebSocket Gateways  │
Editor B ── WebSocket ────┤  sessions, auth, I/O  │
                         └──────────┬────────────┘
                                    │
                              Shard Router
                                    │
                         ┌──────────▼────────────┐
                         │ Document Owner        │
                         │ validate + dedupe     │
                         │ OT / merge + version  │
                         └──────────┬────────────┘
                                    │
                         Durable Operation Log
                                    │
                         Commit acknowledgement
                                    │
                            Broadcast Layer
                                    │
                           Other subscribers

Separate supporting services:
  Presence / cursors     Snapshots / history
  Offline resync         Permission changes
  Ownership coordination Monitoring

Why WebSockets? Edits, acknowledgements, cursors, and presence updates are small and bidirectional. A persistent connection avoids repeated polling overhead and supports low-latency delivery. HTTP remains useful for document creation, initial snapshot retrieval, sharing, and history browsing. WebSockets provide transport—not correctness, ordering, or durable storage by themselves.

The gateway authenticates sessions, applies message-size and rate limits, and routes traffic. It must not be the source of document truth: a gateway can fail while the document's committed history remains intact.

4. Define the operation protocol before discussing OT

Instead of sending the entire document after every keystroke, a client sends a logical operation. A simplified message might be:

{
  "type": "edit_op",
  "docId": "doc_842",
  "sessionId": "session_19",
  "clientSeq": 391,
  "opId": "op_9012",
  "baseVersion": 1842,
  "operation": {
    "kind": "insert",
    "position": 128,
    "text": " collaborative"
  }
}

Each field solves a different problem:

docId routes the operation to the correct document.
opId identifies a logical edit across retries, allowing idempotent deduplication.
sessionId and clientSeq help identify gaps and ordering within a client session.
baseVersion tells the server which authoritative document version the operation was prepared against.
operation describes the intended mutation, using a schema understood by the synchronization engine.

The server obtains its authoritative ownership epoch from trusted routing/coordination state; it must not trust an arbitrary client-supplied epoch as proof of ownership. The durable writer checks the owner's fencing token.

For a rich-text editor, the operation model also needs structural semantics: paragraphs, formatting spans, lists, tables, and embedded objects. A plain string index is useful for explaining concurrency, but is not a complete rich-document model. Text-offset conventions (for example, Unicode code points versus UTF-16 units) must be consistent across clients and server.

A committed response might look like:

{
  "type": "op_ack",
  "opId": "op_9012",
  "clientSeq": 391,
  "appliedVersion": 1845,
  "status": "committed"
}

Here committed means durably recorded under our chosen durability policy, not merely received by a gateway. A separate received acknowledgement could be sent earlier, but it must not be displayed as saved.

5. The central challenge: concurrent edits

Imagine the document is:

Hello world

Alice inserts beautiful immediately before world. Bob, editing from the same base version, inserts wide at that same position. Applying both original positional inserts without coordination can yield different results depending on arrival order.

A correct collaborative protocol needs deterministic conflict rules. For simultaneous inserts at the same position, a tie-break rule (for example, an agreed ordering of operation identifiers) can produce one consistent outcome. That outcome may be Hello beautiful wide world or Hello wide beautiful world; the algorithm must make replicas agree, not guess the writers' intended sentence.

Naive last-write-wins whole-document saves are unsuitable because one complete document can overwrite the other. Naive application of stale character offsets is also unsafe because earlier edits shift later positions.

6. Operational Transform (OT): reconcile edits against newer history

In a centralized OT-style architecture, the document owner serializes accepted operations and maintains an authoritative version sequence.

Suppose Alice and Bob both start at version 42:

Alice submits an insertion based on version 42.
The owner validates it, commits it as version 43, and distributes the accepted operation.
Bob submits an operation also based on version 42.
The owner transforms Bob's operation against the change committed at version 43, using the editor's defined transformation rules.
The transformed operation is committed as version 44 and distributed.
Clients reconcile their optimistic local edits with the authoritative sequence.

For a simple positional example, an insertion earlier in the text may shift another insertion's position. For overlapping deletes, replacements, and structured content, the rules are substantially more involved. OT is not simply “increment the index whenever someone inserts text.” Correct implementations must handle operation types, ordering, tie-breaking, outstanding local edits, and transformation properties together.

The server needs enough accepted history since baseVersion to transform a stale operation. If that history has been compacted away, it cannot pretend the transform is still possible. It needs a supported snapshot-based rebase or a conflict-resolution path that may require user assistance.

OT is a sensible teaching choice when one authoritative owner assigns a per-document order. It is not the only viable approach.

7. OT vs CRDT: which should we choose?

A Conflict-free Replicated Data Type (CRDT) uses data structures and merge rules designed so replicas can converge despite concurrent updates. For collaborative text, CRDT approaches often associate content with stable element identifiers rather than relying solely on changing character offsets.

Consideration	Operational Transform	CRDT
Core technique	Transform operations against concurrent accepted history	Merge operations/state under defined CRDT rules
Natural fit	Centralized, versioned collaboration	Replica-local and offline-heavy editing
Main complexity	Transformation correctness, outstanding edits, retained history	Identifiers, causal context, metadata growth and safe garbage collection
Offline support	Possible, but requires explicit reconciliation/rebase	Often a strong fit, subject to implementation details
Backend required?	Yes for this architecture	Still needed for permissions, persistence, sync, and other platform services

Neither strategy automatically guarantees perfect semantic intent preservation. Neither removes the need for security, persistence, or reconnect logic. Nor does the word CRDT mean every arbitrary message order and metadata-compaction scheme is safe; the particular algorithm's assumptions matter.

Our design choice: use an authoritative, OT-style per-document owner for the main architecture. A CRDT-based editor is a valid alternative, especially if offline-first behavior is a primary product requirement. This is an architectural choice, not a claim about Google's internal synchronization algorithm.

8. One authoritative owner per document—not one server for the whole platform

We distribute documents across a fleet of document servers. A shard router resolves docId to the current owner. Each owner serializes authoritative edits for its documents and assigns monotonically increasing versions within each document. Unrelated documents do not need a shared global operation order.

Document A ──► Owner 1 ──► versions A: 1, 2, 3...
Document B ──► Owner 2 ──► versions B: 1, 2, 3...
Hot Doc C  ──► Dedicated owner capacity

This makes concurrent-edit reasoning local to one document while allowing the overall fleet to scale horizontally. The trade-off is that one very hot document can still overwhelm its owner, so we must distinguish many writers from many passive viewers. Those are different bottlenecks.

Owner failover and fencing

A server being designated as the owner is not sufficient to prevent split brain. During a network partition, an old owner might still be alive when a new owner is promoted.

We assign a monotonically increasing fencing epoch to each ownership term. The durable write path validates the epoch on writes and rejects stale owners. Promotion must use a coordination mechanism that prevents conflicting epochs from both being accepted as current.

If ownership is uncertain, briefly pause authoritative writes for the affected document rather than creating two valid-looking, divergent histories. Clients may continue typing into local pending queues while waiting to reconnect.

9. When is an edit actually saved?

This is one of the most important correctness boundaries in the design.

Consider a dangerous implementation:

Owner receives edit → applies in RAM → replies "saved" → crashes
                                          ↓
                              durable write never happened

The user saw saved, but the edit disappeared. To avoid that failure, define a strict contract:

Validate authorization, operation schema, version context, and duplicate identity.
Reconcile the operation with the current authoritative document state.
Assign the next document version and append the accepted operation to the durable log, using a write path that enforces the current fencing epoch.
Wait for the configured durability policy to be satisfied.
Only then acknowledge the edit as committed and publish it as an authoritative update.

A client may render the edit locally before step 4; that is optimistic UI, not durable acceptance. If the owner crashes after the durable append but before sending the acknowledgement, the client retries the same opId. Deduplication returns the original committed result rather than applying the edit twice.

The exact durability guarantee depends on storage and replication policy. A local disk append, a quorum-replicated commit, and synchronous cross-region replication provide different failure guarantees. Do not promise zero data loss during regional disasters unless the storage design actually supports it.

10. Operation log, snapshots, and version history

Persisting a full document body after every tiny edit is expensive. Replaying its entire lifetime of operations every time someone opens it is also expensive.

Use a hybrid approach:

Snapshot at version 1,000
      + committed operations 1,001 ... 1,080
      = current document at version 1,080

A snapshot contains a materialized document state at an exact authoritative version, with format/version metadata and integrity checks. The operation log records accepted changes. A bootstrap or recovering owner loads a verified snapshot and replays the verified log tail.

The materialized document view and the operation history may live in different stores and need not be updated in one distributed transaction. In this design, the durable operation log is authoritative; snapshot/materialized views are versioned projections that can be rebuilt idempotently. This is why recovery needs explicit version boundaries and reconciliation.

For user-visible version history, keep snapshots or revisions according to the product's retention requirements. Restoring an earlier revision should be a new authorized mutation at a new current version, not a silent rewind that invalidates connected clients' understanding of the timeline.

Why compaction is subtle

An old operation may still be needed for:

Transforming a client's stale OT operation.
Replaying an offline client's missing history.
Reconstructing a user-visible revision.
Meeting audit or retention requirements.

A snapshot does not automatically make all earlier operations disposable. Retention and compaction must be designed around these consumers, and long-offline clients need a defined fallback when the necessary history is no longer available.

11. Presence and cursors are not document truth

A collaborator's cursor position, selection, and online indicator should not have the same durability requirements as a text edit.

A separate presence service tracks active sessions, not merely user IDs. One person may have the same document open in multiple tabs; closing one tab should not make the other disappear from the presence list. Heartbeats and TTLs help expire abandoned sessions.

Cursor updates can be sampled, coalesced, or dropped when clients are slow. A failed presence service may temporarily hide collaborator cursors, while committed document editing continues. Conversely, an accepted document edit must not be silently discarded just because a gateway's outbound queue is full.

Cursor positions also need reconciliation with document changes. An offset meaningful at version 100 may no longer identify the same location at version 105. A practical editor transforms selection anchors or uses position representations compatible with its document model.

12. Offline editing and reconnection

A disconnected user should usually be able to continue typing. The client applies changes locally and stores pending operations with stable IDs and sequencing information.

On reconnect:

Authenticate and recheck current document permissions.
Report the last known committed version and outstanding operation IDs.
Determine which submitted operations were already committed but lost their acknowledgements.
Fetch missing authoritative operations, if the relevant history is retained.
Reconcile local pending operations against the updated state, respecting client-side ordering and the chosen OT/CRDT protocol.
Resubmit unresolved operations idempotently and resume live subscriptions.

If the client is too far behind for incremental OT transformation, use a snapshot-based rebase only if the implementation has a valid way to map the local edits onto that snapshot. A snapshot by itself cannot reconstruct the user's original intent. For difficult conflicts, preserve the unsynced local copy and ask for manual resolution rather than silently dropping text.

If access was revoked while the user was offline, queued edits must not be accepted on reconnect. The UI may preserve local unsynced work where appropriate, but the backend enforces the current permission state.

13. Authorization must work during a live session

Checking permission only when a WebSocket opens is insufficient. An editor may be removed from a document while the socket remains connected.

Enforce authorization at the authoritative mutation path and ensure that permission changes invalidate or update live subscriptions promptly. A revoked viewer must stop receiving protected document updates; a revoked editor must stop submitting accepted edits. Reconnect and history/restore endpoints need their own authorization checks too.

Also validate every incoming operation's schema, size, ranges, base-version plausibility, and allowed document mutation type. A buggy or malicious client must not be able to corrupt shared state by submitting invalid operations.

14. Capacity planning: connections, operations, and fan-out

A collaborative editor is not sized by document storage alone. Here is an illustrative, not measured, workload:

Assumption	Example
Concurrent actively edited documents	270,000
Average live editors per active document	3
Active editing connections	810,000
Average logical operations per active editor per second	0.8
Incoming edit operations per second	648,000
Additional peer deliveries at two peers per operation	~1.30 million/sec
Total open WebSocket connections including viewers	3 million
Heartbeat interval	20 seconds
Incoming heartbeat messages per second	150,000

The arithmetic is useful because it separates ingress edit throughput, outbound fan-out, and connection-management load. It is not a claim about Google Docs traffic.

A document with three editors is different from a document with three editors and 100,000 viewers. The latter may have little write traffic but enormous broadcast pressure.

To scale, distribute gateways horizontally, shard ownership by document, isolate hot documents, use gateway-local subscription fan-out, and keep per-connection buffers bounded. For large documents, consider incremental loading and efficient client rendering before attempting complex block-level authoritative sharding. Splitting a single document into independently writable blocks complicates cross-block edits, selections, and global operations.

15. Failure scenarios that decide whether the architecture is production-ready

Failure	Correct behavior
WebSocket gateway crashes	Reconnect to another gateway; retry unacknowledged ops using stable IDs
Owner crashes after durable commit but before ack	New owner recovers committed log; retry deduplicates the op
Owner crashes before durable commit	Do not claim the edit was saved; client keeps it pending
Two owners believe they are active	Durable writer rejects stale fencing epoch; affected clients resync
Broadcast misses version 105 but delivers 106	Detect version gap and fetch missing history or snapshot
Snapshot disagrees with log	Reject suspect snapshot; recover from last verified checkpoint and log
Presence service fails	Degrade cursors/online indicators without corrupting edits
Permission revoked mid-session	Stop writes and protected subscriptions promptly
Client reconnects months later	Detect missing transform history; use supported rebase/manual conflict path
Slow subscriber accumulates updates	Bound queue; drop ephemeral presence first; trigger authoritative resync
Reconnection storm after outage	Backoff with jitter, admission control, prioritized recovery
Sync algorithm deployment introduces a bug	Stop affected writes if necessary, roll back, validate history and repair
Region fails	Recover from replicated history with higher fenced epoch; honor actual RPO/RTO

A particularly subtle case is broadcast loss. Pub/sub is a delivery mechanism, not the authoritative history. Every committed remote operation should carry a document version so a client can detect gaps and recover from the authoritative log. Even if the transport is generally reliable, application-level version checks remain necessary.

16. Trade-offs worth explaining in an interview

OT or CRDT? OT integrates naturally with a centrally ordered per-document log; CRDT can suit offline-first replication. Both require careful implementation and testing.

Single owner or multi-writer? A fenced owner simplifies ordering and debugging. Multi-writer designs may offer different availability/latency properties but introduce harder coordination and merge semantics.

Optimistic typing or server-confirmed rendering? Apply locally immediately, but distinguish pending from committed. Waiting for server round trips before rendering every keystroke harms usability.

Snapshots or operations? Combine versioned snapshots with a durable operation log. Compaction must respect replay, restore, and offline requirements.

One home region or active writers everywhere? A home-region owner keeps ordering manageable, at the cost of higher network round-trip time for distant collaborators. Multi-region active writing is a separate, more complex consistency design.

Always accept writes or pause during uncertain failover? Prefer a brief per-document write pause over divergent authoritative histories. Local pending edits can preserve the typing experience during that pause.

17. How to answer “Design Google Docs” in a system design interview

A clear interview narrative is more valuable than starting with a giant infrastructure diagram:

Clarify scope: plain text or rich text? Typical collaborators? Offline support? History? Permissions?
State invariants: eventual convergence, no falsely acknowledged edits, one authoritative version sequence per document in this design.
Show the naive design: one buffer, autosave, and why whole-document writes lose concurrent changes.
Introduce WebSockets: persistent transport, gateways, routing, and live fan-out.
Explain concurrency: an actual conflicting-edit example, followed by OT or CRDT and its limitations.
Make durability explicit: operation IDs, commit acknowledgements, durable history, snapshots, and replay.
Add failure handling: owner fencing, reconnect, authorization changes, missing broadcasts.
Scale the system: shard by document, separate presence, isolate hot documents, and manage large fan-out.

The high-signal follow-up is often not “What is OT?” but “What happens when the owner crashes after sending a saved acknowledgement?” Your answer should point to the durability boundary, not merely say that the client reconnects.

Frequently asked questions

How does a Google Docs-style editor prevent users from overwriting each other's changes?

It sends structured edits rather than blindly replacing the entire document. An OT or CRDT synchronization protocol reconciles concurrent operations so clients can converge on a shared state. The protocol still needs correct implementation, ordering or merge semantics, and durable storage.

Does Google Docs use Operational Transform or CRDT?

This article does not assert which algorithm Google's current internal implementation uses. It explains two established approaches for building a Google Docs-style editor and chooses a centralized OT-style architecture as a teaching example.

Why are WebSockets not enough for collaborative editing?

WebSockets move messages bidirectionally, but do not solve concurrent edits, authoritative ordering, duplicate submissions, persistence, permission revocation, or recovery from missed updates.

How are edits recovered after a disconnect?

Clients keep stable operation IDs and pending local edits. On reconnect they reauthenticate, fetch missing authoritative changes or a supported snapshot/rebase path, reconcile outstanding work, and retry idempotently. Severe conflicts may require user intervention.

How do you scale a document with thousands of viewers?

Keep one correctly fenced authoritative edit stream in the baseline design, but distribute already-committed updates through scalable fan-out infrastructure. Coalesce presence and isolate hot documents; do not confuse passive-viewer traffic with write-serialization load.

Final architecture: what really matters

A reliable collaborative editor combines instant local editing with authoritative, durable synchronization. The core path is:

Client → WebSocket Gateway → Shard Router → Document Owner
       → Validate / Deduplicate / Reconcile
       → Durable Operation Log → Commit Ack
       → Broadcast → Other Clients

Around it sit fenced ownership, versioned snapshots, offline replay, authorization, presence, and observability. The difficult engineering is not choosing a WebSocket library. It is making sure users never receive a false saved signal, stale owners cannot create conflicting histories, and disconnected replicas have a safe path back to a converged state.

Explore the full visual system design on SeeItFlow

This article is a condensed architectural walkthrough. For the progressive diagrams, detailed trade-offs, production failure walkthroughs, and interview questions, see the complete six-part module:

SeeItFlow is a free visual learning platform for software engineering and system design.

Recommendation System Design: Candidate Generation, Ranking & Feature Stores

MANGESH MANDLIK — Tue, 27 Oct 2026 13:54:51 +0000

Part of the SeeItFlow series: System Design — Beyond the Architecture Diagram.

How does a recommendation system select 20 relevant videos, products, or posts from a catalog containing millions of items—without scoring the entire catalog on every request?

The answer is not simply “use machine learning.” A production recommendation system is a low-latency retrieval and ranking platform supported by event ingestion, user profiles, feature pipelines, caches, experimentation, and safeguards against stale or ineligible results.

This article develops that architecture from a simple trending feed to a personalized, production-ready design. The examples and capacities are illustrative engineering assumptions, not claims about the internal systems of YouTube, Netflix, Amazon, or any other company.

For interactive diagrams and the full six-part walkthrough, see Recommendation System Design on SeeItFlow.

1. Requirements: What are we actually building?

Suppose we are designing recommendations for a video or commerce platform. Given a user, a recommendation surface (home feed, related items, next-watch), and request context, return a bounded, ordered list of relevant, eligible items.

The system should:

Personalize results when meaningful behavioral history exists.
Work for anonymous users, new users, and new catalog items.
React to recent user actions without rebuilding the entire long-term profile synchronously.
Avoid showing deleted, restricted, unavailable, or otherwise ineligible items.
Log which items were returned and which were actually exposed, not just which were clicked.
Degrade to safe, useful recommendations when personalization services fail.

Important nonfunctional requirements include latency, availability, freshness, relevance, cost, privacy, and the ability to explain quality regressions even when HTTP error rates are zero.

A recommendation system is surface-specific. Home-feed recommendations, “similar products,” and next-video autoplay may use different signals, objectives, freshness budgets, and eligibility rules. Do not assume one universal ranking policy is appropriate for every surface.

2. Estimate the workload before choosing components

Consider an example with 60 million daily active users, each requesting recommendations 25 times per day:

Daily recommendation requests = 60M × 25 = 1.5B
Average QPS = 1.5B / 86,400 ≈ 17,361
Illustrative 10× peak ≈ 174,000 QPS

If each uncached request scores 400 candidates, the theoretical peak demand is approximately 69.6 million candidate scores per second across the fleet. Caching, cheaper pre-ranking, and partial fallbacks can reduce actual main-model scoring volume; the multiplication is a sizing upper-bound scenario, not an assertion that all requests invoke the full ranker.

Now suppose the application ingests 120 behavioral events per active user per day:

Behavior events/day = 60M × 120 = 7.2B
Average events/sec ≈ 83,333

If those 120 events exclude rendered impressions, the impression stream must be sized separately. For example, a feed returning 20 items does not mean all 20 were visible to the user. Returned items and actually rendered/exposed items must not be counted interchangeably.

These numbers immediately explain the architecture: retrieve hundreds, not millions; batch feature reads; avoid unnecessary model execution; and ingest feedback asynchronously.

3. High-level architecture

                       CLIENTS (web / mobile / TV)
                                 |
                       Recommendation API
                                 |
                  +--------------+--------------+
                  |                             |
             Response Cache                User / Session
                  |                         Context Lookup
                  |                             |
                  +--------------+--------------+
                                 |
                         Candidate Retrieval
                   /             |              \
             Popularity     Collaborative      Embedding/ANN
                   \             |              /
                    +------ Merge & Dedup -----+
                                 |
                       Early Eligibility Filter
                                 |
                       Batched Feature Hydration
                                 |
                      Pre-ranker (if needed)
                                 |
                           Main Ranker
                                 |
                    Re-ranking / Diversity Rules
                                 |
                    Final Eligibility Recheck
                                 |
                      Response + Continuation

    Events / impressions -> Durable log -> Stream + batch jobs
                                         |          |
                                  Session overlay  Profiles / features
                                         |          |
                                  Online serving <- Published indexes/models

The diagram separates two kinds of work. Offline and streaming pipelines prepare reusable data. The online serving path combines that data with the current request and produces the final result within a deadline.

4. Start simple: trending recommendations

A first version can return regional or category-level popular items from a precomputed table. This is cheap to cache, easy to operate, and useful when the system knows nothing about a user.

But a trending-only feed has clear limits: it cannot distinguish a sports fan from someone interested in databases, and it may repeatedly expose the same popular items while hiding the long tail.

Treat popularity as a candidate source and fallback, not as a substitute for personalization.

5. Event ingestion: personalization begins with trustworthy signals

Clients and backend services emit events such as impressions, clicks, watch duration, skips, saves, hides, and purchases. A typical event envelope contains:

{
  "eventId": "evt_9012",
  "userId": "user_42",
  "itemId": "video_81",
  "eventType": "watch_complete",
  "surface": "home_feed",
  "requestId": "rec_730",
  "eventTime": "2026-10-27T10:15:00Z",
  "schemaVersion": 2
}

The collector authenticates where applicable, validates the schema, applies abuse controls, and durably records accepted events before acknowledging their ingestion. Downstream consumers can then process events asynchronously.

At-least-once delivery and duplicate events

Network retries, broker redelivery, and operational replay can deliver the same logical event multiple times. Assign a stable event ID and make consumers idempotent. Deduplication must cover the required replay horizon, or aggregates must be rebuildable from deduplicated durable history. A short-lived recent-ID cache alone cannot guarantee correctness across an arbitrarily old replay.

Do not claim exactly-once business effects simply because the message broker offers exactly-once-related features. The end-to-end effect still depends on consumer state updates, transactional boundaries, and external stores.

Event time is not processing time

A mobile device can report a watch event minutes late. Time-window aggregations should distinguish when the action occurred from when it arrived, with watermarks or explicit late-event policies and a defined correction strategy. There is no free global event ordering across users or devices.

Returned is not rendered

A recommendation API may return 20 items, but the user might only scroll far enough to see four. Maintain distinct signals for:

Generated: retrieved by a candidate source.
Returned: included in the API response.
Rendered/exposed: actually shown under the product's defined visibility criteria.
Consumed/acted upon: clicked, watched, purchased, or otherwise engaged with.

A server response cannot prove an impression occurred on screen. Client impression reports may themselves be lost or duplicated, so track reporting quality and avoid treating missing client telemetry as proof that no exposure happened.

Without exposure data, “no click” cannot be interpreted reliably: the item might never have been visible. This is a core source of selection bias in recommendation training and evaluation.

6. Build compact user profiles, not unbounded histories

Raw event logs are unsuitable for synchronous scanning on every recommendation request. Instead, maintain a compact user profile containing long-term category affinities, bounded recent activity, and preference summaries.

Separate it from a short-lived session overlay. If a user suddenly watches several cooking videos, a real-time processor can update session intent for the next request without immediately rewriting every long-term preference feature.

A useful design is:

Durable behavior log
   |--> Batch / stream aggregates --> Long-term user profile
   |--> Fast signal processor -----> Session overlay (short TTL)

Request-time context + profile + overlay --> Retrieval and ranking

Session signals should have bounded influence and expiry. One accidental click should not permanently redefine a user's interests. Also distinguish ordinary relevance preferences from privacy, consent, policy, and eligibility state: the latter cannot safely depend solely on an old profile snapshot.

7. Candidate generation: millions of items become hundreds

The candidate stage is a retrieval problem. Its job is to find a manageable set of plausible items quickly, not to make the final ranking decision.

Typical sources include:

Popularity and trending. Regional or category-level items with recent engagement. Strong for cold start and fallbacks, but biased toward widely exposed items.

Collaborative retrieval. Items associated with similar interaction patterns among users or sessions. Useful when behavioral history is rich; weaker for new users and items.

Content-based retrieval. Items similar in topic, attributes, text, creator, or other metadata. Useful for item cold start when behavioral history is missing.

Embedding and approximate nearest-neighbor (ANN) retrieval. Encode users, sessions, or queries and items into vector representations; retrieve nearby item vectors using a versioned index. ANN trades exact-neighbor recall for faster retrieval at scale. It is one retrieval strategy, not the entire recommender.

Sources can run concurrently with independent deadlines and quotas. Each source should return item IDs, source provenance, index version, and optionally a source-local score. Raw scores from different retrieval models are not inherently comparable. Merge by deduplicating IDs, applying source quotas or calibrated fusion logic, and preserving provenance for downstream features.

Example:

Popularity source:       120 candidates
Collaborative source:    200 candidates
Embedding/ANN source:    250 candidates
Content source:          100 candidates
                         ----------------
Before deduplication:    670 entries
After merge and dedupe:  bounded candidate set

The exact counts are examples. They are not guaranteed to sum to the final unique count because sources overlap and filters remove items.

Failure rule: one slow retrieval source should not hold the entire feed hostage. Enforce timeouts and continue with healthy sources or an approved fallback.

8. Eligibility filtering must happen before and after ranking

An item can be relevant and still be forbidden to show. Examples include deleted content, age-restricted media, blocked creators, region restrictions, inventory unavailability, or items the user is not entitled to access.

Perform an early filter to avoid wasting ranking work on obviously ineligible items. Then perform a final eligibility check before returning results, including cached recommendations.

Why twice? Candidate indexes and cached lists may lag behind catalog updates. A product can go out of stock after candidate generation; a video can be removed while an older index still references it.

The final check must consult a source with freshness and consistency guarantees appropriate to the policy. If that source is unavailable, fail closed for sensitive restrictions rather than silently treating an unknown state as allowed. Where strong real-time guarantees are required, the serving system needs an authoritative or appropriately synchronized eligibility path; merely lowering a cache TTL is not sufficient.

A final recheck narrows the stale-data risk but does not magically make all concurrent changes impossible. Define the required consistency boundary and ensure the action endpoint (such as purchase or playback) enforces its own authorization and availability rules.

9. Feature store: the bridge between offline learning and online serving

Ranking needs more than user IDs and item IDs. Example features include recent category affinity, item popularity in a time window, item age, user-item similarity, device context, and recent negative feedback.

A feature platform usually spans three related concerns:

Definitions and metadata: names, types, transformations, ownership, freshness, and schema versions.
Offline feature history: reproducible features for training, backfills, and evaluation.
Online feature serving: low-latency access to the features needed for inference.

A feature store is not necessarily one physical database, and centralizing definitions does not automatically eliminate training-serving skew.

Point-in-time correctness

Suppose a model learns from a click that occurred on Monday. Its training example must not include an item-popularity count that incorporates Tuesday's clicks. That would leak future information into training and inflate offline evaluation results.

Point-in-time joins must select feature values that were available at the prediction time, accounting for both event time and feature publication/availability time where relevant. Version and timestamp the data used for reproducibility.

Batch feature hydration

Fetching features one candidate at a time creates excessive network fan-out. Batch or multi-get the feature bundles needed for the bounded candidate set, use well-defined defaults for missing features, and check that feature versions are compatible with the selected model.

10. Ranking: why a retrieval score is not the final answer

Retrieval identifies plausible items; ranking estimates their usefulness for a particular user, surface, and moment.

A basic ranking pipeline:

400 retrieved candidates
        |
  Batch feature fetch
        |
  Optional cheap pre-ranker
        |
  Expensive main ranker
        |
  Diversity / repetition controls
        |
  Final eligibility check
        |
      Top 20

At smaller scale, one ranker may be sufficient. Introduce a cheaper pre-ranker when the main model's cost or latency makes scoring all retrieved candidates impractical. The trade-off is that the pre-ranker may discard items the richer model would have liked.

The ranking objective may combine engagement, satisfaction, conversion, long-term retention, or other product outcomes, subject to policy constraints. These objectives can conflict. Optimizing only short-term clicks may reward sensational or repetitive results while reducing long-term satisfaction.

A model score is not automatically a calibrated probability. Depending on the training objective, it may be a relative ordering signal, an uncalibrated logit, or a calibrated estimate of a defined outcome. Do not compare scores across model versions as if they share a universal scale unless calibration and comparability are established.

Post-ranking and diversity

After model scoring, re-ranking can cap repeated creators, diversify categories, suppress recently consumed items, and reserve a controlled number of exploration slots. Such rules should be measured: more diversity can improve discovery but may reduce immediate predicted engagement.

11. Cold start: new users and new items are different problems

New user: no meaningful personal history. Use region-appropriate popularity, contextual information, optional onboarding preferences, and exploration. Personalization can improve as reliable events accumulate.

New item: little or no interaction history. Use metadata/content-based retrieval and carefully controlled exploration to create opportunities for exposure. An item may also receive an embedding derived from content before any interactions occur; therefore, it is incorrect to assume that every new item must wait for behavioral history or a scheduled embedding rebuild.

Track new-item exposure and catalog coverage. Otherwise, the feedback loop can permanently favor already-popular items because those are the only ones receiving training data.

12. Real-time recommendations without recomputing everything

A session overlay lets the system react quickly to recent actions. For example, after a user skips three similar videos, the next feed request can reduce that category's short-term weight.

This is usually near-real-time, not a hard guarantee that every action is reflected instantly. Processing lag, network delay, caching, and concurrent requests can all affect visibility.

Use targeted cache invalidation on strong signals, short-lived overlay state, and freshness SLOs appropriate to each surface. Keep slower long-term aggregation in batch or streaming pipelines based on its actual business need—not because all features must update on every event.

13. Caching and pagination: fast results must still be correct

Cache regional trending lists, hot item metadata, frequently requested features, and, where useful, short-lived personalized recommendation results.

Personalized cache keys must be scoped to the right user/session, surface, locale, relevant context, policy version, and model/index version. Omitting a user-specific dimension can cause a cross-user cache leak, which is a privacy issue, not just a relevance issue.

Use request coalescing and soft TTL refresh to prevent many simultaneous misses from overwhelming retrieval and ranking services. Never let a cache hit bypass the final eligibility check.

Infinite scroll needs session consistency

Page 1 and page 2 may be generated seconds apart, during which item popularity, user activity, or model versions can change. Naively rerunning retrieval can show duplicate items or skip previously ranked ones.

A continuation token can bind requests to a short-lived recommendation snapshot, candidate-set reference, or seen-item state. The token should be tamper-resistant and expire appropriately. It does not have to contain the entire candidate list; it can reference server-side state.

14. Model and index rollouts: silent failures are the real risk

Recommendation failures often return HTTP 200. A new model can be fully available and still show worse content.

For a model rollout, use offline validation, compatibility checks, shadow evaluation where useful, a controlled canary or A/B experiment, progressive expansion, and rollback thresholds based on both system health and recommendation quality.

For an index rollout, build a new version alongside the old one, validate its integrity and coverage, then publish it through a versioned pointer or explicitly controlled gradual rollout. Retain the prior good version for rollback. An atomic pointer update in one metadata store does not imply every serving process worldwide instantly switches at the same moment; use version pinning and rollout visibility to manage mixed versions safely.

A/B assignment should be sticky and based on stable identifiers where appropriate, and the assigned variant must be propagated into exposure and outcome logs. Guard against sample-ratio mismatches, missing telemetry, and nonrandom logging loss before trusting experiment conclusions.

15. Failure scenarios worth designing for

Failure	User-visible or silent impact	Safe response
Ranking service overloaded	Slow feed, timeouts	Reduce candidates, use cheaper model, serve safe cached fallback
One candidate source times out	Lower retrieval coverage	Respect deadline; merge healthy sources
Feature store unavailable or incompatible	Generic or incorrectly scored items	Use known-compatible defaults/snapshots; never bypass hard policy checks
Event ingestion lag	Recent actions do not influence feed	Track event age; use bounded session state when available
Bad model or index rollout	Worse relevance with HTTP 200	Detect via quality metrics; roll back version
Deleted or restricted item in stale cache	Correctness or policy breach	Mandatory current-state eligibility check; fail closed for sensitive rules
Duplicate events	Inflated popularity/profile signals	Stable IDs, idempotent updates, replay-safe reconstruction
Trending item creates hot key	Counter and cache pressure	Pre-aggregate, shard hot counters, coalesce reads
One source dominates retrieval	Narrow, repetitive feed	Monitor source share; quotas and diversity controls
Cross-user cache-key collision	Privacy leak	Strict key scoping, tests, purge and incident response

Quality monitoring must go beyond p95 latency and error rate. Track candidate counts, per-source contribution, feature freshness, fallback rate, rendered impressions, exposure concentration, new-item coverage, repeat-item rate, and experiment outcomes. A system that is fast and always returns 200 can still be failing its users.

For more detailed incident walk-throughs, see Recommendation System Failure & Scale Scenarios.

16. Key architectural trade-offs

Popularity vs. personalization: popularity is cheap and resilient; personalization improves relevance but needs behavioral state and more online work. Use a deliberate blend.

Precomputation vs. freshness: compute reusable embeddings, features, and indexes offline; reserve request-time work for context-dependent decisions. Use streaming only where freshness earns its cost.

Single-source vs. multi-source retrieval: one source is simple; several sources improve coverage and resilience but introduce merge complexity and extra fan-out.

Exact search vs. ANN: exact vector search can be appropriate for small catalogs or offline evaluation; approximate retrieval becomes attractive when exact search exceeds the online budget. Benchmark recall, latency, memory, and update behavior for the chosen workload.

Single-stage vs. multi-stage ranking: a pre-ranker reduces expensive model load but may eliminate valuable candidates too early.

Relevance vs. exploration: serving only the current highest-scoring items reinforces exposure bias; exploration creates information about underexposed content, at a measurable short-term cost.

Eventual freshness vs. hard correctness: most relevance signals tolerate bounded staleness; policy, authorization, deletion, and inventory-sensitive decisions need stronger checks appropriate to their semantics.

For the reasoning behind these choices, read Recommendation System Trade-offs & Decisions.

17. How to explain this in a system design interview

Start with the surface and the objective. A “recommendation system” is not one fixed product requirement.

Then evolve the architecture in this order:

Trending baseline: deliver a useful list without personalization.
Event collection: record durable actions and exposure context.
Profiles: summarize history into fast-serving state.
Candidate generation: narrow the catalog using multiple bounded retrieval sources.
Ranking: score only a manageable candidate set.
Feature store: make online inference and offline training consistent and reproducible.
Real-time overlay: respond to current-session intent.
Production safeguards: enforce eligibility, caching discipline, versioned rollouts, experimentation, and degraded modes.

Interviewers often ask:

Why not rank every item? Because the cost of expensive per-item inference across a huge catalog exceeds the latency budget. Retrieval narrows the search space first.

Why log impressions rather than just clicks? Because an unclicked item may never have been exposed. Without exposure data, negative labels and offline evaluation can be biased.

What happens when the feature store is down? Fall back to known-compatible features or a cheaper model where safe; independently enforce policy and eligibility using an appropriate current-state source.

How do you handle new items? Metadata/content retrieval, content-derived embeddings where available, and controlled exploration rather than relying solely on historical engagement.

How do you detect a bad ranking model? Monitor controlled experiments and quality metrics, not only API errors and latency; retain a fast rollback path.

See the Recommendation System Interview Perspective for a larger question bank.

18. Final takeaway

The core of recommendation system design is a sequence of bounded decisions:

Collect reliable signals → build compact state → retrieve plausible candidates → hydrate compatible features → rank and diversify → recheck eligibility → log actual exposure → learn and improve.

Machine-learning models matter, but so do event semantics, feature freshness, latency budgets, cache isolation, model compatibility, exposure bias, and safe degradation. Neglecting any of those can produce recommendations that look technically healthy while quietly becoming irrelevant—or incorrect.

Explore the complete visual walkthrough on SeeItFlow

SeeItFlow is free to use, with interactive visual explanations and no sign-in required.

YouTube System Design: Upload, Transcoding, CDN & Video Streaming

MANGESH MANDLIK — Mon, 19 Oct 2026 07:04:45 +0000

YouTube System Design: From Video Upload to Global Playback

A viewer presses Play. A few seconds later, video is streaming smoothly, even on a changing mobile network. That looks simple until you consider what happened before the first frame: a creator uploaded a large file over an unreliable connection; the platform validated it, transcoded it into compatible formats, published a consistent set of segments, and distributed those bytes across a global caching network.

Designing a YouTube-like video-sharing platform is really two different system-design problems: safely turning uploads into playable assets, and delivering those assets to enormous numbers of viewers at low latency and reasonable cost.

This article walks through both, including the failure cases that often get omitted from architecture diagrams. It describes a proposed architecture, not YouTube's proprietary internal implementation.

For an interactive explanation with diagrams and step-by-step flows, see Video Streaming Platform System Design on SeeItFlow.

1. Scope and requirements

We'll design video-on-demand (VOD): creators upload videos, and viewers discover and play them later. Live streaming, DRM internals, recommendation algorithms, ads, and moderation internals are separate systems and are outside this design.

The functional requirements are:

Upload large videos, with pause/resume and per-part retries.
Process originals into playable representations; generate thumbnails and metadata.
Publish only validated, complete media outputs.
Stream at adaptive quality, with seeking and playback recovery.
Support public and restricted visibility, search, metadata edits, and deletion.
Record view events and show an eventually consistent display count.

The non-functional requirements drive the architecture: durable finalized uploads, safe retries, bounded processing delays, fast startup, low rebuffering, global delivery, origin protection, and explicit privacy/deletion semantics.

A useful separation is ingest/processing, control/metadata, and delivery/playback. A video being uploaded, processed, discoverable, and playable are distinct states; one flag cannot safely represent all four.

2. Estimate the workload before choosing infrastructure

In a video platform, playback demand will often dwarf upload count. Rather than inventing real-world YouTube statistics, use clearly stated assumptions.

Suppose, purely as a capacity-planning example, we have:

100,000 uploads/day, averaging 500 MB each.
1 million simultaneous viewers at peak.
An average delivered bitrate of 3 Mbps per active viewer.

Then:

Upload ingress: 100,000 × 500 MB ≈ 50 TB/day of original uploads, before replication and derived media.

Peak playback bandwidth: 1,000,000 × 3 Mbps ≈ 3 Tbps delivered to viewers. That is an aggregate delivery-network figure, not a claim that a single origin must serve it.

Processing work: depends on media duration, resolution, codec, and how many representations are encoded. A two-hour 4K upload and a 20-second clip should not count as equal-sized processing jobs.

Storage: originals + transcoded representations + audio + thumbnails + retention/replication overhead. Segmentation mostly divides a rendition into smaller objects; additional renditions and codecs are what significantly multiply encoded-media storage.

These illustrative numbers explain why the upload service, processing workers, and playback delivery tier need to scale independently.

3. High-level architecture

CREATOR
   |
   | 1. Create upload session
   v
Upload API / Metadata DB
   |  short-lived, scoped upload credentials
   v
Object Storage (original media) <---- Creator uploads parts directly
   |
   | 2. Verified, idempotent finalize
   v
Metadata transaction + durable processing job / outbox
   |
   v
Dispatcher -> Processing Queue -> Transcoding Workers
                                     |
                                     v
                              Probe / Encode / Package
                                     |
                                     v
                           Validate versioned outputs
                                     |
                                     v
                             Atomic publish pointer
                                     |
                                     v
                              Object Storage (origin)
                                     |
                                     v
VIEWER -> Playback API / Authorization -> CDN Edge -> Shield -> Origin
                 |                        |
                 v                        v
             Metadata DB              HLS/DASH segments
                 |
                 +--> Search indexing / async view aggregation

This is a logical diagram. Actual implementations may combine services or use different managed products. The important properties are the durability boundaries and which dependencies sit on the critical playback path.

4. Resumable uploads: don't make the app server carry every byte

Sending a multi-gigabyte file through the application tier consumes network bandwidth and connections for work that object storage can handle more efficiently.

A safer upload flow is:

The creator authenticates and requests an upload session. The server creates a stable upload_id, records ownership, size limits, and expiry.
The server issues short-lived, tightly scoped upload permissions for the intended object/session.
The client uploads independently identifiable parts directly to object storage, retrying failed parts without restarting the entire file.
The client asks to finalize. The backend checks authoritative storage-side part/completion state and applicable integrity constraints. It does not trust a client-side “all chunks uploaded” assertion.
Once the object is finalized and metadata/job state is committed, the client receives a stable video identity and a processing status—not a promise that playback is ready.

Finalization must be idempotent. If the server commits and its response is lost, retrying the same upload_id must return the existing result rather than create a second video or an unrelated processing workflow.

There is an important ordering nuance: object-storage multipart completion and a metadata database transaction are not automatically one atomic operation. The implementation needs a recoverable state machine and a reconciliation sweep for cases where storage finalization succeeds but metadata completion fails, or vice versa. A durable source object must not be silently orphaned or incorrectly treated as ready.

Expired sessions and abandoned multipart uploads need automated cleanup. Upload credentials must never grant broad bucket access.

5. Durable processing: the database-to-queue gap

Video processing is CPU/GPU intensive and highly variable in duration. It belongs outside the upload request.

A tempting implementation is:

UPDATE videos SET state = 'PROCESSING';
publish_to_queue(video_id);

But these are two separate systems. The database commit can succeed while publishing to the queue fails, leaving a creator waiting forever.

A transactional outbox or equivalent durable job record closes that gap:

One database transaction:
  - Mark the upload finalized / processing pending
  - Insert processing_job(job_id, source_version, state='PENDING')
Commit

Separate dispatcher:
  - Read undispatched jobs
  - Publish to processing queue
  - Record dispatch progress after broker acknowledgement

A crash after queue publish but before recording success may cause duplicate delivery. That's acceptable only because processing is designed to be idempotent at its publication boundary. An outbox gives durable at-least-once handoff, not exactly-once execution.

Each job should identify its source version, processing profile, and attempt. Retry transient failures with backoff and limits; quarantine deterministic failures such as unsupported codecs or irreparably corrupt sources. Monitor oldest job age, queued media minutes, and processing SLO attainment, not just raw queue depth.

6. Transcoding: a rendition is more than a resolution

A creator might upload a high-bitrate file in a format that some viewers cannot decode or stream efficiently. The platform produces multiple compatible representations.

A representation can vary in resolution, bitrate, codec, frame rate, audio format, and HDR profile. An illustrative ladder might include 1080p, 720p, 480p, and 360p, but there is no universal ladder. Choose representations based on source characteristics, device compatibility, network diversity, encoding cost, and expected demand. Avoid upscaling merely to populate a preset rung.

A practical processing workflow is a dependency graph:

Probe and validate source
        |
        +--> Video encode: representation A --+
        +--> Video encode: representation B --+--> Package/segment
        +--> Audio encode --------------------+         |
        +--> Thumbnail extraction                      v
                                                   Validate
                                                      |
                                                   Publish

Independent representations can process concurrently. Track progress and failure per stage, so a failed optional rendition can be retried without recomputing everything. Whether a reduced ladder is publishable depends on an explicit minimum-playable-set policy; it is not always all-or-nothing.

Encoding is expensive. Precomputing every possible quality improves first-play readiness but wastes work on rarely viewed videos. Selective or on-demand encoding can reduce cost, at the price of more complex coordination and potentially higher first-view latency.

7. The most important correctness boundary: safe publication

Imagine a transcoding worker crashes after writing half the segments. Its job is redelivered while the original worker is still running. If both attempts overwrite the same public object keys, a viewer may see a manifest referencing incomplete or mixed output.

The solution is attempt-scoped staging and immutable published versions:

Each processing attempt writes into a private, unique staging namespace.
The system verifies required representations, segments, manifest references, and basic media integrity.
All objects referenced by the candidate manifest must already be durably readable before publication.
A conditional/transactional metadata update changes the video's active published version only if the candidate is valid and the publishing worker still has authority.
Older published versions remain intact for in-flight readers and rollback until a safe retention/cleanup policy removes them.

The atomic operation is the metadata pointer change, not an imaginary transaction spanning thousands of object-storage writes. Duplicate or late workers may waste compute, but they cannot publish over the active version unless they pass the required fencing/conditional checks.

Versioned URLs are also CDN-friendly: a segment at video/v7/720p/segment-42.m4s never changes its bytes. Reprocessing creates v8 instead of silently mutating v7.

8. HLS, MPEG-DASH, and adaptive bitrate streaming

The platform packages media into short, time-aligned segments and manifests/playlists that tell the player which representations and segments exist.

Two common adaptive streaming families are HLS and MPEG-DASH. Their exact manifest structures differ: for example, HLS commonly uses a multivariant playlist and media playlists, whereas DASH uses an MPD describing adaptation sets/representations and segment addressing. They should not be described as byte-for-byte identical formats.

For a typical segmented VOD session:

The player gets authorized playback information and a manifest URL.
It selects a supported representation using device capability, estimated throughput, and current buffer health.
It requests media segments over HTTP and buffers ahead.
When throughput drops, it requests subsequent segments at a lower sustainable bitrate; when conditions improve, it may step up cautiously.
On seeking, it locates the segment covering the requested playback time rather than downloading everything from the beginning.

Rendition switching works best when segments and suitable random-access/keyframe boundaries are aligned. A startup algorithm does not always begin at the lowest resolution. Nor should it blindly chase the highest bitrate: rebuffering usually hurts the viewing experience more than a temporary reduction in picture quality.

Segment duration is a trade-off: shorter segments allow finer-grained adaptation but increase request and packaging overhead. It is a design choice, not a fixed number that every streaming service must use.

9. CDN design: hot videos are a caching problem—with exceptions

A million viewers cannot all fetch every segment from one storage origin without enormous egress and request pressure. A CDN places popular segments closer to viewers.

Viewer -> CDN edge (hit: serve locally)
                |
                +-- miss -> Regional/shield cache
                                  |
                                  +-- miss -> Object-storage origin

Immutable segments can use long-lived, versioned cache keys. Manifests and authorization-related responses may require different caching policies.

An origin shield reduces the number of separate edge misses that reach origin. Request coalescing helps prevent multiple simultaneous misses for the same object from triggering duplicate upstream fetches. Neither makes origin load literally zero: cold content, cache-key fragmentation, regional traffic skew, and purge events still matter.

One subtle bug can destroy the cache hit ratio: including a viewer-specific signed token in the cache key when the underlying segment bytes are identical. Normalize the cache key only if authorization is still independently enforced before cached bytes are served. Removing tokens from the key without enforcing access at the CDN or trusted delivery layer would create a security vulnerability.

A broad CDN outage also needs a bounded fallback strategy. Sending unlimited global traffic directly to origin can turn a CDN incident into an origin outage. Consider alternate healthy edges or providers where configured, controlled origin fallback, and load shedding.

10. Metadata, visibility, and playback authorization

A video record might track:

video_id, creator_id, title, description, duration,
source_object, processing_state, active_media_version,
visibility, moderation_state, created_at, updated_at

Keep processing readiness, discoverability, and authorization separate. A video can have a valid encoded output but still be private, blocked, or intentionally unpublished.

For restricted content, the playback API verifies entitlement and issues a short-lived credential that the delivery path actually checks. A metadata flag alone does not protect segments if anyone with a cached URL can fetch them.

Unlisted is not the same as private. In a common product model, unlisted means omitted from normal discovery but accessible to people with the link; private means access is restricted to authorized viewers. The exact policy should be stated, not assumed.

A public-to-private change requires more than updating one database row:

Stop issuing new access credentials and make the authoritative authorization decision reflect the restriction.
Invalidate or bypass stale authorization caches, including remote-region replicas where necessary.
Remove the video from search/discovery asynchronously, with priority over ordinary metadata edits.
Bound or revoke previously issued delivery credentials and purge cached media when the policy requires it.

Short-lived signed URLs do not provide instantaneous revocation by themselves. Already-issued credentials may remain valid until expiry unless the delivery architecture supports a revocation check or equivalent enforcement. CDN purge propagation is also not necessarily instantaneous. A correct design states the permitted exposure window and chooses enforcement accordingly.

11. Search and view counts are separate workloads

Metadata lives in a transactional store; video binaries live in object storage. As catalog size and ranking needs grow, a dedicated search index supports text matching and ranking without forcing search traffic through the metadata database.

Index updates are usually asynchronous. Search may briefly show stale results, but playback authorization remains the final access gate. A search outage should not break playback for a viewer who already has a valid video link.

View counting has its own scale trap. Incrementing one database row synchronously for every view creates a hot write key for viral content. Instead:

Player event -> Event ingestion -> Stream/queue
                                  |
                                  v
                         Dedupe + aggregation
                                  |
                                  v
                       Periodic counter updates

First define what counts as a view; there is no universal watch-time threshold. Events need stable identities for bounded deduplication, and a viral video may require sharded or hierarchical aggregation to avoid a single hot reducer.

The user-visible count can be approximate and eventually consistent if the product permits it. Bounded dedupe does not guarantee an exact lifetime count, and a dropped event may never be recovered. Financially consequential metrics—such as ad billing or creator payouts—require a separate, stricter measurement and audit design.

12. Multi-region delivery is not the same as multi-region ownership

A CDN can serve cached segments worldwide, but it does not automatically make upload processing, metadata writes, or source objects resilient to a regional failure.

At a large enough scale, regional ingest and origin infrastructure can improve locality and availability. That adds harder questions:

Which region owns writes for a given video or metadata partition?
Where are originals and derived objects replicated?
Can a region serving a cache miss actually reach a copy of that segment?
How do we prevent two regions from accepting conflicting authoritative writes during a partition?

One workable model uses a single authoritative write owner per video/partition, asynchronous read replicas, selective object replication, and fencing before write ownership moves. An in-progress multipart upload tied to a failed region may have to restart; seamless continuation requires explicit design, not a routing rule.

Selective replication saves storage and transfer cost but means some content may become unavailable when its only reachable origin fails. The availability promise must match the actual replication policy.

13. Failure scenarios that test whether the design is real

Failure	What can go wrong	Safe response
Upload finalize response lost	Client retries and creates duplicate work	Stable `upload_id`, idempotent finalize, recoverable state
Metadata commit succeeds, queue publish fails	Video remains processing forever	Durable outbox/job record and retrying dispatcher
Worker lease expires while worker still runs	Two attempts race to publish	Attempt-specific staging and fenced, conditional publication
One rendition fails	Partial ladder exists	Retry failed stage or apply an explicit minimum-rendition policy
Manifest published before segments	Early viewers get 404s	Validate object availability before switching active version
Viral video's first few requests	Cold-edge misses overwhelm origin	Coalescing, origin shield, rate limiting
CDN cache-key fragmentation	Cache hit ratio collapses	Normalize safe keys; enforce authorization independently
Network bandwidth collapses	Buffer drains, playback stalls	ABR downshift, buffer-aware retries
Privacy change meets stale cache	Restricted video remains accessible	Authoritative access checks, revocation strategy, cache invalidation
Region fails	Some origins or metadata owners disappear	Replication-aware routing and fenced failover
Telemetry pipeline fails	View count stops updating	Keep playback independent; buffer or accept bounded event loss
Video deletion stalls	Artifacts remain at CDN/storage	Durable, observable deletion workflow with retries

A few details are worth emphasizing.

A failed transcode should not destroy an already-playable video. Reprocessing writes a new version and leaves the current version active until validation succeeds. Rollback should restore a known-good version pointer, while respecting current privacy/deletion state.

A missing segment is not always a transient network error. Repeated 404s may signal a publication race, incorrect manifest, or missing origin replica. Player retries can help temporarily, but they do not replace fixing the source inconsistency.

A deletion is a workflow, not one SQL statement. Stop new authorization promptly, update discovery, revoke delivery as policy requires, and track eventual cleanup of CDN and stored artifacts. Do not promise immediate worldwide physical deletion without an enforcement mechanism that actually provides it.

14. What to measure in production

A successful architecture needs observability for both creator and viewer experiences.

For uploads, track session success, part retries, finalize latency, abandoned sessions, orphaned objects, and finalization reconciliation failures.

For processing, track oldest-job age, queued media minutes, stage/rendition failure rate, retry/DLQ growth, processing SLO attainment, and cost per encoded media minute.

For playback, track startup time to first frame, rebuffer ratio, fatal playback errors, bitrate/quality switches, segment 4xx/5xx, CDN hit ratio, shield misses, and origin egress.

For security and lifecycle, monitor authorization denials, privacy/deletion propagation lag, revoked-credential behavior, stuck deletion jobs, and unexpected access to deleted/private media.

Keep operational dashboards separated by region, video/version, codec, and device class where practical; otherwise a small but severe compatibility or regional problem may disappear inside global averages.

15. Architectural trade-offs: no universal answer

Decision	Baseline choice	When to reconsider
Upload bytes	Direct-to-object-storage multipart upload	Small prototype or inline inspection requirement
Processing	Asynchronous jobs with durable handoff	Tiny files and truly negligible processing work
Encoding	Precompute a useful baseline ladder	Large long tail makes selective/lazy encoding worthwhile
Media publication	Immutable versions + atomic active pointer	Generally retain; this is a correctness mechanism
Delivery	CDN + shield + protected origin	Single-region, tiny viewer population
Search	Dedicated index at sufficient scale	Database full-text search may suffice early
Display views	Async, approximate aggregation	Audited financial counters need stronger guarantees
Regions	Start simple; add regional origins when justified	Strict global availability/upload locality requires them
CDN providers	Single provider initially	Provider-level availability or routing requirements justify multi-CDN

The point of a system-design interview is not to name every possible component. It is to explain which requirement forces each component to exist, and how the design behaves when that component fails.

16. How to present this in a system-design interview

Start by clarifying VOD versus live, upload versus playback scope, privacy, target scale, and SLOs. Estimate ingest, processing work, storage, and peak playback egress. Then draw the ingest/processing and delivery paths separately.

Spend your deepest technical time on four boundaries:

Upload finalization: what does the server actually know, and how do retries behave?
Durable job handoff: what happens if the database commits but queue publication fails?
Safe media publication: can duplicate workers or incomplete segments ever become viewer-visible?
Playback and privacy: how do CDN caching, ABR, and access revocation coexist?

Then test the design with a viral-video spike, a processing backlog, a CDN outage, and a public-to-private transition. These scenarios reveal much more than adding more boxes to the architecture diagram.

Final takeaway

A robust video platform doesn't become reliable just because it has object storage, Kafka, transcoding workers, and a CDN. It becomes reliable when the transitions between those systems are explicit: an upload is verified before being finalized, a processing job cannot be silently lost, incomplete media cannot be published, cached bytes cannot bypass authorization, and retries cannot corrupt a previously good version.

That is the difference between a diagram that looks scalable and a video-streaming architecture that can survive production failures.

Explore the full video streaming system design on SeeItFlow

The article is a condensed, standalone walkthrough. For the full visual and technical treatment:

SeeItFlow is a free visual learning platform for software engineering and system design: seeitflow.com.

Ticket Booking System Design: Prevent Double Booking at Scale

MANGESH MANDLIK — Fri, 16 Oct 2026 04:53:46 +0000

Ticket Booking System Design: How to Prevent Double Booking Under Extreme Demand

At 10:00 a.m., tickets for a popular concert go on sale. Thousands of customers refresh the same seat map. Two people click the same front-row seat almost simultaneously. Both see it as available. Both press Reserve.

Only one of them can own it.

That sounds like a small database problem, but a production ticket-booking system must preserve that guarantee while handling abandoned checkouts, delayed payments, expired holds, retries, lost messages, and sudden traffic spikes. Preventing the first double-booking race is relatively straightforward. Preventing an old expiry task from releasing a new customer's seat—or a late payment from confirming a seat already sold to someone else—is where the design gets interesting.

This article develops a practical architecture for assigned-seat booking, such as concert, cinema, or stadium ticketing. General-admission inventory is a related variant, but its capacity-counter model differs from the per-seat model used here.

1. Start with the invariants, not the microservices

The primary invariant is that one event-specific seat can have at most one live owner: either an active, valid hold or a confirmed booking. A seat must never be confirmed for two unrelated bookings.

A second invariant is that one logical customer request should not create multiple holds or bookings merely because the client retries. Atomicity and idempotency solve different problems. Atomicity stops competing buyers from acquiring the same seat. Idempotency stops a repeated request from being interpreted as a new purchase attempt.

Other requirements follow from those two:

Users can browse events, inspect seat maps, select one or more seats, pay, receive tickets, view their bookings, and cancel when policy permits.
A temporary hold must expire safely if checkout is abandoned.
Payment outcomes may be successful, failed, or unknown; a timeout does not prove failure.
Availability displays may be slightly stale, but reservation decisions must use authoritative state.
Under exceptional demand, the system should shed or pace excess work without weakening reservation correctness.
Booking, payment, and ticket-issuance discrepancies must be discoverable and recoverable.

We should also clarify whether the event uses assigned seats or general admission. For assigned seats, a customer purchases a particular event-specific seat, such as Section A, Row 3, Seat 12. For general admission, a customer purchases a quantity from a bounded capacity pool. Both must prevent overselling, but they need different inventory operations.

2. A useful first-cut architecture

A booking platform benefits from two deliberately different paths.

Browsing path: Client → API gateway → Availability service → cache or read model. This path returns seat maps quickly. The underlying view can be a cached snapshot, a read replica, or a materialized projection. Its freshness target is a product decision.

Reservation path: Client → admission control, where needed → Booking service → authoritative transactional database. This path creates durable holds using atomic operations. It never grants a reservation because the cache says a seat is available.

The Booking service coordinates with a separate payment integration. After a valid confirmation, an outbox relay publishes a booking.confirmed event. Ticket issuance and notifications consume that event asynchronously and idempotently. Expiry workers release abandoned holds, while reconciliation workers investigate unresolved or inconsistent states.

Not every deployment needs a separate service for each box. A smaller system can implement several responsibilities in one application while retaining the same transactional boundaries and invariants. The important separation is between displaying availability and authoritatively allocating inventory.

3. Model the event, seat, hold, booking, and payment separately

A physical seat belongs to a venue, but the sellable inventory unit belongs to a particular event. Seat A12 can be sold for Friday's concert and independently for Saturday's concert. Therefore, the unique sellable identity is event-scoped, conceptually (event_id, seat_id).

A minimal domain model looks like this:

Entity	Responsibility
Event	Schedule, sale window, venue, inventory mode
Event seat	Event-specific sellable seat, pricing zone, authoritative allocation state
Hold	Durable identity, owner, selected seats, creation/expiry time, lifecycle state and version
Booking	Customer purchase intent and confirmation lifecycle, linked to hold and payment reference
Payment	Provider-owned outcome and attempt/reference information
Ticket	Usable admission entitlement issued for a confirmed booking

A hold is not just seat.status = HELD. It has an identity (hold_id) that survives retries and lets delayed workers prove which allocation they are acting on. A multi-seat hold also gives several seats one shared logical reservation identity.

A possible set of separate state machines is:

Hold:     ACTIVE → CONFIRMED
                 ↘ EXPIRED / RELEASED

Booking:  CREATED → PAYMENT_PENDING → CONFIRMED
                  ↘ FAILED / CANCELLED (as applicable)

Payment:  PENDING / UNKNOWN → SUCCEEDED / FAILED

These are illustrative state names, not a requirement to use precisely these enums. Cancellation, refund, and ticket validity can each have additional independent states. A booking can be payment-pending while its hold is active; it can also be confirmed while a refund is pending. Combining everything into one booked boolean loses essential information.

4. Why a simple availability check double-books seats

Consider a naive implementation:

User A reads A12 → AVAILABLE
User B reads A12 → AVAILABLE
User A writes A12 → HELD
User B writes A12 → HELD

Both requests made a decision from an earlier observation. The sequence read → decide → write is not atomic. Application-level checks do not fix that race when concurrent requests can interleave.

For a single assigned seat, a conditional update in an authoritative transactional database is a strong baseline:

UPDATE event_seats
SET state = 'HELD',
    hold_id = :hold_id,
    held_until = :expires_at,
    version = version + 1
WHERE event_id = :event_id
  AND seat_id = :seat_id
  AND state = 'AVAILABLE';

Exactly one competing update can transition a currently available row to held, assuming the database provides the expected atomic update semantics and all allocation writers use the same authoritative ownership rules. The winner sees one affected row; the loser sees zero and gets a seat-conflict response.

The seat update, creation of the durable hold, and the request's idempotency record should commit in one transaction. A database uniqueness constraint on (event_id, seat_id) identifies the sellable row; an allocation constraint or guarded transition must also prevent competing live owners if allocations are stored in a separate table. Otherwise a crash could leave an allocated seat without the durable identity needed to manage it safely.

An alternative is optimistic concurrency using WHERE version = :expected_version; another is SELECT ... FOR UPDATE with a short row lock. All can enforce the invariant. The choice depends on contention and the number of related rows being changed.

Never keep a database row lock open while the customer completes payment. The lock protects the short transaction that establishes the durable hold. The hold's persisted ownership and expiry—not a long-running transaction—protect the checkout window.

5. Reserving multiple seats must be all-or-none

Suppose a customer selects A1, A2, and A3. If A2 is already held, silently reserving A1 and A3 usually violates the purchase contract.

When all selected seats share one transactional database boundary, acquire or conditionally update them in a single transaction. Process seats in a deterministic order, such as ascending seat ID, verify that every requested seat was acquired, and commit only if the complete set succeeded. If any seat conflicts, roll back the entire attempt.

Deterministic acquisition order reduces deadlocks: otherwise one request could lock A1 then wait for A2 while another locks A2 then waits for A1. It does not eliminate every possible database deadlock, so transactions still need sensible error handling and bounded retries for genuinely transient failures.

Cross-partition bookings are harder. If seats are distributed across separate transactional shards, there is no automatic all-or-none commit. Options include constraining multi-seat selections to one partition or implementing coordinated temporary sub-holds with compensation and reconciliation. A saga can aim for an eventual all-or-none business outcome using provisional sub-holds and compensation, but it does not magically provide a single instantaneous atomic transaction across shards. During partial progress, provisional allocations must remain unavailable to competing buyers; compensation can fail and therefore requires durable retries and reconciliation.

6. Hold idempotency protects against lost responses

Imagine the booking database commits a hold, but the HTTP response is lost. The customer presses Reserve again. Without idempotency, the server may treat the retry as a second independent request.

A stable idempotency key should identify the logical hold request. Persist the key and a request fingerprint—such as event ID and sorted selected seat IDs—under a durable uniqueness constraint, alongside the hold creation transaction. On retry:

If the key already exists with the same fingerprint, return the existing hold and its current state, including its original expiry.
If the same key is reused for a different seat selection, reject the conflict.
If the key is new, attempt the atomic reservation.

Concurrent requests using the same key must also be serialized or resolved through the database's uniqueness enforcement. Merely checking for an existing key before inserting one is another check-then-act race.

A retry must not silently extend the original hold's expiry or create a fresh hold unless the product explicitly defines and safely implements that separate operation.

7. Expiry is another concurrency problem

Temporary holds prevent other buyers from taking a seat while the first buyer pays. But abandoned holds cannot occupy inventory forever. A server-side expires_at establishes the business validity window; the countdown shown in the browser is only a display.

The subtle bug is a delayed expiry worker releasing the wrong customer's seat:

10:00  Hold H1 owns A12
10:05  H1 expires and A12 is released
10:06  Hold H2 acquires A12
10:07  A delayed expiry task for H1 runs

If the task executes release A12 WHERE state = 'HELD', it releases H2, even though H2 is legitimate.

The release must identify the specific old hold:

UPDATE event_seats
SET state = 'AVAILABLE',
    hold_id = NULL,
    held_until = NULL,
    version = version + 1
WHERE event_id = :event_id
  AND seat_id = :seat_id
  AND hold_id = :expired_hold_id
  AND state = 'HELD'
  AND held_until <= :db_now;

Here :db_now represents a consistently evaluated authoritative time, ideally derived from the database's time semantics. The seat release and transition of the corresponding hold to its terminal state must be coordinated transactionally when they share the same database. In particular, a worker must not mark the hold expired and commit before releasing its seats in an unrelated transaction: that can strand inventory or create inconsistent ownership. For a multi-seat hold, release the complete set according to its lifecycle contract, not as unrelated best-effort seat updates.

A delayed or duplicated expiry task now becomes a safe no-op if the seat has changed owners. Confirmation must use equally strict ownership, state, and expiry checks. Expiry and confirmation cannot independently succeed for the same hold.

Worker lag is not extra hold validity. Even if an expiry worker is down, reservation and confirmation operations must honor the persisted expiry semantics. A hold with expires_at <= authoritative_now is no longer eligible for ordinary confirmation, even if its stored state has not yet been cleaned up. A new reservation may need to reclaim an expired allocation transactionally rather than wait for a background scan, but it must do so using the same identity/version protections.

Expiry workers can use indexed (state, expires_at) scans, bounded batches, delayed messages, or durable workflow timers. A Redis sorted set can accelerate scheduling but should not become the only authority for seat ownership. Every mechanism needs idempotent, identity-scoped release. Monitor the oldest overdue hold and expiry-processing lag.

8. Payment success and seat ownership are separate facts

A successful payment is necessary for a paid booking, but payment success alone cannot grant a seat. The Booking service must still prove that the corresponding hold validly owns the requested inventory when confirmation commits.

The payment service can return three broad outcomes:

Known success: Attempt confirmation through the guarded authoritative transaction.
Known failure: Transition the booking appropriately and release the hold safely, provided no other payment attempt or outcome remains unresolved.
Unknown: Preserve the ambiguity and resolve it through supported provider status queries, authenticated webhooks, safe same-identity retries where explicitly supported, or reconciliation reports and escalation.

A timeout is not evidence of failure. Retrying a charge as a brand-new provider operation can create a duplicate payment. A PSP status-query endpoint is useful when available, but it is not required by the design: authenticated callbacks, settlement/reconciliation files, and provider escalation can be the supported resolution channels. If none gives definitive evidence by an operational deadline, move the case to an unresolved exception queue—not automatically to FAILED. Payment and booking each need durable, stable identities so their histories can be correlated.

The hardest case occurs when a payment succeeds after the hold expires. There are two materially different outcomes.

The seat is still unallocated. The product may choose to attempt an atomic reacquisition and confirm, but only if that reacquisition wins against all competing buyers. Alternatively, it can void or refund the payment. It cannot pretend the expired hold was still valid.

The seat has been reallocated. The original customer must not reclaim it. The booking becomes an exception requiring void/refund or another explicitly supported recovery policy, customer communication, and reconciliation. The newer customer's legitimate allocation remains untouched.

Extending a hold for a bounded period after payment initiation can reduce late-success incidents. It cannot guarantee their elimination: a provider outage or delayed reconciliation can outlast any finite timeout.

The critical implementation detail is that booking confirmation, hold transition, seat ownership finalization, and the durable event-to-publish should commit together when they share one authoritative transactional database. If the design splits these facts across stores, it needs explicit coordination, compensation, and reconciliation rather than an assumed atomic commit.

9. Booking confirmation is not the same as ticket issuance

Once payment and ownership checks succeed, the booking is committed as confirmed. The system must then issue one or more tickets and notify the customer. These downstream operations can fail independently.

A direct sequence—commit booking, then publish booking.confirmed—has a dual-write gap. The process may crash after committing but before publishing. The customer has a confirmed booking but never receives a ticket.

The transactional outbox closes this durability gap:

BEGIN TRANSACTION
  Verify valid hold ownership and payment-success evidence
  Mark booking CONFIRMED
  Finalize hold and seat allocation
  Insert booking.confirmed into outbox
COMMIT

Outbox relay → broker → ticket issuance / notification

The relay can crash after publishing but before marking the outbox row delivered. It may therefore publish again. Outbox delivery is not automatically exactly-once. Consumers must be idempotent.

For ticket issuance, a stable business uniqueness constraint such as (booking_id, seat_id) for assigned seating—or (booking_id, ticket_sequence) for quantity-based tickets—prevents a repeated event from minting a second independent entitlement. The constraint must reflect the actual product's ticket identity and reissue rules.

If ticket generation fails, the already-confirmed paid booking should remain confirmed. Retry issuance, alert on missing entitlements, and provide a way for the customer to retrieve the ticket later. Notification failures should similarly not undo a valid purchase.

10. Cached seat maps can be wrong without corrupting bookings

Seat-map reads are frequent and often vastly outnumber successful reservations. Serving every refresh from the authoritative write database would create unnecessary contention during popular on-sales.

A cache or read model can be slightly stale in either direction:

Stale positive: The UI shows A12 as available, but someone else already holds it. The authoritative reservation rejects the attempt. This hurts conversion, not correctness.

Stale negative: The UI still shows A12 as held after its hold expired and released it. Another customer misses an opportunity until the view refreshes. Again, a display issue rather than a double-allocation.

Choose a freshness target based on product expectations. TTLs, event-driven invalidation, read-model updates, polling, and WebSocket/SSE pushes are implementation choices, not correctness mechanisms. Push updates make the interface feel live; they do not make a reservation authoritative.

If the cache fails, do not let every client request fall through to the booking database at once. Rate-limit, serve acceptable stale snapshots, reduce refresh frequency, or shed nonessential browsing traffic to protect the reservation path.

11. A virtual waiting room protects the bottleneck; it does not allocate seats

A blockbuster event might attract far more simultaneous buyers than the booking database or payment provider can safely handle. Scaling API servers alone does not increase the authoritative database's ability to commit conflicting seat transitions.

A virtual waiting room controls admission to the booking flow. It can issue scoped, time-limited admission tokens and pace entry at a measured rate based on the slowest critical dependency: booking-write throughput, payment processing capacity, and operational ability to expire and reconcile holds.

The waiting room is a fairness and load-management mechanism, not a reservation guarantee. A user admitted to checkout can still lose a seat to another user who wins the atomic allocation first. Queue position may be approximate; reconnects and token reuse need deliberate handling. A token must not grant extra admission or bypass booking idempotency.

A backend message queue is not the same thing as a user-visible waiting room. An internal queue buffers work between services; a waiting room deliberately gates users before they enter a scarce-inventory checkout experience. For smaller events, clear rejection and backoff may be simpler than operating a waiting room.

The system should distinguish temporarily unavailable (all seats held, some may return) from fully sold (all inventory confirmed or otherwise permanently unavailable). Otherwise it may unnecessarily stop admission while abandoned holds are about to expire.

12. Sharding by event ID does not solve one hot event

Partitioning by event_id is useful because unrelated concerts or screenings no longer compete on the same database partition. But a single major concert still concentrates all its reservation traffic on one shard.

For that case, finer partitioning by event → section/block → seat can distribute independent seat operations. The trade-off is important: a multi-seat request crossing partitions no longer benefits from one local all-or-none transaction.

There is no universal rule that every system should shard down to individual seats. Choose partition boundaries using observed demand, transaction patterns, and the product's multi-seat requirements. Admission control may be sufficient before the complexity of cross-partition reservation is justified.

Multi-region ownership adds another layer of risk. Two regions must not independently grant the same seat during a network partition or failover. A single authoritative writer per seat/partition, with explicit leader fencing and safe failover, is a practical default. A naive active-active setup is not a harmless availability optimization; it can violate the core inventory invariant.

13. An end-to-end reservation and confirmation contract

It helps to connect the components into one precise workflow. The following is a logical sequence, not a claim that every deployment needs these exact APIs:

POST /events/{eventId}/holds
  Idempotency-Key: <stable request key>
  body: { seatIds: ["A1", "A2"] }
  → 201 { holdId, expiresAt, state: "ACTIVE" }
  → 409 if any seat cannot be acquired

POST /bookings/{bookingId}/payment
  → starts or retrieves the payment operation with its stable identity
  → { paymentState: "PENDING" | "UNKNOWN" | "SUCCEEDED" | "FAILED" }

GET /bookings/{bookingId}
  → returns the durable booking, hold, and ticket status

The server must authenticate and authorize each request; a caller cannot select another user's hold simply by knowing its identifier. The seat IDs must belong to the requested event and sale window, prices and eligibility must be validated on the server, and any client-submitted price must not be trusted. The booking must bind the charged amount and currency to the server-approved order snapshot; if price or seat selection changes, the system must create an explicitly new authorized operation rather than silently reusing an old payment identity.

The hold creation transaction must make three things agree: the seats' allocation, the durable hold row, and the idempotency mapping. A second transaction later confirms the booking only when payment-success evidence and current, unexpired ownership are both valid. In the local-transaction baseline, that confirmation also finalizes seats and inserts the outbox event. No database transaction stays open across the external payment call.

For an asynchronous payment method, a PENDING response is a legitimate checkout result, not a failure. A client can poll its own booking-status API; the backend may learn the financial outcome through whichever provider channels actually exist. An API reporting our recorded booking state must not be confused with an API querying the provider's payment state.

14. Ticket validity and cancellation need their own rules

Issuing a ticket is not equivalent to authorizing entry. A scannable ticket should map to a durable entitlement with a unique identity and a defined validity lifecycle. A duplicate booking.confirmed delivery must retrieve or reuse that entitlement, not mint a second independently usable one. For a venue that checks tickets at the gate, the scan path must also enforce the product's admission rule—typically preventing a single-use entitlement from being accepted twice. Offline scanners and multi-gate operation require their own synchronization/conflict policy; the booking database alone cannot guarantee online, globally coordinated admission when scanners are disconnected.

Cancellation is another state transition, not a synonym for refund completion. Validate the cancellation window and whether the ticket has already been used; revoke or invalidate the entitlement where required, release inventory only when resale is permitted, and initiate a separate refund workflow. Refunds can be pending or ambiguous just like charges. A used ticket or an offline gate that has not received a revocation requires an explicit policy rather than a claim of instant global invalidation.

15. Failures that should be in the design review

A useful review asks what the system does after a partial success, not just whether the happy path works.

Failure	Required safe behavior
Hold committed, response lost	Retry with the same idempotency key returns the existing hold
Two users select one seat	Only one authoritative transition wins
One seat conflicts in a multi-seat request	Roll back all seats if the contract is all-or-none
Old expiry task runs after a new hold	Identity-scoped release affects zero rows
Expiry worker stops	Expiry remains a business rule; backlog drains in bounded batches after recovery
Payment times out	Remain unresolved; never infer a decline or create a fresh independent charge
Payment succeeds after seat reallocation	Preserve the new owner's seat; resolve the original payment via exception handling
Duplicate or out-of-order webhook	Deduplicate and validate allowed state transitions
Booking commit succeeds, event publish fails	Durable outbox row is retried by relay
Ticket consumer runs twice	Business-key uniqueness prevents duplicate usable entitlements
Availability cache fails	Degrade browsing without overwhelming the booking-write database
One hot event saturates a partition	Apply event-specific admission control; consider finer partitioning
Region fails during reservation	Fence the former writer; reconcile uncertain in-flight operations

Two additional operational cases deserve explicit policies.

Cancellation versus ticket use: A cancelled booking does not mean its refund has succeeded. If tickets are scanned for admission, cancellation must coordinate ticket revocation with the admission-validation system. A ticket already used before cancellation may require a policy exception or manual review; a simple database status update is insufficient.

Hold storms: A large number of users may acquire seats but abandon payment, making an event look sold out even though many seats will return. Track hold-to-confirmation conversion and active-hold pressure, not just raw reservation attempts.

16. Reconciliation is part of the architecture, not an incident script

Retries and asynchronous messaging do not resolve every ambiguity. Reconciliation continuously compares the booking system's durable records with payment evidence, inventory ownership, outbox delivery, and ticket issuance.

Examples include paid-but-pending bookings, successful payments attached to expired holds, confirmed bookings without the expected payment evidence, seats marked held without a matching active hold, confirmed bookings without tickets, unpublished outbox events, and cancellations with unresolved refunds.

Reconciliation jobs should be idempotent, bounded, observable, and safe to rerun. A provider status API, when offered, can help operational reconciliation; otherwise use the provider's supported reports, settlement evidence, callbacks, or escalation. Settlement may arrive substantially later than the checkout request, so unresolved financial cases need an age-based escalation policy without inventing a success/failure outcome. They should apply the same guarded state transitions as live requests, not bypass correctness checks in the name of repair. Financially sensitive or ambiguous cases should enter an exception workflow rather than be guessed into a terminal state.

Metrics that matter include reservation conflict rate per event/seat, active holds and expiry lag, payment-unknown age, booking-confirmation latency, reconciliation mismatch count and oldest unresolved case, outbox lag, ticket-issuance lag, waiting-room depth, and booking-database saturation.

17. The architectural trade-offs worth explaining

A strong design does not claim every mechanism is always necessary. It explains which risk justifies each mechanism.

Decision	Sensible baseline	What changes the choice
Seat concurrency	Conditional atomic update	Complex multi-row transactions may favor short row locks; versions help with other concurrent updates
Checkout	Durable temporary hold	Very fast purchase flows or ample general admission may use different sequencing
Multi-seat booking	One local transaction	Cross-shard selections need constraints or compensation
Expiry	Indexed scan with safe conditional release	Delay queues/timers can improve scheduling; none replace ownership checks
Availability	Cache/read model with freshness target	High-demand UX may justify invalidation or push updates
Traffic spikes	Admission control sized to write-path capacity	Ordinary demand may need only rate limits/rejections
Event partitioning	Event-level isolation	One hot event may need section-level partitioning
Confirmation events	Transactional outbox	Simpler direct publishing leaves a crash-induced dual-write gap
Regions	One fenced write owner per partition	Active-active requires explicit conflict-prevention architecture

The key is to distinguish optimizations from guarantees. A faster seat map improves UX; it does not prevent double booking. A longer hold improves checkout conversion; it does not guarantee payment resolution. A waiting room improves admission control; it does not guarantee inventory. An outbox improves durable event delivery; it does not eliminate duplicate processing.

18. How to explain this in a system design interview

Start with the two workloads: fast, potentially stale availability reads and correct, authoritative reservation writes. State the no-double-sell invariant and the separate idempotency requirement. Show the naive two-buyer race before introducing a conditional atomic transition.

Then build the design in the order complexity appears: a durable hold, all-or-none multi-seat acquisition, identity-scoped expiry, payment ambiguity and late success, booking confirmation, outbox-driven ticket issuance, and reconciliation. Only after that introduce the waiting room, hot-event partitioning, and regional failover.

A useful interviewer follow-up is: “The payment succeeded, but the hold expired and somebody else now owns the seat. What happens?” The answer should not be “increase the TTL.” The answer is that the old booking cannot be confirmed against someone else's allocation; the late payment needs a defined refund/void or alternate-fulfillment path, and the system must reconcile the financial outcome.

Another is: “An expiry job for yesterday's hold runs against a seat that is held again today.” If the release checks only state = HELD, the design is unsafe. If it checks the specific hold_id and authoritative expiry/version in the same guarded transaction, the stale job is harmless.

These scenarios distinguish an architecture that works in a diagram from one that remains correct when events arrive late, twice, or in the wrong order.

Closing thought

The hardest part of ticket booking is not rendering a seat map or processing a payment. It is maintaining a consistent account of who owns scarce inventory when the network, workers, customers, and payment providers do not move in lockstep.

Atomic reservation chooses one winner. Durable, idempotent holds remember that winner. Identity-scoped expiry prevents stale cleanup from stealing the next winner's seat. Payment coordination refuses to mistake uncertainty for failure. The outbox and reconciliation paths make the system recoverable after partial success.

That is the foundation of a ticket-booking system that can handle high demand without turning operational failures into double bookings.

Explore the complete visual and technical breakdown on SeeItFlow:

E-commerce System Design: Inventory, Checkout, Orders & Scale

MANGESH MANDLIK — Thu, 15 Oct 2026 09:43:14 +0000

A shopper adds the last available item to a cart. Another shopper does the same. Both click Place Order. One payment times out; the other succeeds. Meanwhile, the product price changes, the fulfillment queue is delayed, and the checkout service restarts.

A credible e-commerce system design must explain what happens next—not merely draw boxes labeled Catalog, Cart, Order, and Payment.

The central challenge is that an online store contains several workloads with different consistency requirements. Browsing and search benefit from caches and asynchronous indexing. Inventory reservations and order transitions require guarded writes. Payments involve external systems whose outcomes may remain unknown after a timeout. Fulfillment happens on a different timeline altogether.

This walkthrough builds an Amazon-like online shopping platform architecture for a general retailer. It is an illustrative design, not a claim about Amazon's internal implementation. It emphasizes the decisions and failure modes that matter in production and in a system design interview.

For the full interactive learning path, start with E-commerce Platform — Complete Design on SeeItFlow.

1. Requirements: what are we actually building?

Our baseline supports product browsing, keyword search and filters, SKU selection, guest and signed-in carts, checkout, inventory reservation, payment, order history, fulfillment tracking, cancellation, returns, and refunds.

We will initially assume a single retailer selling physical goods, with a finite stock policy and an external payment service provider. Multiple sellers, warehouse selection, sophisticated promotion engines, fraud detection, and international tax calculation can be added as explicit extensions. A marketplace changes the ownership of offers, inventory, and fulfillment; it is not merely the same architecture with more users.

Rather than assigning one latency or consistency target to the whole platform, specify them per workflow:

Workload	Optimization goal	Correctness boundary
Catalog	Low-latency, high-volume reads	Product data can be briefly stale in display
Search	Relevance, filtering, independent scaling	Search index is a derived discovery view
Cart	Fast mutable session state	Cart does not grant a stock reservation
Pricing	Efficient display and quoting	Checkout uses an accepted, versioned quote
Inventory	Safe concurrent reservations	Allocation never exceeds the defined stock policy
Checkout and orders	Recoverable workflow	Durable identities and guarded transitions
Payment	Safe external side effects	Unknown outcome is not a decline
Fulfillment	Reliable asynchronous processing	No duplicate fulfillment for the same operation

Availability and latency SLOs should be derived from business needs and measured load; numbers quoted in interviews are design assumptions, not universal industry guarantees.

2. Model products, SKUs, offers, and inventory separately

A Product describes an item: title, category, brand, images, and attributes. A SKU identifies a purchasable variant, such as a black T-shirt in size M. An Offer describes the commercial terms under which a merchant sells a SKU, including currency, price, validity, and seller identity when applicable.

Inventory is not a property of the abstract product. It belongs to the relevant stock pool, often keyed by SKU and fulfillment location. For a marketplace, seller-specific offers and stock ownership need additional modeling.

Product (product_id, title, attributes)
   ├── SKU A (sku_id, color=black, size=M)
   │      ├── Offer (seller, currency, price_version)
   │      └── Inventory (sku_id, location_id, on_hand, reserved)
   └── SKU B (sku_id, color=blue, size=L)
          └── ...

An order must also preserve a snapshot of what the customer accepted: SKU, quantity, product description as needed, unit price, currency, discounts, taxes, shipping allocation, and totals. Never recalculate a historical order using today's catalog price.

This separation prevents frequent price and stock updates from requiring the same storage and cache policies as relatively stable catalog metadata.

3. Catalog architecture: cache the browsing path

A straightforward catalog read path uses a database plus a cache-aside layer. Images and other static media are served through a CDN.

Browser ──> API ──> Catalog Service ──> Cache
                           │               │ miss
                           └──────────────> Catalog DB

Browser ──> CDN ──> Product images / static media

Popular products benefit from caching, but cache hit rate depends on popularity distribution, catalog size, TTL, and capacity. Images can be updated, so version asset URLs or invalidate CDN entries rather than assuming every image URL is immutable.

Two operational details matter. First, a cache outage can suddenly redirect a large fraction of traffic to the database. Protect the origin with load shedding, circuit breakers, and capacity-aware fallbacks. Second, a hot entry expiring during a traffic spike can cause a cache stampede. Single-flight request coalescing, TTL jitter, and carefully chosen stale-while-revalidate behavior help on read-only display data.

A cached product page may show an approximate stock count or an older price. Neither value authorizes a reservation or a charge.

4. Search is a derived read model, not a second source of truth

Product search needs full-text matching, ranking, filters, and potentially typo tolerance and autocomplete. A relational database's full-text search may be sufficient initially; a dedicated search index becomes valuable when search functionality or load justifies it.

Catalog DB ──> Durable change pipeline ──> Indexer ──> Search Index
                                                      │
Customer query ──> Search API ──────────────────────────┘

Index updates are asynchronous, so the search index can lag behind the catalog. Define an indexing-lag SLO and measure it. A product may briefly appear in search after being disabled or sold out; checkout must revalidate SKU status, accepted pricing, and inventory independently.

This is a useful CQRS-style separation: the transactional catalog is authoritative, while search is optimized for discovery. It does not mean every e-commerce application needs a full CQRS framework.

5. Cart design: purchase intent is not inventory ownership

A cart normally stores sku_id, quantity, timestamps, and perhaps a displayed-price snapshot, keyed by user or guest session. A key-value store can fit frequent small updates, though a relational store is also viable.

Adding an item to the cart generally does not reserve stock. Otherwise abandoned carts could hold scarce inventory for long periods. Reservation at checkout is a sensible baseline for ordinary retail, while ticketing and limited drops may deliberately reserve earlier for a short, bounded window.

Guest-cart merging after login needs an explicit policy: combine quantities, prefer one cart, or ask the shopper. It should not silently create invalid quantities or imply that a saved cart guarantees availability.

At checkout, revalidate every SKU's purchasability, quantity, price, and availability. Cart state is a starting point, not authority.

6. Pricing: distinguish display price, checkout quote, and order price

One subtle failure is treating the price rendered on a product page as the price the payment service must charge.

There are three distinct values:

Display price: a potentially cached browsing value.
Checkout quote: the authoritative commercial decision for the current items, currency, applicable discounts, tax and shipping policy, with a version and/or expiry.
Order line price: the accepted terms snapshotted onto the order for history and financial reconciliation.

An authoritative quote does not necessarily mean a direct database read on every request. A pricing service or appropriately versioned and invalidated cache can provide it. What matters is that the checkout decision obeys the pricing policy and quote validity rules.

If the price changes between browsing and checkout, show the updated terms and obtain acceptance as required by the product policy. Do not silently charge a different amount. Once accepted, ensure the payment amount and currency correspond to the persisted quote and order snapshot.

7. Inventory reservation: the last-item race

Suppose there is one sellable unit of SKU-42. Two shoppers attempt to buy it simultaneously. A read-then-write sequence without concurrency control is unsafe: both requests may observe one available unit and both proceed.

For a simple stock pool, define:

available_to_sell = on_hand - reserved - safety_stock

This is a simplified model. Production inventory may also distinguish damaged, allocated, in-transit, and backordered units. Some retailers intentionally permit backorders; the real invariant is that reservations do not exceed the allocation policy for the SKU and stock pool.

A relational implementation can reserve atomically:

BEGIN;

UPDATE inventory
SET reserved = reserved + :qty
WHERE sku_id = :sku_id
  AND location_id = :location_id
  AND on_hand - reserved - safety_stock >= :qty;

-- Proceed only if exactly one inventory row was updated.
-- Insert a uniquely identified reservation in the SAME transaction.
INSERT INTO reservations
  (reservation_id, checkout_id, sku_id, location_id,
   quantity, state, expires_at)
VALUES
  (:reservation_id, :checkout_id, :sku_id, :location_id,
   :qty, 'ACTIVE', :expires_at);

COMMIT;

The application must roll back the transaction if the conditional update affects no row, if the reservation insert fails, or if any invariant check fails. A retry with the same reservation identity must return the existing reservation rather than incrementing reserved again. That requires a uniqueness constraint and idempotent request handling; the SQL sketch alone does not implement the complete retry protocol.

Other valid mechanisms include optimistic version checks, row locks, strongly consistent conditional writes, and serialized per-SKU processing. A distributed lock by itself is not a substitute for an authoritative conditional write.

A reservation is a durable entity, not merely a changed counter. Typical states are ACTIVE, COMMITTED, RELEASED, and EXPIRED. Every transition must be idempotent and guarded so that commit and expiry cannot both win.

8. Reservation expiry must survive crashes

An abandoned checkout should eventually release its held units. Store expires_at durably and enforce expiry through a recoverable mechanism: indexed database scans, a durable delayed task, or a workflow timer.

Do not rely solely on an in-memory timer or cache-key expiry notification. Those can be missed. The expiry worker should perform a conditional transition from ACTIVE to EXPIRED, update the stock accounting consistently, and tolerate retries.

There is a critical exception: a payment timeout does not prove payment failure. If the payment outcome is unknown near reservation expiry, the business needs a bounded policy—extend the hold where possible, or let it expire with a defined late-success allocation-or-refund procedure. Do not hold inventory forever, but do not casually release it on a network timeout either.

9. Checkout is a durable saga, not one global transaction

Inventory, orders, and an external payment provider cannot generally participate in one shared ACID transaction. A checkout therefore coordinates local transactions and external effects through a durable saga.

A defensible baseline sequence is:

Customer
   │ Place Order (idempotency key)
   ▼
Checkout / Order Service
   │ 1. Create or reuse durable PENDING checkout/order
   │ 2. Persist accepted quote and line-item snapshot
   ▼
Inventory Service
   │ 3. Reserve stock idempotently
   ▼
Payment Service / Provider
   │ 4. Initiate or resume the same logical payment
   │    ├── confirmed success
   │    ├── definitive failure
   │    └── pending / unknown
   ▼
Checkout Orchestrator
   │ 5. Resolve payment and reservation conditions
   │ 6. Commit reservation and confirm order via
   │    guarded, recoverable cross-service workflow
   ▼
Order DB + Transactional Outbox
   │ 7. Persist CONFIRMED and order.confirmed together
   ▼
Event Broker ──> Fulfillment / Notification / Analytics

This is a logical workflow, not one atomic operation spanning the inventory and order databases. If reservation commit succeeds but order confirmation fails, the orchestrator must resume or reconcile that intermediate state. Fulfillment must not begin merely because a payment response looked successful; it begins from a durable, valid confirmed order according to the platform's policy.

The checkout idempotency key identifies one logical attempt and maps to a durable checkout/order record. Store a request fingerprint so reusing the same key for a different cart or quote is rejected rather than silently treated as the same purchase. Separate checkout_id, order_id, reservation_id, and payment identifiers remain linked but should not be conflated.

What if the orchestrator crashes?

Record durable workflow progress and give every step a stable business idempotency identity. When execution resumes, query or reconcile the previous step's outcome before deciding whether to repeat an external side effect. Idempotency requires cooperation from each boundary, especially the payment provider; a local database key cannot by itself prevent an unrelated provider charge.

A saga's compensation is not rollback. Releasing stock, voiding an authorization, issuing a refund, or cancelling an order are separate operations that can themselves fail and need retries and reconciliation.

10. Payment success, failure, and unknown are different outcomes

An e-commerce checkout should integrate with a dedicated payment component instead of reimplementing payment-network internals.

A definitive provider decline can trigger an idempotent reservation release and a failed checkout state. A payment still processing should remain pending. A network timeout, dropped response, or delayed webhook leaves an unknown outcome, not a definitive failure.

Use a stable provider-side idempotency mechanism where supported, authenticated webhook processing, provider status queries, and reconciliation. Provider capabilities and guarantees vary by payment method and API, so avoid claiming universal exactly-once charging.

Also distinguish authorization, capture, refund, and settlement. They are different financial events. Some integrations authorize before capture; others capture immediately or complete asynchronously. An order may be confirmed according to a defined payment policy without claiming that bank settlement has already occurred.

The detailed payment state machine belongs in Payment System Design on SeeItFlow.

11. Orders, payments, and fulfillment have separate state machines

A single order_complete flag cannot represent the real lifecycle.

Domain	Illustrative states
Order	PENDING, CONFIRMED, CANCELLED, COMPLETED
Payment	PENDING, UNKNOWN, AUTHORIZED, CAPTURED, FAILED, REFUNDED
Reservation	ACTIVE, COMMITTED, RELEASED, EXPIRED
Fulfillment	UNALLOCATED, ALLOCATED, PICKING, PACKED, SHIPPED, DELIVERED

These examples are not one universal state model: providers and merchants may need partial captures, multiple payment attempts, partial refunds, and per-line or per-shipment fulfillment states.

An order can be confirmed but not shipped. One order can generate several shipments. A cancellation can be accepted while the refund is still pending. Keep each domain's ownership and transitions explicit, and derive the customer-facing status from the combination rather than overwriting one boolean.

12. Transactional outbox: don't lose the fulfillment event

Consider this sequence:

1. Order DB commits CONFIRMED.
2. Process crashes before publishing order.confirmed.
3. Fulfillment never hears about the order.

Publishing to a message broker immediately after the database commit creates a dual-write gap. The transactional outbox addresses it by inserting the order transition and an outbox event in the same local database transaction. A relay later publishes the event and retries on failure.

Order DB transaction:
  UPDATE orders SET status = 'CONFIRMED' ...
  INSERT INTO outbox (event_id, order_id, event_type, payload) ...
  COMMIT

Outbox relay ──> Broker ──> Fulfillment consumer
                           ├── Notification consumer
                           └── Analytics consumer

The outbox protects the durability of the event record, but it does not guarantee exactly-once delivery. A relay can publish successfully and crash before recording the acknowledgment, then publish again. Consumers must deduplicate or apply operations idempotently using stable business identities.

For fulfillment, the uniqueness boundary must match the real operation—such as a fulfillment request or shipment operation per relevant order lines—not blindly order_id if partial shipments are supported. A carrier-label API also needs idempotency or an external-result reconciliation strategy, because database deduplication alone cannot undo a duplicate external shipment request.

Consumers must handle out-of-order events using order versions and guarded transitions where ordering is not guaranteed.

13. Fulfillment, cancellation, returns, and refunds

After order confirmation, fulfillment allocates a location, picks and packs goods, creates shipments, and tracks delivery. This workflow should normally be asynchronous so a slow warehouse does not block the customer's checkout response.

Cancellation eligibility depends on current fulfillment state, not the state the shopper saw seconds earlier. A cancellation request racing with shipment creation must use guarded transitions and coordination with the fulfillment owner. If the shipment has passed the cancellable point, the request may need to become a return.

Returned items should not automatically be marked sellable: inspection, damage, and disposition policy may apply. Similarly, order cancellation is not refund completion. Refunds can be pending, partially completed, failed, or ambiguous. The customer interface and reconciliation process must reflect those distinct facts.

14. Flash sales: one hot SKU defeats naive horizontal scaling

Imagine 100,000 shoppers competing for 1,000 units of the same SKU. A CDN can absorb image and catalog traffic, and stateless API instances can scale horizontally. But every successful stock reservation must still respect one authoritative inventory allocation boundary.

Adding more API servers does not eliminate contention on that hot row, partition, or reservation authority.

Possible mitigations include bounded concurrency, rate limits, exponential backoff with jitter where retries are appropriate, early sold-out rejection, and—when fairness or extreme demand requires it—a waiting room or admission tokens. At the highest contention levels, a serialized per-SKU reservation path or carefully designed inventory preallocation can help, but adds operational complexity and reconciliation needs.

A waiting room is not a correctness guarantee by itself. The final stock allocation must still be guarded. Partitioning by SKU spreads different SKUs across shards; it does not automatically distribute one extremely popular SKU.

For a deeper failure walkthrough, see E-commerce Failure & Scale Scenarios.

15. Failure scenarios worth designing before launch

Failure	Unsafe reaction	Safer design
Two buyers request last unit	Read stock then decrement separately	Atomic conditional reservation
Duplicate Place Order request	Create a new order/payment each time	Durable checkout identity and fingerprint
Reservation succeeds; response lost	Reserve again	Query/reuse same reservation identity
Payment request times out	Treat as declined and release stock	Mark unknown; query/webhook/reconcile
Payment succeeds; order remains pending	Forget the attempt	Recover from durable checkout/payment linkage
Reservation expires during confirmation	Commit and release concurrently	Guarded single-winner state transition
Price changes during checkout	Charge whatever cart displayed	Revalidate and obtain accepted quote
Order commits; publish fails	Lose fulfillment event	Transactional outbox
Fulfillment event redelivered	Create duplicate shipment	Idempotent fulfillment operation
Cancellation races with shipping	Cancel based on stale display	Guarded eligibility and return fallback
Inventory database unavailable	Trust cached stock	Pause/fail reservation safely
Region fails during checkout	Allow both regions to write same stock	Explicit authority, fencing, recovery and reconciliation

Regional failover deserves particular care. Replication alone does not preserve idempotency or prevent two regions from accepting conflicting reservations. Define write ownership, promotion/fencing, recovery-point objectives, and reconciliation before enabling failover. Active-active inventory is an advanced design, not a default checkbox.

16. Reconciliation catches what retries cannot

Even careful idempotency and retry handling cannot guarantee that every cross-system workflow will resolve without intervention.

A reconciliation job should look for:

Payment succeeded, but order remains pending.
Order confirmed without the expected committed reservation.
Reservation committed for a checkout that never confirmed.
Reservation expired while the payment later succeeded.
Outbox event stuck beyond its expected publication lag.
Duplicate or missing fulfillment operations.
Refund expected after cancellation but not completed.

Run bounded, indexed scans or event-driven exception checks; track discrepancy age and severity. Repairs must be idempotent, logged, and safe to rerun. Do not automatically recreate stock allocations, issue refunds, or create shipments whenever the true physical or financial outcome is uncertain. Some cases require quarantine and human review.

Reconciliation is a correctness mechanism, not an excuse to ignore broken invariants on the live request path.

17. Architecture decisions and trade-offs

Decision	Reasonable baseline	When to reconsider
Catalog search	DB search initially, dedicated index when justified	Complex relevance, facets, high search load
Product display	Cache/CDN	Personalization or strict freshness needs
Cart stock hold	No reservation until checkout	Scarce time-boxed inventory
Inventory concurrency	Conditional atomic write	Extreme hot-SKU contention
Reserve vs. pay	Reserve before payment	Authorization-first or domain-specific flows
Checkout coordination	Durable orchestrated saga	Simpler local workflow or event-driven choreography
Order timing	Create durable pending identity before external payment	Alternative models with equivalent recovery anchors
Downstream delivery	Outbox and idempotent consumers	Simpler noncritical events with accepted loss risk
Deployment	Modular monolith can be sufficient	Independent scale, team ownership, release cadence

These are design choices, not a mandate to deploy a dozen microservices. Clean domain boundaries can exist inside a modular monolith. Independent services become worthwhile when deployment ownership, scaling, or isolation needs justify the operational cost.

See E-commerce Trade-offs & Decisions for the full comparisons and Architecture Deep Dive for implementation mechanisms.

18. How to answer e-commerce system design interview questions

How do you prevent overselling? Use an authoritative atomic reservation per SKU/stock pool, a durable reservation identity, and guarded commit/release/expiry transitions. The display count can be stale; the reservation decision cannot be based on that stale count.

What if payment succeeds but the order service crashes? Persist a checkout/order recovery identity before the payment attempt, resolve the provider outcome using the linked payment identity, and resume confirmation or apply the predefined recovery/refund policy. Never blindly charge again.

Why not use one database transaction for checkout? An external payment provider cannot generally join the inventory/order database transaction. Use local commits coordinated by a durable saga with idempotent compensation and reconciliation.

Why do you need an outbox if you already use Kafka or another broker? Because the order database commit and the broker publish are separate writes. The outbox closes that local dual-write durability gap; it does not eliminate duplicate deliveries.

What if 100,000 people buy the last 1,000 units? Protect admission, serialize or guard the inventory allocation boundary, fail fast on sold-out inventory, and keep payment calls downstream of successful reservation. More stateless instances alone do not solve one hot stock record.

Can you cache order status? Yes, a read-only display view can be cached with clear staleness semantics. Order-state mutations still require authoritative guarded transitions.

What changes for a marketplace? Seller-scoped offers, seller/location inventory ownership, split fulfillment, merchant settlement, and additional policy boundaries. Do not assume one product has one global price or one global stock pool.

For more interviewer follow-ups, see E-commerce Interview Perspective.

19. Final architecture and takeaways

                           Customers
                               │
                        API / Gateway
                               │
         ┌─────────────────────┼──────────────────────┐
         │                     │                      │
   Catalog Service       Search Service         Cart Service
     │       │                 │                      │
 Catalog DB  Cache         Search Index            Cart Store
     │                         ▲
     └── Change pipeline ──────┘

                      Checkout / Order
                               │
                   Durable Saga / Order DB
                    │          │         │
               Pricing      Inventory   Payment Service
               Quote        Service     │
                             │          Payment Provider
                       Reservation DB
                               │
                       Order CONFIRMED
                               │
                    Order DB + Outbox
                               │
                         Event Relay
                               │
                            Broker
                         /     |      \
                  Fulfillment Notify  Analytics

         Reconciliation checks order, inventory,
         payment, outbox, and fulfillment outcomes.

The most important lesson in e-commerce system design is not to make every component strongly consistent. It is to identify which decisions change money, stock, or contractual order state—and protect those boundaries precisely.

Cache catalog and search where staleness is tolerable. Keep carts as intent, not inventory ownership. Quote prices authoritatively. Reserve stock atomically. Make checkout durable and idempotent across retries. Treat payment timeouts as ambiguous. Publish confirmed orders through an outbox, make downstream consumers idempotent, and reconcile the gaps that distributed failures leave behind.

The result is an architecture that can scale browsing aggressively without trading away correctness at checkout.

Explore the complete SeeItFlow learning path: Complete Design · Architecture Deep Dive · Trade-offs & Decisions · Failure & Scale Scenarios · Interview Perspective

Payment System Design: Idempotency, Webhooks, Ledger & Failures

MANGESH MANDLIK — Thu, 15 Oct 2026 05:45:49 +0000

A customer taps Pay. The screen spins, the request times out, and the customer taps again.

Did the first payment fail? Did the provider charge the customer but lose the response? Can the second tap safely retry? And if the payment succeeded, what happens when the order service never receives the confirmation?

These are the questions that make payment system design fundamentally different from a typical CRUD service. The difficult part is not sending a request to a payment gateway. It is preserving financial correctness when different systems observe different outcomes.

This article designs a one-time payment orchestration system, starting with a simple flow and evolving it through the failures that justify each architectural component. It covers idempotency, payment attempts, webhooks, transactional outbox, refunds, financial ledgers, reconciliation, provider outages, and scaling.

For an interactive walkthrough, see Payment System — Complete Design on SeeItFlow and the Visual Design Journey.

1. Define the payment system we are designing

We are building a service that accepts a customer's payment intent, coordinates with an external payment service provider (PSP), records the result, and reliably informs downstream systems.

It must support:

Creating and retrieving a logical payment.
Preventing accidental duplicate payments from client retries and concurrent requests.
Handling immediate, pending, and ambiguous provider outcomes.
Receiving authenticated provider webhooks.
Reconciling local records with provider records.
Issuing full or partial refunds when supported.
Recording an auditable history of financial operations.
Triggering downstream actions without coupling their availability to the payment request.

The baseline assumes one-time payments through an external provider. Subscription billing, card-network authorization internals, tax, FX, fraud scoring, and merchant settlement infrastructure are separate systems or extensions. Actual capabilities depend on the provider, payment method, country, and merchant configuration.

Core invariant: retries of one logical payment must not create additional independent charges. This is a design objective enforced by several layers, not an unconditional guarantee that an external financial system can never produce a duplicate.

2. Start with the simplest payment flow

Customer
   |
   v
Payment API --> Payment Database
   |
   v
Payment Provider (PSP)
   |
   +--> synchronous result or pending/next action
   |
   +--> later webhook / status lookup

A safe basic flow is:

Authenticate and validate the request, including the merchant/order, amount, and currency.
Atomically create or find the existing logical payment associated with its idempotency key.
Persist a payment attempt and its stable provider-facing identity before contacting the provider.
Send the operation through the provider adapter, using the provider's documented idempotency mechanism when available.
Persist the outcome actually known: succeeded, definitively failed, requires customer action, pending, or unresolved.
Return the current state and a stable payment identifier. Resolve nonterminal outcomes through subsequent provider messages and reconciliation.

The provider never writes directly to our database. Our service processes the provider's API response or webhook and performs its own guarded database updates.

The payment API should not tell the customer “paid” merely because the provider accepted an asynchronous request. Accepted, authorized, captured, and settled can represent different events.

3. Model payment intent separately from payment attempts

A single payment is a business operation: for example, paying ₹1,500 for order order_847.

An attempt is a particular interaction with a provider to achieve that operation.

Payment: pay_847 (INR 150000 minor units)
  |
  +-- Attempt A: Provider X, definitive decline
  |
  +-- Attempt B: Provider X, successful charge

The second attempt is permitted only after the system knows the first cannot also result in a charge, or after an explicitly designed recovery process. A timeout is not enough evidence to start another independent attempt.

A conceptual schema:

payments (
  payment_id, merchant_id, order_id,
  amount_minor, currency, current_state,
  state_version, created_at
)

idempotency_records (
  merchant_id, operation, idempotency_key,
  request_fingerprint, payment_id,
  UNIQUE (merchant_id, operation, idempotency_key)
)

payment_attempts (
  attempt_id, payment_id, provider,
  provider_idempotency_key, provider_reference,
  attempt_state, created_at
)

The exact constraints depend on the business domain. If one order may legitimately have multiple payments, do not introduce a simplistic globally unique order_id constraint. Conversely, where the business requires only one successful capture for an order, enforce that invariant explicitly at the appropriate transactional boundary.

Use integer minor units or fixed-precision decimal for money, with an explicit currency. Do not use binary floating point for authoritative monetary calculations, and do not assume all currencies have two fractional digits.

4. Payment state is a state machine, not `paid = true`

A simplified flow:

INITIATED --> PROCESSING --> SUCCEEDED
                  |             \
                  |              --> REFUND operation (separate lifecycle)
                  +--> FAILED (definitive)
                  +--> UNRESOLVED (outcome unknown)
                  +--> REQUIRES_ACTION (where supported)

These are internal illustrative states, not a universal provider vocabulary. A real design may also distinguish authorization, capture, cancellation/void, expiry, partial capture, reversal, and partial refund. Some of these belong to separate operation-level state machines rather than a single overloaded payment status.

State changes must be conditional. For example:

UPDATE payments
SET current_state = :next_state,
    state_version = state_version + 1
WHERE payment_id = :payment_id
  AND current_state = :expected_state
  AND state_version = :expected_version;

A zero-row update means the caller must reload and evaluate the current state; it does not automatically mean the provider event is wrong. Transition rules, provider event identity, and, where available, provider sequence/version information help avoid applying stale events.

A delayed PROCESSING webhook must not regress a confirmed successful capture. But refunds, reversals, and chargebacks are new financial facts, not simply stale messages to discard because an earlier payment succeeded.

5. Idempotency: protect the customer from duplicate requests

Suppose the client submits the same payment twice:

POST /payments
Idempotency-Key: checkout-847-v1

{"order_id":"order_847","amount_minor":150000,"currency":"INR"}

The first request creates a durable payment. The second, using the same key and identical meaningful parameters, resolves to that same payment rather than creating a new independent charge.

The implementation needs four pieces.

A durable uniqueness constraint. Scope the key to the appropriate merchant/account and operation. An atomic insert or equivalent transaction arbitrates concurrent requests. A Redis lookup or distributed lock alone is not sufficient: cache eviction, lock expiry, and crashes can erase the only deduplication signal.

A request fingerprint. The same key with a different amount, currency, order, or operation is a conflict, not a request to charge a new amount. Reject it explicitly.

In-progress behavior. If the original request is still running, return its current state, a pending response, or wait for a bounded period. Do not let the duplicate independently call the provider.

A separate provider-side idempotency boundary. Retrying the same provider attempt should reuse the same provider operation identity if that provider documents safe idempotent retries. Local uniqueness prevents duplicate logical records; provider idempotency helps prevent duplicate external effects. Neither mechanism substitutes for the other.

Provider idempotency retention periods and exact replay semantics differ. Do not assume a provider remembers a key forever. Keep durable local business identity and reconciliation records for the periods required by the product and applicable obligations.

Also, an idempotency key cannot deduplicate two logically identical requests if a buggy client generates a new key on every retry. Where required, supplement request-level keys with a stable order/payment-intent invariant.

6. The most important failure: provider timeout

Consider this sequence:

Payment Service             Provider
      |                        |
      |--- charge attempt ---->|
      |                        |  charge succeeds
      |<--- response lost ------X
      |
      |  local outcome: UNKNOWN

The timeout says only that our service did not receive a conclusive response. The request might never have arrived, might still be processing, might have failed, or might already have succeeded.

Never equate a timeout with a declined payment. And never automatically create a new attempt with a new provider identity because the old attempt timed out.

A safe resolution strategy depends on the provider contract:

Query the provider using the stable attempt/reference ID, where supported.
Retry the same provider operation with the same idempotency identity only if the provider documents that as safe.
Accept an authenticated webhook reporting the outcome.
Escalate unresolved attempts to operational reconciliation and, when necessary, financial reconciliation.

These channels can operate in parallel or in a different order; the provider's documented semantics determine which is safest. A provider API status response may itself be provisional, so do not treat every response as a final settlement fact.

If the process crashes after the provider succeeds but before local SUCCEEDED is committed, the persisted attempt and provider reference make later recovery possible. If the local success commit happens but the API response is lost, a retry with the same client key returns the existing payment.

This is why durable identity must exist before the external side effect.

7. Webhooks are necessary, but they are not a single source of guaranteed truth

Many payment methods finish asynchronously. The provider notifies us later:

Provider --> Webhook Endpoint --> Durable Inbox --> Worker
                                               |
                                               v
                                        Payment Database

A robust webhook handler:

Verifies the provider's documented authentication/signature mechanism against the exact bytes and headers required by that mechanism; checks timestamp/replay protection where supported.
Durably records the incoming event or equivalent recoverable work item.
Acknowledges according to the provider's retry contract.
Processes it asynchronously and idempotently.
Applies a guarded state transition and records any consequential event or ledger posting with its own deduplication key.

Do not assume all providers sign the same payload or use the same event-ID semantics. A delivery ID is not always the same as a business-event ID; deduplication should use the provider's documented stable identifier, plus the underlying operation identity where needed.

Expect duplicate delivery, out-of-order events, missing webhooks, and messages arriving after a status query has already resolved the payment. A signature-valid webhook also does not eliminate the need to check that the event belongs to the expected merchant, payment, amount, currency, and operation.

A webhook for an unknown provider transaction should be investigated or quarantined rather than blindly creating a successful payment record.

8. The database-to-queue dual-write problem

Once a payment is confirmed, the system may need to release an order, grant access, notify the customer, and update financial records.

A naïve implementation does this:

1. UPDATE payments SET state = 'SUCCEEDED'
2. PUBLISH payment.succeeded

What if the database commit succeeds and the process crashes before publishing? The payment is recorded, but the order service never hears about it.

The transactional outbox solves that gap:

One local database transaction:
  - apply guarded payment state transition
  - insert outbox event with stable event_id
Commit

Outbox relay --> Message Broker --> Consumers

The relay may publish the same event twice if it crashes after broker acknowledgement but before recording publication. Therefore the outbox supports reliable at-least-once delivery, not magical end-to-end exactly-once effects.

Each consumer needs its own idempotent business operation. For example, order fulfillment can enforce uniqueness on the payment/fulfillment operation it has already applied. Deduplicating by event ID alone may be insufficient if different event IDs represent the same underlying business action.

An important accounting design choice: if the authoritative ledger and payment state live in the same transactional database, write both in the same local transaction when appropriate. If ledger posting is asynchronous, explicitly monitor and reconcile the gap between payment confirmation and ledger posting; do not imply the outbox itself makes the ledger instantly consistent.

9. Payment state, ledger, and settlement are different

These concepts are often conflated in system design interviews.

Record	Question it answers	Typical property
Payment state	What is the workflow's current status?	Guardedly mutable
Audit history	What changed, when, and by which actor/system?	Append-only or tamper-evident
Financial ledger	Which monetary obligations or movements have been recorded?	Append-only journal with corrections as new entries
Provider settlement report	What did the provider actually settle, deduct, or transfer?	External reconciliation evidence

A financial ledger commonly uses double-entry bookkeeping: each journal transaction contains balanced postings across accounts under the chosen accounting model. A ledger entry is not automatically proof that funds have settled in the bank account.

For example, an authorized card payment is not the same as a captured payment; a captured payment is not necessarily already settled to the merchant. Provider fees, reserves, disputes, reversals, and payout timing may affect reconciliation.

Avoid saying that a payment.succeeded event means cash is already in the merchant's bank account. What it means depends on the provider event and payment method.

10. Authorization, capture, void, and refund

Not every payment should be charged immediately.

Immediate charge/capture can suit purchases where fulfillment and amount are known upfront.

Authorize then capture can suit hotels, rentals, or orders where inventory or final amount needs confirmation. An authorization generally reserves or approves funds subject to network/provider rules; capture is a later step requesting the charge. Authorizations can expire, and partial/multiple captures are provider- and payment-method-dependent.

Void/cancel typically applies before capture or while an operation remains cancellable. Refund is a separate operation after a charge/capture, subject to provider rules and payment state.

Refunds need their own IDs, attempts, idempotency identities, statuses, and reconciliation. A refund timeout is ambiguous for the same reason a charge timeout is ambiguous.

For partial refunds, protect the available refundable amount against concurrent requests. A safe local reservation/transaction must account for successful and still-in-flight refund amounts, not merely the sum of completed refunds, otherwise two concurrent provider calls can each appear individually valid and together exceed the captured amount. Provider-side refund limits remain an additional protection, not a substitute for local coordination.

Refund ledger postings should be new entries, not edits that erase the original charge history.

11. Reconciliation is a core subsystem, not an emergency script

Webhooks are not guaranteed to arrive promptly or exactly once. Provider APIs and internal databases can temporarily disagree. Reconciliation detects and resolves the difference.

Operational reconciliation checks in-flight, stale, and unresolved operations against provider APIs or other supported status channels. It can identify a charge that succeeded externally but never updated the internal state.

Financial reconciliation compares internal financial records against provider reports, settlement exports, fees, payouts, and adjustments. It can reveal mismatched amounts, missing transactions, unexpected duplicates, refunds, and reversals that payment-status polling alone would not catch.

A production reconciliation job should have:

Stable provider and internal reference mappings.
Bounded, restartable batches with checkpoints or watermarks.
Rate-limit-aware status lookups.
Idempotent repair operations with durable action IDs.
Exception queues for unresolved mismatches requiring investigation.
Alerts for aging unresolved payments, ledger gaps, and settlement differences.

Do not automatically trust whichever record has the newest timestamp. Interpret the provider's event semantics and the settlement evidence, then record corrections without silently rewriting financial history.

12. Multi-provider routing has a double-charge trap

A second provider can improve availability, but failover is not simply retrying elsewhere.

Provider A: request timed out, but charge actually succeeded
Provider B: automatic failover charge also succeeds
Result: customer may be charged twice

Different providers generally cannot deduplicate each other's charges using a shared idempotency key.

Only route a new independent attempt when the previous attempt is known not to have created a charge, or when a specifically designed orchestration and recovery protocol makes that decision safe. Keep each provider attempt separately identifiable under the same logical payment.

Circuit breakers, rate limits, exponential backoff, and jitter protect capacity, but none of them resolves an already ambiguous financial outcome. A 5xx is not universally proof that a provider performed no side effect; follow that provider's error semantics.

13. Security and compliance boundaries

For most integrations, prefer provider-hosted payment collection or tokenization so your application avoids directly handling raw card details wherever possible.

Tokenization can reduce PCI DSS scope, but does not automatically remove all compliance obligations. Exact scope depends on how the integration works, what systems can affect the payment page, and the applicable standards and assessments.

Keep payment API authorization strict: a user must not retrieve or refund another merchant's payment by guessing an identifier. Verify webhook authenticity, prevent replay where the provider supports it, protect credentials, limit sensitive data in logs, and audit privileged refund and payment-data access.

Never store card verification values after authorization; any handling of cardholder data must follow the applicable payment security rules. Payment references, personal information, and webhook payloads can also be sensitive even when they are not raw card numbers.

14. Scaling without weakening correctness

Payment APIs can be horizontally scaled because authoritative state belongs in durable systems, not process memory. The harder bottlenecks tend to be database contention, provider quotas, webhook throughput, and reconciliation backlog.

A pragmatic progression:

Start with a strongly consistent transactional store for payment identity, idempotency, attempts, and guarded state changes.
Add appropriate indexes for merchant/payment lookup, unresolved attempts, provider references, and reconciliation scheduling.
Scale webhook processing, outbox relay, and downstream workers independently with bounded concurrency and backpressure.
Use replicas for reporting and stale-tolerant reads, not for authoritative decisions such as whether a charge already exists.
Partition or shard only when measured bottlenecks justify the complexity; preserve uniqueness and ownership invariants across partitions.

Multi-region payment writes require especially careful ownership and fencing. If two regions can simultaneously accept the same idempotency key without a shared uniqueness guarantee, each may initiate a separate provider charge. A failover design must also preserve or recover in-flight attempt identities, not just database rows.

Operational metrics should include provider timeouts by operation, ambiguous-attempt age, idempotency conflicts, webhook verification failures and processing lag, outbox age, refund reservation backlog, reconciliation mismatches, and ledger/settlement discrepancies. Ordinary request latency alone cannot tell you whether a payment system is financially correct.

15. Walk through the failures that matter

Failure	Unsafe reaction	Safer design
Client retries after timeout	Create a new charge	Reuse logical payment and provider attempt identity
Two concurrent payment requests	Both pass a cache check	Atomic durable idempotency constraint
Provider response is lost	Mark payment failed	Preserve unresolved state; query, safely retry, or reconcile
Webhook delivered twice	Reapply financial effects	Durable event/business-operation deduplication
Older webhook arrives late	Overwrite latest state	Guarded transitions and provider semantics
DB commits but event publish fails	Lose downstream confirmation	Transactional outbox
Refund requests race	Each sees old refundable total	Transactionally reserve refundable amount
Provider A times out	Immediately charge at Provider B	Resolve A before independent failover
Payment recorded but ledger posting missing	Assume reporting will fix itself	Durable delivery, idempotent posting, monitored reconciliation
Primary region fails	Accept writes everywhere	Explicit write ownership, fencing, recovery of ambiguous attempts

For a fuller production failure analysis, see Payment System — Failure & Scale Scenarios.

16. The final architecture

                  Customer / Checkout
                           |
                           v
                     Payment API
                           |
                           v
                 Payment Orchestrator
                    /      |       \
                   /       |        \
          Payment DB   Provider      Status API
          + Attempts   Adapter           |
          + Idempotency   |              |
          + Outbox        v              |
          + Ledger*    External PSP <----+
                |          |
                |          v
                |     Webhook Endpoint
                |          |
                |     Verified Durable Inbox
                |          |
                |     Idempotent Worker
                |          |
                +<---------+
                |
             Outbox Relay ---> Broker ---> Order / Notification
                |
          Reconciliation Jobs <---- Provider API / Reports

* Ledger may be in the same transactional boundary or a
  separately operated financial subsystem, with explicit
  consistency and reconciliation guarantees.

Every component exists for a reason:

Payment orchestrator: one logical payment and guarded state transitions.
Durable idempotency: duplicate client requests cannot create independent logical payments within the key's contract.
Provider adapter: normalizes provider-specific operations and outcome semantics.
Webhook inbox: authenticated, durable, duplicate-tolerant asynchronous updates.
Outbox: durable handoff to downstream business processes.
Ledger: auditable financial postings, distinct from workflow status.
Reconciliation: finds and resolves external/internal discrepancies.

The architecture is not meant to imply that all components must be separate microservices. Several can begin as modules within one well-structured service and database.

17. How to explain this in a system design interview

A strong payment-system answer can follow this order:

Clarify the payment method, synchronous versus asynchronous outcomes, authorization/capture, refunds, and provider count.
Define logical payment identity, attempt identity, money representation, and the state machine.
Walk through the happy path, persisting the attempt before the provider call.
Demonstrate duplicate-request handling using durable idempotency.
Introduce timeout ambiguity, safe provider retries, webhooks, and reconciliation.
Close the database-to-queue gap with an outbox and idempotent consumers.
Separate operational payment state from financial ledger and settlement.
Discuss refunds, multi-provider failover, security, and scaling only to the depth the interview requires.

The strongest answers do not promise “exactly-once payments” without qualifications. They explain which boundary enforces which invariant, where external uncertainty remains, and how discrepancies are detected and corrected.

Explore the deeper decisions in Architecture Deep Dive, compare options in Trade-offs & Decisions, and practice the follow-up questions in Interview Perspective.

Final takeaway

A payment system is not correct merely because its happy path succeeds. It is correct to the extent that it can preserve durable identity, avoid duplicate external effects, represent uncertainty honestly, recover from crashes, and reconcile what it believes happened against what the provider and financial records show.

The question is not only “Did the API succeed?” It is “What financial effect actually occurred, how do we know, and what happens when we do not know yet?”

For the complete visual and technical learning path, explore Payment System Design on SeeItFlow.

Distributed Task Scheduler System Design: Jobs, Retries, Leases, and Failures

MANGESH MANDLIK — Tue, 13 Oct 2026 05:09:43 +0000

Distributed Task Scheduler System Design: What Happens When a Job Is Due?

A user schedules a payment for 9:00 AM. At 9:00, the scheduler finds the job and hands it to a worker. The worker calls the payment provider, the charge succeeds, and then the worker crashes before recording success.

What should happen next?

The scheduler cannot simply forget the job: it has no durable confirmation that execution finished. But blindly running it again might charge the customer twice.

That tension is central to distributed task scheduler system design. Finding work at the right time is only the beginning. A production scheduler must preserve jobs across failures, coordinate concurrent schedulers, handle retries without inventing exactly-once guarantees, and make recovery safe even when the outcome of an earlier attempt is unknown.

This article designs a distributed job scheduler for one-time and recurring tasks, with the focus on the correctness boundaries that matter in production.

First, define what the scheduler promises

A task scheduler answers when work becomes eligible to run. It does not inherently guarantee that a worker starts at an exact wall-clock instant, that a remote API succeeds, or that an external side effect happens exactly once.

A useful contract separates four timestamps:

due_at: the earliest instant a job is eligible to run.
dispatched_at: when the system hands an execution request to its delivery mechanism.
started_at: when a worker begins the attempt.
completed_at: when the system durably records a terminal outcome.

A 9:00 AM job may begin after 9:00 because of polling, queueing, worker capacity, or an outage. The product should define an acceptable scheduling lag and whether a late job should still execute. For workloads with hard deadlines, expiry is a separate policy: running a stale trade or promotion after its validity window may be worse than skipping it.

We'll assume jobs can be scheduled once or on a recurrence, cancelled or rescheduled before execution, retried for eligible failures, and queried for status. We will not assume every job is safe to repeat.

The basic architecture

Client / API
     |
     v
Durable Job Database  <----- Recurrence Generator
     |                       (creates occurrences)
     v
Due-Job Discovery / Claimers
     |
     | atomic DB transaction:
     | claim + dispatch-outbox record
     v
Transactional Outbox
     |
     v
Outbox Relay -----> Durable Queue -----> Workers
                                           |
                                           v
                                    Business Side Effect
                                           |
                                           v
                                  Status / Attempt Store

Lease Reaper ------> expired attempts / safe retry decisions

The database owns the authoritative job state. The queue transports work to workers; it does not replace the scheduler's lifecycle record in this design. The outbox bridges the database-to-queue failure gap. A lease reaper finds attempts that stopped making progress.

This is one reasonable architecture, not a mandatory stack. A smaller scheduler can begin with indexed database polling and workers claiming rows directly. Some workloads use broker-native delayed delivery; others add Redis sorted sets or timing wheels for due-time discovery. Each alternative has different limits and failure semantics.

1. Store jobs durably before promising acceptance

A minimal model might look like this:

jobs
  job_id
  tenant_id
  payload_ref
  due_at
  status
  version
  creation_idempotency_key
  deadline_at

attempts
  attempt_id
  job_id
  generation
  status
  lease_expires_at
  started_at
  completed_at
  error_category

outbox
  event_id
  attempt_id
  payload
  published_at

For recurring work, add a schedule definition containing the recurrence rule, IANA timezone, misfire policy, overlap policy, and the next occurrence to generate. Each generated occurrence gets its own job identity.

The creation API should make a durable write before returning an unqualified success response. If a client times out after the commit, a stable creation idempotency key and a uniqueness constraint let it retry without creating a second logical job.

Job status is not the same as an attempt's status. A job may have multiple attempts; one attempt can fail transiently while the logical job remains eligible for retry. Keep enough attempt history to explain what happened without treating an old attempt's completion as permission to overwrite newer state.

2. Discover due jobs without scanning everything

The simplest viable due-work detector is a database index supporting a query shaped like:

SELECT job_id
FROM jobs
WHERE status = 'SCHEDULED'
  AND due_at <= CURRENT_TIMESTAMP
ORDER BY due_at, job_id
LIMIT 500;

The relevant index, bounded batch size, and query plan matter more than the particular batch size shown here. In a multi-scheduler design, merely selecting rows is not sufficient: two instances could read the same jobs. A claim must be atomic and conditional on current state, commonly using row locks such as FOR UPDATE SKIP LOCKED inside a transaction or an equivalent compare-and-swap transition.

SKIP LOCKED is a useful mechanism, not a universal requirement. Another design gives one elected leader ownership of due-time discovery. At higher scale, ownership can be sharded so independent schedulers process different partitions. Leader-based systems need fencing: a leader whose lease expired may resume after a pause and must not be allowed to make stale authoritative writes.

A Redis sorted set can speed up discovery when database due-time queries become a measured bottleneck. But if Redis is only an index, its loss must not erase jobs: rebuild it from the durable store, validate candidate jobs against authoritative state, and reconcile missing entries. A timing wheel similarly needs a durable backstop if its in-memory state can disappear.

3. The database-to-queue dual-write trap

Consider this dispatch sequence:

Update a job to QUEUED in the database.
Publish an execution message to Kafka, SQS, or another broker.

If the process crashes between steps, the database says the job was dispatched, but the broker never received it. Reversing the order creates the mirror failure: the broker receives a message, but the database update fails, so a retry may publish it again.

A transactional outbox handles the first problem without pretending the database and broker share one atomic transaction. In one database transaction, the scheduler records the job/attempt transition and inserts an outbox event. A relay later publishes that event to the broker and records publication progress.

The relay itself can crash after publishing but before marking the event published. It may then publish the same event again. Thus the outbox supports eventual, at-least-once publication, assuming the relay and broker recover and retry successfully; it does not make downstream execution exactly once.

Messages need stable identities. Repeated delivery of the same dispatch event should not create an unrelated business operation. The worker checks authoritative job and attempt state before claiming execution, and downstream effects still require their own idempotency protections.

4. Worker leases handle crashes, not exactly-once effects

A worker claims an attempt with a lease. While running, it renews the lease if the design supports long-running work. If the worker dies, a separate reaper can detect the expired lease and make the job eligible for another attempt.

But lease expiry is not proof that the old worker stopped. A long garbage-collection pause, lost heartbeat, or network partition can cause a live worker to lose its lease while it continues processing. Another worker may start at the same time.

To protect scheduler state, attach an increasing attempt generation or fencing token to conditional writes. A stale worker must not be able to mark a superseded attempt successful or overwrite the current state. Fencing protects only resources that actually validate it; it cannot retroactively undo a payment sent to an external provider.

The lease duration and heartbeat cadence must be chosen against real pause times, execution durations, and acceptable recovery latency. There is no universally correct TTL or heartbeat ratio.

5. Retry correctness depends on where the side effect happens

Return to the payment example. Worker A successfully charges the customer but crashes before saving the result. Worker B eventually receives a retry. From the scheduler's local records, the outcome is ambiguous.

A database flag saying started is not enough. If a worker writes that flag and crashes before the charge, treating the flag as proof of completion silently loses the payment. A completion marker written before an external effect has the same flaw.

There are three importantly different cases:

Local database mutation. If the business change and completion record live in the same database, commit them in one transaction, with appropriate uniqueness or conditional constraints. Either both commit or neither does.

External API supporting idempotency. Send a stable business-operation idempotency key that survives retries and worker replacement. The provider's documented deduplication semantics and retention window determine what guarantee this actually gives. An internal attempt ID is usually the wrong key because it changes across attempts.

External API without reliable idempotency. A timeout cannot reveal whether the operation happened. Durable pending state, provider-side lookup where available, and reconciliation may resolve uncertainty. If the provider offers neither deduplication nor reliable outcome lookup, the scheduler cannot promise exactly-once external effects. The product must explicitly choose how to handle that residual risk.

This is why at-least-once execution attempts and exactly-once business effects are not interchangeable claims.

6. Retry policy is more than exponential backoff

Retries begin with failure classification:

Transient failures, such as some timeouts or temporary service unavailability, may be retried if the operation is safe to retry.
Rate-limit responses require respecting the provider's applicable retry guidance and quotas; an HTTP 429 is not permission to retry immediately.
Permanent validation or business-rule failures generally should not be retried unchanged.
Ambiguous outcomes require idempotency or reconciliation, not optimistic assumptions.

Use bounded attempts, capped exponential backoff with jitter, execution deadlines, and a dead-letter or terminal-failure path. Backoff spreads attempts over time, but does not by itself cap aggregate load on a struggling dependency. A dependency-scoped rate limit or retry budget, and circuit breaking when appropriate, prevent thousands of independently reasonable retries from combining into a retry storm.

A dead-letter queue is not a garbage bin. Preserve job identity, failure reason, and attempt history. Replays should be authorized, observable, rate-limited, and idempotent. A replay may create a new attempt, but it must not invent a new business operation identity.

7. Recurring schedules need separate occurrence identities

“Every day at 9:00 AM” is a schedule definition, not one job repeatedly moved back to SCHEDULED.

Generate individual occurrences, each with a deterministic identity based on the schedule and nominal occurrence. A unique constraint ensures two generators recovering from a failover do not create duplicate logical occurrences.

For a recurring schedule, store the recurrence rule and an IANA timezone, such as Asia/Kolkata or America/New_York. A fixed UTC offset is not sufficient for local-time recurrences across daylight-saving changes. Every occurrence must be resolved using the timezone rules applicable to that date.

Two edge cases need explicit product decisions: during spring-forward, a configured local time might not exist; during fall-back, it might occur twice. Decide whether to skip, shift, or select a particular occurrence rather than relying silently on a date library's default.

After downtime, decide whether to skip missed runs, catch up all valid runs, or coalesce missed runs. For billing, every period might matter; for a transient reminder, replaying all missed notifications could be actively harmful. Also specify whether successive occurrences may overlap, should queue behind one another, or should skip while an earlier occurrence is still running.

8. Cancellation is a race, not just a status update

Cancelling a scheduled job before it is claimed is relatively straightforward. Cancelling a queued job is harder because its message may already be in transit. A worker should check authoritative state and the dispatched version before beginning work; a stale message must not bypass a recorded cancellation.

That check still does not make cancellation of running external work instantaneous or reversible. Cancellation may be cooperative, and there can be a race after the final pre-execution check. A precise API distinguishes cancellation requested, cancelled before execution, and already running or completed. Never promise that a successfully returned cancellation can undo an email already sent or a charge already made.

Rescheduling also changes job version and due time; old queued messages must not cause execution against an obsolete schedule.

9. What happens at the 9:00 AM burst?

Imagine millions of jobs become due at the same moment. It is tempting to prescribe Redis or a bigger queue immediately. Neither is a universal answer.

The first bottleneck might be the due-time index, claim contention, outbox relay, broker ingress, worker concurrency, network egress, or a third-party API's quota. Measure due-to-dispatch lag, queue waiting time, worker start lag, oldest overdue job age, retry rate, and downstream throttling to identify where capacity is actually exhausted.

Bound due-job scans, partition hot scheduling ranges when needed, enforce per-tenant quotas and fairness, and pace dispatch to sustainable downstream throughput. Predictable bursts may justify pre-scaling workers.

Pre-staging work before its due time can help only if execution is still gated by eligible_at. Starting a 9:00 AM payment at 8:59 merely to flatten the graph is a correctness regression unless early execution is explicitly allowed.

A backlog is not necessarily data loss. But a scheduler that keeps accepting jobs faster than they can execute needs an explicit lateness and admission-control policy; a queue cannot manufacture downstream capacity.

10. Recovery paths worth designing before production

Scheduler crashes. Durable due jobs remain discoverable. Another claimer or a fenced new leader takes over, then performs a bounded catch-up scan rather than dispatching every overdue job at once.

Database primary fails. Job creation and authoritative transitions may become unavailable. Reject uncommitted new requests explicitly instead of pretending an in-memory buffer is durable. Replica promotion's data-loss exposure depends on replication acknowledgments and actual lag, not a universal recovery number.

Broker is unavailable. Outbox events remain durable and publication lag rises. Once the broker returns, drain the outbox at a controlled rate.

Worker crashes or an ACK is lost. A lease expiry or broker redelivery may cause another attempt. That is expected at-least-once behavior; side-effect idempotency or reconciliation is the safety boundary.

A leader resumes after losing ownership. Conditional state transitions and fencing reject stale authoritative writes. Merely holding a time-limited lock is insufficient.

A region fails. A standby region can take over only under a defined ownership and fencing protocol, using whatever state was durably replicated. Recovery time and potential data loss are properties of the chosen replication and failover design, not free guarantees of “multi-region.”

11. The trade-offs that change the design

Decision	Simpler starting point	When a different approach earns its cost
Due-time discovery	Indexed database polling	Redis index, timing wheel, or sharding when measured due-discovery limits justify them
Scheduler coordination	Atomic competing DB claimers	Leader election or shard ownership for particular contention and operational requirements
Dispatch	Direct database-backed worker claims for a small design	Queue + transactional outbox for decoupled, durable, high-throughput handoff
Worker recovery	Fixed leases for bounded short jobs	Renewable leases and checkpoints for long or variable-duration jobs
Missed recurrences	Explicit per-schedule misfire policy	Different catch-up behavior by workload, not a universal replay-all rule
Regional topology	One authoritative region with standby	More complex cross-region ownership only when availability needs justify it

The goal is not to collect the most distributed components. It is to add mechanisms that solve a demonstrated bottleneck or close a specific correctness gap.

A practical way to reason through the interview

Start by asking what “run at 9:00” means, how late jobs may be, whether retries and overlap are acceptable, and which external side effects cannot safely repeat. Establish peak due-job rate and burstiness as assumptions, not universal scale thresholds.

Then evolve the design in order: durable jobs; indexed due discovery and atomic claiming; a dispatch boundary; worker leases; retry classification; idempotency at the business-effect boundary; and failure recovery. Add an outbox when dispatch crosses a database/broker boundary. Add a fast due-time index or sharded ownership only when its benefits outweigh the consistency and operational costs.

A strong system design explanation follows one job through an actual failure. The most revealing question is often not “how do you schedule a million jobs?” but “what do you do when the external effect succeeded and your worker cannot prove it?”

The essential takeaway

A distributed task scheduler is a durable coordination system under uncertain execution outcomes. Its clock determines eligibility, its store preserves intent, its dispatcher delivers work, and its workers perform effects. Those boundaries can fail independently.

Reliable scheduling therefore comes from a combination of atomic claims, durable dispatch, leases and fencing, bounded retries, explicit recurrence semantics, and idempotency or reconciliation at the point where the business effect actually occurs. None of these alone provides a universal exactly-once guarantee.

For the full visual and production-oriented treatment, explore Distributed Task Scheduler — Complete Design, Architecture Deep Dive, Trade-offs & Decisions, and Failure & Scale Scenarios on SeeItFlow. The Visual Design Journey walks through the design's evolution interactively.

WhatsApp System Design: Building a Reliable Messaging Platform

MANGESH MANDLIK — Mon, 12 Oct 2026 13:52:31 +0000

A chat message is easy to send when both users are online. The difficult part is ensuring that one logical message survives retries, reaches multiple devices, appears in a consistent order, and remains recoverable when connections or services fail.

Imagine Alice sends Bob a message. Bob's laptop is connected, but his phone is offline. The server accepts and durably stores the message, then its acknowledgement to Alice is lost. Alice's app cannot tell whether the send succeeded, so it retries. Meanwhile, Bob's laptop receives the message, and his phone will reconnect hours later.

What should happen? Alice must not create a second logical message by retrying. Bob's laptop should not display duplicates. Bob's phone should eventually catch up, subject to the system's retention and access policies. And the sender's sent, delivered, and read indicators must not pretend to mean the same thing.

That is the real challenge behind designing a WhatsApp-like messaging platform. This article develops an illustrative architecture for a real-time chat system; it does not claim to describe WhatsApp's private implementation.

1. Requirements before architecture

Start with one-to-one messages, group conversations, low-latency delivery to connected clients, offline recovery, multi-device synchronization, delivery and read receipts, and media attachments. Presence and typing indicators improve the experience, but they are not part of the durable-message guarantee. Broadcast channels, message editing, and end-to-end encryption introduce additional design constraints that we will address separately.

The critical non-functional requirements are:

Durability: an acknowledged send has satisfied a clearly defined durable-write contract, even if the recipient is offline.
Idempotency: retrying the same logical send does not create another accepted message.
Ordering: messages have a consistent order within a conversation, without requiring one global order for the entire platform.
Recovery: disconnected devices can reconcile missed changes from durable state.
Scalability: socket connections, message writes, history reads, group delivery, and media bytes can scale independently.
Security: authenticate devices, authorize conversation operations, and define encryption and retention policies explicitly.

We should not promise unconditional instant delivery, exactly-once network transmission, or zero data loss under every possible disaster. The guarantees depend on the storage write policy, failure model, retention window, and client behavior.

A scale model, not a claim about WhatsApp

For a sizing exercise, assume 500 million daily active users, 200 million concurrent connections, and 100 billion messages per day. That is approximately 1.16 million messages per second on average. If the effective stored message record averages 1 KB, the raw logical ingest is about 100 TB/day, or 3 PB over 30 days, before replication, indexes, attachments, and other overhead. Peaks may be substantially higher.

These are illustrative assumptions, not published operating figures. They tell us to plan separately for persistent sockets, sustained writes, fan-out amplification, reconnect bursts, and storage growth. A hypothetical 50,000 sockets per gateway would imply about 4,000 gateways for 200 million connections before spare capacity; actual density depends on workload, memory, TLS, runtime, network throughput, and failure headroom.

2. Start simple, then find the breaking point

A small chat application can start with clients connecting to an application server backed by a relational database. The server authenticates requests, writes messages, and forwards them to connected recipients.

Alice ──> App Server ──> Message Database
                │
                └──────> Bob (if connected)

This works at modest scale. But once users are connected to many application instances, Alice's server cannot assume Bob is connected to the same one. Long-lived sockets consume a different kind of capacity from database writes. Offline delivery and group messaging also need durable recovery rather than in-memory queues.

The next architecture separates connection management, message acceptance and storage, and delivery/synchronization. These boundaries solve different problems; they are not microservices added merely for appearance.

3. WebSocket gateways and connection routing

WebSocket is a practical default for interactive chat because it supports persistent bidirectional communication. The client can send messages and acknowledgements while the server pushes new events over the same connection.

Server-Sent Events plus HTTP requests can work for primarily server-to-client applications; long polling is another compatible fallback. Neither is inherently invalid, but frequent bidirectional chat tends to favor WebSocket. HTTP/2 Server Push is not the relevant modern alternative.

Alice's phone ──> Load Balancer ──> Gateway A ──┐
                                                │
Bob's laptop ──> Load Balancer ──> Gateway B ────┼──> Messaging Service
                                                │           │
Bob's phone (offline)                           │           v
                                                │      Message Store
                  Connection Registry <────────┘

A gateway authenticates the device session, maintains heartbeats and bounded socket buffers, and registers its route. The connection registry conceptually maps:

user_id -> {
  device_id/session_id -> gateway_id, connection_id, session_epoch
}

The registry is ephemeral routing information, not the source of truth for messages. One user can have several devices and several sessions. A stale registry entry may send a delivery attempt to a dead connection; the gateway can reject it, the routing layer can re-check the route, and durable sync remains the fallback.

Session epochs or equivalent generation identifiers prevent old sockets from silently replacing newer connections. Gateways must also drain connections during deployments and bound their outbound queues. If Bob's laptop stops reading, retaining unlimited pending messages in gateway memory is not reliable delivery; it is an eventual memory incident. Disconnect the slow session when necessary and let it recover from durable storage.

4. Define the message-send contract

Alice generates a stable client_message_id before her first send attempt:

{
  "type": "message.send",
  "conversation_id": "conv-42",
  "client_message_id": "alice-phone-unique-123",
  "content_type": "text",
  "payload": "Hello Bob"
}

The server authenticates Alice, checks that she can send to the conversation, validates the request, and atomically resolves the client-generated request identity to one canonical server message ID and one conversation-order position. It then commits the message according to its durable-write contract and returns an acceptance acknowledgement.

The sender acknowledgement means the server durably accepted the logical message. It does not mean Bob's device received it, the message was read, a push notification succeeded, or every downstream fan-out task finished.

The idempotency mapping might be scoped by (sender_id, sender_device_id, client_message_id); its precise scope and retention policy are part of the protocol. Retrying with the same ID but different content should be rejected or handled under an explicitly defined policy, not silently treated as a new message.

The lost-ACK problem

Alice                     Messaging Service              Durable Store
  |  SEND (client_id=123)        |                            |
  |----------------------------->|--- atomic accept/write --->|
  |                              |<--------- committed -------|
  |   ACK is lost   X <----------|
  |                              |                            |
  |  RETRY (client_id=123)       |                            |
  |----------------------------->|--- resolve same identity -->|
  |<--- original message_id -----|                            |

A naïve SELECT followed by INSERT is insufficient: two concurrent requests can both observe that the ID does not exist. Use a uniqueness constraint, conditional write, transaction, or equivalent atomic mechanism. The deduplication record, canonical message identity, and ordering decision must be coordinated so a crash does not create two accepted versions of the same logical send.

This is exactly-once logical acceptance, under the defined idempotency scope and retention assumptions—not exactly-once packet delivery. Network sends, event deliveries, and client replays can still happen more than once.

5. Ordering is per conversation, not global

If Alice and Bob send at nearly the same time, their devices may observe different network arrival orders. Timestamps are not a reliable strict-order mechanism because clocks differ and concurrent requests race.

For this design, an ordering authority assigns each accepted message a monotonically increasing conversation sequence:

Conversation conv-42
  seq 418 -> Alice: Hello
  seq 419 -> Bob: Hi
  seq 420 -> Alice: Are you free?

Possible implementations include an ordered log partition, a single conversation owner/sequencer protected by fencing, or an appropriate storage-side conditional allocation mechanism. Consistent hashing can help route requests to an owner, but routing to the same instance is not proof of exclusive write authority when instances fail or ownership changes.

Strict contiguous numbering is a stronger promise than merely increasing order. If we promise contiguous committed sequences, the sequence allocation and durable acceptance protocol must not leave permanent unexplained holes after crashes. An implementation can instead define visible committed order separately from provisional reservations. The important point is to state which contract the client may rely on.

Suppose Bob receives sequence 420 before 419. His client tracks the highest contiguous, safely applied sequence, buffers newer events within limits, and requests the missing range. It should not advance its checkpoint past an unresolved gap simply because 420 arrived. If targeted repair fails, it can fetch conversation history or reconcile through device sync.

6. Persist first; deliver afterward

The central write path is:

Client send
   |
   v
Authenticate + authorize + deduplicate
   |
   v
Assign canonical identity and conversation order
   |
   v
Durable message acceptance
   |                    \
   v                     v
Sender ACK          Durable delivery/sync work
                              |
                     Online device routing
                              |
                     Offline catch-up / push hint

A successful write must not be followed by a best-effort event publish that can disappear forever if the process crashes. If delivery workers or the sync feed depend on an event stream, use a transactional outbox, storage-derived change feed, or another recoverable handoff with appropriate atomicity and replay guarantees. Otherwise the database can contain an accepted message that downstream systems never learn about.

The event bus might be Kafka, but Kafka is not mandatory for every messaging architecture. It can handle asynchronous group fan-out, notification work, receipt propagation, and operational events. If a feature relies on Kafka, an outage delays that feature. If Kafka is on the synchronous acceptance path, its availability also affects sender ACKs. Both designs are possible; the dependency boundary must be explicit.

Consumers should tolerate at-least-once event processing, maintain idempotent effects, and track lag. In-memory producer buffers alone do not provide durable outage recovery.

7. Offline delivery and multi-device sync

Bob's laptop is online, so the system can attempt immediate delivery to that session. Bob's phone is offline, so there is no usable live route. Neither condition changes whether Alice's message was durably accepted.

When Bob's phone reconnects, it authenticates and presents its own durable sync cursor. The server returns changes after that checkpoint: messages, membership changes, relevant receipts, and other supported conversation events. The device applies each batch safely, deduplicates by stable identity, and advances its checkpoint only after successful local application.

Bob laptop: cursor 847  ── receives new event ──> 848
Bob phone:  cursor 842  ── offline
Bob phone reconnects ── fetch events after 842 ──> apply ──> 848

A cursor can represent a device-scoped ordered change feed rather than the sequence number of any one conversation. Conversation sequence and device sync cursor are different ordering domains. They should not be treated as interchangeable integers.

Realtime pushes and mobile notifications are wake-up hints, not authoritative history. A missed WebSocket event or failed APNs/FCM notification must not prevent the next sync from discovering the accepted message.

A device offline beyond the retained incremental change history may have an expired cursor. The server must explicitly require a snapshot or full reconciliation path, respecting deletion, membership, and retention semantics. No design can promise recovery of content already removed under a valid retention or deletion policy.

8. Sent, delivered, and read are separate events

A useful state model is:

CLIENT_PENDING -> SERVER_ACCEPTED -> RECIPIENT_DELIVERED -> READ

These labels are product semantics, not automatic network guarantees.

In this illustrative design:

Server accepted: the canonical message has passed the durable write contract.
Delivered: at least one eligible recipient device has acknowledged that the app received and safely accepted the message locally.
Read: a recipient action or client state meets the product's defined read criterion.

A gateway successfully writing bytes to a socket is not sufficient evidence that the recipient application processed them. Likewise, delivered-to-one-device and delivered-to-all-devices are different possible products. We choose the former for this example and track richer per-device state internally if needed.

Read state can often be represented by a monotonic read_up_to_sequence for each user and conversation, provided the product defines what that high-watermark means. Large groups may aggregate, restrict, or omit detailed per-member read receipts because otherwise receipt traffic can overwhelm message traffic.

Receipt events can be late or duplicated. Processing must be idempotent and must not regress a valid later state to an earlier one.

9. Group messaging changes the cost model

One-to-one delivery has a small number of recipients. A group send may need to notify hundreds or thousands of devices; a broadcast channel may have an audience orders of magnitude larger.

Two common distribution strategies are:

Write fan-out: create recipient-oriented delivery or inbox work when a message is accepted. This makes individual catch-up and notifications convenient but amplifies writes with audience size.

Read fan-out: store the message once in a canonical conversation or channel log, then let members fetch it. This reduces per-message write amplification but shifts cost toward reads, indexes, and caching.

A hybrid design often makes sense: a canonical conversation log plus recipient-oriented sync indexes for interactive chats, while very large broadcast channels use store-once, notify-and-fetch behavior. There is no universal group-size cutoff that decides the architecture.

Group membership is also a correctness problem. If Carol is removed while Alice sends a message, which membership version applies? Bind acceptance and fan-out to a defined membership snapshot or equivalent serialized rule. Otherwise different workers may produce different recipient sets for the same message. Encryption-enabled groups add key-management and membership-epoch concerns on top of that authorization decision.

The sender's ACK should not wait for thousands of devices to acknowledge delivery. Durable acceptance and downstream group fan-out are separate operations, with explicit lag monitoring and replay behavior.

10. Store messages for ordered reads and bounded partitions

The durable message store needs high write throughput, efficient history pagination, and access by conversation and sequence. One possible physical layout is:

Partition key: (conversation_id, bucket_id)
Sort key:      message_sequence
Value:         message_id, sender_id, payload/ciphertext, metadata

Buckets may use sequence ranges, time windows, or bounded-size policies. This avoids an indefinitely growing physical partition for a long-lived conversation, though bucketing does not automatically fix a hot current bucket. Very active conversations can still overload their current storage replicas or ordering authority.

A relational database can be suitable at modest scale, particularly when transactional requirements dominate. At extreme append volume, a wide-column or distributed key-value store may better match partition-local history reads. Cassandra, ScyllaDB, DynamoDB, and relational systems have different conditional-write, indexing, transaction, and consistency characteristics; choosing a brand does not itself prove the protocol correct.

The canonical conversation log and a per-user or per-device sync index are different views. If a sync index is derived asynchronously, the durable handoff and replay path must ensure that acknowledged messages eventually become discoverable by authorized devices, subject to retention and availability constraints.

11. Presence, typing, and push notifications are not durable messages

Presence is usually derived from active sessions with heartbeat-refreshed leases or TTLs. It is approximate, affected by mobile background behavior, and subject to privacy rules. If any permitted active session is live, a product may show a user as online—but this is a policy, not a universally accurate fact.

Typing indicators are short-lived events that can be coalesced, rate-limited, or dropped under load. Persisting a backlog of stale typing events adds cost without value.

Mobile push notifications are also best-effort wake-up mechanisms. They may be delayed, duplicated, suppressed, or unavailable. A push payload must respect privacy and, where enabled, end-to-end encryption constraints. The client still fetches authoritative state from the messaging service after waking.

These workloads should degrade before the system sacrifices the integrity of already accepted messages.

12. Media messages take a separate byte path

Sending a photo should not mean streaming megabytes through the same gateway buffers that handle text messages and receipts.

Sender ── authorized upload ──> Media Service / Object Storage
   |
   └── message.send(media reference) ──> Message Service
                                          |
Recipient <── authorized download / CDN <─┘

The client uploads the media through an authorized, potentially resumable object-storage path. The message contains a reference and the metadata needed for retrieval. The service must define when that reference is considered valid, how orphaned uploads are cleaned up, and how access is checked on download.

For private content, a CDN requires its own appropriate access-control strategy; an object-storage presigned URL is not automatically a secure CDN authorization scheme. Media can fail independently of text chat, and clients should surface that failure rather than treating a missing upload as a successfully delivered attachment.

13. End-to-end encryption changes the trust model

End-to-end encryption (E2EE) means the service should not possess the plaintext keys needed to read protected message content under the chosen threat model and protocol. Servers still need to authenticate users and devices, route ciphertext, store encrypted payloads, coordinate device enrollment, and manage operational metadata.

Multiple devices and groups make key distribution more complicated. Adding or removing a device, changing group membership, and restoring backups all need explicit cryptographic policies. Encryption also constrains server-side search, moderation, and content-derived analytics.

E2EE is not achieved merely by storing encrypted strings in a database or enabling TLS. A real deployment requires a reviewed cryptographic protocol, secure client key handling, device verification and recovery semantics, and careful metadata-leakage analysis. This article treats E2EE as an architectural constraint rather than inventing a complete secure messaging protocol.

14. What happens when components fail?

The gateway crashes. Connected clients lose their sockets, reconnect with backoff and jitter, register new routes, and resume from durable sync cursors. An accepted message is not dependent on the dead gateway's memory.

Alice's ACK is lost. She retries using the same client-generated ID. Atomic idempotency returns the original canonical message and order, rather than writing a duplicate.

Bob receives the same event twice. His device deduplicates by canonical message ID and applies the event idempotently. At-least-once delivery is compatible with one logical chat bubble.

Bob receives sequence 103 before 102. His client detects the gap, buffers within a limit, fetches missing history, and does not advance a contiguous checkpoint incorrectly.

The message store cannot meet its write contract. The service must not send a success ACK for an uncommitted message. Clients can retry safely, and already acknowledged data retains only the durability guarantee the chosen storage configuration actually provides.

Kafka or a delivery worker is down. Kafka-dependent fan-out, push, or receipt processing may lag. Recoverable outbox or change-feed state preserves pending work, and workers replay idempotently after recovery. If Kafka is on the acceptance path, new sends may also be blocked.

Millions of devices reconnect together. Gateways, authentication, registries, and sync APIs face a recovery surge. Use jittered reconnect, admission control, bounded catch-up batches, and capacity headroom rather than allowing recovery to create a second outage.

One conversation becomes extremely hot. Current-bucket write pressure and sequencer contention can persist even with historical bucketing. Apply workload-aware limits or specialized handling while preserving the stated ordering contract.

A region fails. Cross-region routing alone is not enough. The system must define replication lag, recovery point and time objectives, and how a new conversation writer gains authority without a stale writer continuing to accept conflicting sequences.

15. Multi-region messaging and split-brain prevention

A defensible baseline gives each conversation one current write authority, often associated with a home region. Users may connect to nearby gateways around the world, but accepted writes for that conversation pass through its authoritative ordering path. Other regions can handle routing, caching, delivery, and replication.

Active-active writes across regions are possible, but require a more sophisticated ordering or conflict contract. Two regions must not independently claim to own the same strict conversation sequence during a network partition. Lease ownership must be backed by effective fencing or another mechanism that rejects stale writers; a lease timestamp alone is not sufficient if old owners can still write.

Cross-region replication may be asynchronous, which means regional disaster recovery can involve a nonzero recovery point objective. Claiming both immediate local writes everywhere and strict global per-conversation ordering without explaining the coordination cost would be misleading.

16. Putting the architecture together

                 Mobile / Web / Desktop Clients
                              |
                       Load Balancer
                              |
                    WebSocket Gateways
                      |           |
                      |      Connection Registry
                      v
                  Messaging Service
                (auth, idempotency,
                conversation ordering)
                      |
                Durable Message Store
                      |
          Durable Outbox / Change Feed
                      |
               Event Stream / Workers
                 /       |         \
          Live fan-out  Sync index   Push hints
                |          |           |
             Gateways   Sync API   APNs / FCM
                           |
                       Reconnecting
                         devices

       Separate: conversation/membership metadata,
       presence/typing, media object storage + CDN,
       key/device management if E2EE is enabled

Return to Alice and Bob. Alice sends M with a stable client ID. The service authorizes the send, assigns a canonical message identity and conversation order, commits the durable message and its recoverable downstream handoff, then attempts to ACK Alice. The ACK disappears, but the write does not. Alice retries, and the service returns the original result.

Bob's connected laptop receives M and acknowledges receipt according to the product's delivery rule. Bob's offline phone later reconnects and obtains M through its own sync cursor. If the laptop gets a duplicate event, it ignores the duplicate logical message. If the phone discovers a sequence gap, it repairs it from authoritative history. None of these cases requires pretending that a WebSocket is a durable queue.

The design succeeds because it draws precise boundaries: durable acceptance is not device delivery; device delivery is not reading; live routing is not offline recovery; and per-conversation order is not global order. Once those contracts are explicit, component choices become much easier to evaluate.

References and further reading

For deeper explanations of the design decisions and their failure boundaries, see SeeItFlow's Messaging Platform module:

Designing Dropbox: File Storage & Sync System Design

MANGESH MANDLIK — Sun, 11 Oct 2026 06:55:50 +0000

Uploading a file is straightforward. Keeping its bytes, metadata, permissions, history, and copies on several intermittently connected devices consistent is the real system-design problem.

Consider a document named report.docx. You edit it on your laptop, close the lid halfway through an upload, rename its folder on another device, and then open your phone after several days offline. Meanwhile, a collaborator may have edited the same document or lost access to it.

A file storage and synchronization system must answer more than “where do we store the bytes?” It must know which version is current, which changes each device has observed, what to do when histories diverge, and whether a user is still authorized to download a particular version.

We'll design a Dropbox- or Google Drive-like system from a simple upload API to a reliable sync architecture. This is a conceptual design, not a claim about either company's internal implementation.

1. Requirements and scale

The system supports uploading and downloading large files, resumable transfers, folders, rename and move, automatic synchronization across devices, offline edits, version history, deletion recovery, and viewer/editor sharing.

The priorities are durability, integrity, secure access, availability despite individual failures, reasonable synchronization freshness, and bandwidth efficiency. “Instant sync” and “never lose a file” are not defensible unconditional guarantees: network partitions, retention rules, disasters, and client availability impose real limits.

For a scale exercise, suppose we have 500 million registered users, 100 million daily active users, and 200 million uploads per day. With an illustrative average upload of 5 MB, that's approximately 1 PB of new uploaded data per day before compression, deduplication, replication, or retention effects. The mean arrival rate is about 2,315 uploads/second, but peak capacity must be sized above the mean. Assume individual files can reach 50 GB. These are invented design inputs, not published statistics for Dropbox or Google Drive.

Notice that upload requests, bytes ingested, metadata operations, download traffic, and device sync connections scale differently. We should not expect one service or database to solve them all.

2. Start with the simplest upload architecture

Client ── file bytes ──> Application server ──> Local disk
                             │
                             └───────────────> Metadata database

This works for small workloads. But when a large upload is interrupted, a one-shot upload may have to restart. Application servers also spend bandwidth and connection capacity relaying bytes rather than handling application logic. Local disks complicate replication, capacity expansion, and recovery.

Proxying bytes is not inherently wrong: inspection, transformations, restricted networks, or specialized clients can justify it. For general large-file transfers at our assumed scale, however, separating the control plane from the byte-transfer path is a useful next step.

3. Separate file metadata from object storage

                       ┌──────────────────────┐
Client ── API calls ──> │ File / Metadata API  │ ──> Metadata DB
                       └──────────────────────┘      identity, folders,
                                  │                 ACLs, versions, state
                                  │ signed capability
                                  ▼
Client ── file bytes directly ──> Object storage
                                 immutable version objects

A metadata record might contain file_id, owner_id, parent_folder_id, filename, current_version_id, lifecycle_state, and a concurrency token. A separate versions table stores version_id, file_id, parent_version_id, content checksum, size, and object key. The file's stable ID is independent of its path: renaming report.docx or moving it into another folder changes metadata, not the file's identity or necessarily its stored bytes.

Folders also have stable IDs and parent relationships. Listing a folder means querying its immediate children with pagination. A folder move can change the folder's parent pointer without rewriting every descendant, although subtree permissions, path caches, and asynchronous indexes may still require careful handling. If sibling names must be unique, enforce that invariant within the correct namespace, including the root-folder case.

Object storage holds large binary objects; the metadata store handles queryable relationships and conditional updates. This is a workload-driven decision, not a rule that databases can never store small blobs.

The metadata–object consistency gap

The database transaction that publishes a file version cannot ordinarily atomically include the object-storage upload. Therefore, use an explicit lifecycle:

CREATED → UPLOADING → VERIFYING → AVAILABLE
                ↘ FAILED / EXPIRED
AVAILABLE → DELETING → DELETED (subject to retention)

A safe publication sequence is:

Create a metadata record and an upload session in a non-available state.
Upload bytes to a server-selected object key.
Complete the object upload and verify the integrity evidence supported by the storage API.
Atomically update the file's current-version pointer and record its sync change using a transaction or another durable, correctly ordered publication mechanism.
Reconcile abandoned sessions and orphan objects asynchronously.

The fourth step matters. If a file becomes visible in metadata but its corresponding change event is lost, other devices may never discover it. A transactional outbox or a change log committed alongside metadata can bridge that boundary; publishing to an unrelated queue after the database commit without recovery is not enough.

If storage succeeds but the metadata commit fails, the object may be orphaned. A reconciler checks the session, object, checksum, and current metadata state before safely retrying publication or scheduling deletion after a grace period. If metadata says AVAILABLE but the object is missing, fail visibly and investigate or recover from a replica/backup; never serve an empty object as success.

For deeper treatment, see File Storage & Sync — Architecture Deep Dive.

4. Design the upload API and idempotency contracts

An illustrative API is:

Endpoint	Purpose
`POST /files`	Create file metadata; client supplies an idempotency key
`POST /files/{id}/uploads`	Start or resume an upload session
`POST /files/{id}/complete`	Complete and verify a session; safe to retry
`GET /files/{id}`	Return metadata and current version
`GET /files/{id}/download`	Authorize and issue a short-lived download capability
`PATCH /files/{id}`	Rename/move with a version precondition
`GET /sync/changes?cursor=...`	Fetch durable changes for a device

These are example contracts, not a mandated REST standard. A production implementation must also define deletion, restore, sharing, version retrieval, and session-status endpoints.

A retry of POST /files must not accidentally create two files. An idempotency key scoped to the caller and operation allows the server to return the same logical result. Completing an upload should similarly be retry-safe: a timeout may mean “completion succeeded but the response was lost.” Repeating that request must not publish a duplicate version.

Idempotency handles retries of the same operation. It does not solve two genuinely concurrent edits; those require version preconditions.

5. Direct uploads and presigned URL security

For the normal large-file path, the application authenticates the caller, checks permission and quota, creates the session, and returns a short-lived presigned upload URL. The client sends bytes directly to object storage.

A presigned URL is generally a bearer capability. Anyone who obtains it can potentially use it until expiration, subject to signature and storage-policy constraints. It is not automatically bound to the original user's identity. Therefore:

Authorize before issuing it; generate object keys on the server.
Scope it to a specific operation and object or multipart part.
Use HTTPS, short expirations, appropriate request constraints where supported, and avoid leaking URLs in logs.
Configure browser CORS separately; CORS is not a replacement for authorization.
Recheck authorization and the expected version before publishing a completed upload, because permissions or file state may have changed while bytes were in flight.

Revoking a user's ACL does not automatically revoke a previously issued, unexpired URL. If near-immediate revocation is mandatory, an edge or proxy path that validates current authorization on access may be necessary. Even then, bytes already downloaded cannot be recalled.

6. Resumable multipart uploads

A single 10 GB PUT still has an expensive failure mode: a dropped connection can force a complete restart. Native object-storage multipart upload lets the client send separately acknowledged parts.

Upload session U17
  part 1 ──────── ✓
  part 2 ──────── ✓
  part 3 ── X     retry part 3
  part 4 ──────── ✓
                ↓
  complete known parts → verify → publish new version

A durable session tracks its owner, file ID, expected base version, storage upload ID, expiration, and status. The client can upload parts concurrently and, after reconnecting, reconcile confirmed parts against the storage provider's actual state where supported. Its own local progress counter may be stale if a part succeeded but the response was lost.

Part size is a trade-off: smaller parts reduce failed-retry cost but increase request and tracking overhead; larger parts reduce coordination but increase retry cost and buffering. Provider minimums, maximum part counts, file size, client memory, and network behavior determine the right choice. Don't treat a single part size as universal across providers.

An object-store multipart part is not necessarily the same abstraction as an application-level delta-sync chunk. Native multipart simplifies upload assembly; a content-addressed chunk store is a separate, substantially more complex design.

Integrity is not the same as upload completion

TLS protects bytes in transit, but it does not prove that the application selected the correct file. Where supported, validate per-part checksums and the assembled object's expected checksum. A mismatch in the final whole-object hash shows that something differs; it does not identify the corrupted part without finer-grained evidence.

Do not assume an object-storage ETag is always an MD5 digest of the entire file. ETag semantics vary by provider and upload/encryption method, especially for multipart objects. Use explicit checksum mechanisms and clearly specify what has been verified before publication.

Expired sessions and abandoned multipart uploads require cleanup. Unassembled parts may still consume storage and cost money; expiry policies and periodic reconciliation are part of the design, not optional housekeeping.

7. Downloads, range requests, and private CDN delivery

For downloads, the file API checks the current ACL and returns a short-lived signed capability for a specific immutable file version. The client downloads bytes from storage, or through an appropriately secured CDN where caching is useful.

HTTP Range requests can resume an interrupted download or retrieve part of a large file. A robust client should ensure the resumed ranges belong to the same immutable object version and verify the completed file's expected checksum. Simply appending bytes from a mutable URL can mix different versions.

CDN caching is workload-dependent. Frequently requested, immutable shared objects may benefit; private files downloaded once per user may not. A private CDN needs its own edge authorization and private-origin configuration. A presigned origin-storage URL does not magically make a CDN path private or authorized. Already-issued CDN capabilities have similar revocation windows to signed origin URLs.

8. Multi-device sync is a reconciliation protocol

A sync client needs persistent local state, not merely a filesystem watcher. For each file it tracks the stable remote ID, local path, last synchronized server version, local fingerprint, pending operations, and its durable checkpoint.

Laptop: edit report.docx
  local state: LOCAL_DIRTY
           ↓
  upload candidate version based on v5
           ↓
  server checks current version == v5
           ↓
  publish v6 + durable change record
           ↓
  push hint to other online devices
           ↓
Phone: GET /sync/changes?cursor=C
           ↓
  apply v6, then advance local cursor

Local filesystem notifications can be missed, so a periodic reconciliation scan is a useful safety net. Server push via WebSocket, mobile push, or long polling is only a wake-up hint. The durable, cursor-based change log is the source of truth.

A cursor identifies a position in an ordered change stream within its defined scope. It is not simply the client's wall-clock timestamp. The client must not persist a cursor beyond changes it has durably applied; otherwise a crash can permanently skip an update. Applying duplicate events must be safe through operation IDs, version checks, and idempotent local updates.

One subtlety: when changes are paginated or generated concurrently, the server must define a consistent ordering/checkpoint contract so clients cannot skip a late-committed entry with an earlier cursor position. A snapshot-plus-change-stream handoff likewise needs a defined boundary; “take a snapshot, then start polling” without such a boundary can miss mutations.

If a device has been offline longer than change-log retention, its old cursor may be invalid. It must obtain a consistent namespace snapshot, compare it with local state, reconcile offline operations, and establish a fresh cursor without losing changes during the transition.

9. Whole-file hashing is not delta synchronization

A content hash answers: are these bytes equal? It does not answer: which blocks changed?

There are three practical strategies:

Strategy	Works well when	Main limitation
Whole-file replacement	Files are small or binary deltas are ineffective	Transfers the entire new file
Fixed-size block comparison	Large files change in place without shifting offsets	Insertions shift later block boundaries
Rolling-signature or content-defined chunking	Large files have localized edits and reusable regions	More CPU, signatures, chunk metadata, and GC complexity

An rsync-style rolling matching algorithm and content-defined chunking are related approaches, not identical algorithms. Both can locate reusable regions despite offset shifts, but their chunking and matching procedures differ.

For compressed archives, encrypted data, and some document formats, a tiny logical edit can change much of the serialized byte stream. In those cases, a delta algorithm may save little bandwidth. The client should choose based on measured transfer savings versus computation and coordination cost, not promise that every one-paragraph edit uploads only a few kilobytes.

10. Detect conflicts using version lineage

Suppose both laptop and phone begin with report.docx at version v5. The laptop edits and successfully publishes v6. The phone was offline and independently edited its copy based on v5.

             v5 (shared base)
             /            \
   laptop candidate     phone candidate
        accepted v6       base=v5
             │              │
       current=v6      compare-and-set fails
                            │
                      preserve as conflict

The server accepts a new current version only when its expected base version matches the current version. A stale writer must not silently overwrite the newer version. This check must be atomic at the metadata commit boundary.

Possible policies include preserving a conflict copy, last-write-wins, or a three-way merge for supported text formats. Conflict copies reduce silent overwrites but require user intervention and do not guarantee zero data loss if the underlying storage or backup system fails. Last-write-wins is simple but can discard edits. Automatic merging is format-dependent; it is not a general solution for arbitrary binary files. Real-time collaborative editing with OT or CRDTs is a different problem from background file synchronization.

Metadata edits also race. Rename and move operate on stable IDs; delete should produce a tombstone that offline devices can observe. When a deleted file receives a stale offline edit, the product must explicitly choose to reject it, restore a conflict copy, or create a recovered file. A naive sync client that sees a missing file and uploads its old local copy risks resurrecting a deletion.

Content version checks and metadata-operation concurrency tokens may need separate treatment: a rename need not create a new content blob, yet it must still be protected against conflicting namespace mutations.

11. Sharing and access control

The metadata layer records ownership, direct shares, and possibly inherited folder permissions. Each operation checks whether the caller may read, edit, rename, delete, share, or restore the target. Folder inheritance and override rules must be defined rather than guessed.

For a file shared by Alice with Bob, Bob requests a download through the application, which checks Bob's access and issues a scoped signed URL. Direct object-store access after that check is intentional. The file bytes need not pass through the authorization service.

A recipient-oriented index for “shared with me” avoids scanning every owner's ACLs as metadata is sharded. That index must be maintained carefully, but an eventually updated discovery index must not become the sole authority for granting access. Authorization-sensitive paths should check authoritative permission state or an explicitly safe consistency mechanism. If current authorization cannot be determined, fail closed.

Malware scanning can also be a security gate. If the product requires scanning before sharing or downloading, a failed scanner must leave the file quarantined or pending, not silently mark it usable. Thumbnail generation and search indexing, by contrast, can often fail without making the file itself unavailable.

12. Scaling the metadata and sync planes

Object storage handles bulk bytes; the metadata database handles frequently changing file, folder, permission, and version records. Read replicas may help appropriate read workloads, but stale replicas are unsafe for decisions requiring the latest ACL or concurrency state.

Owner-based sharding keeps a user's namespace relatively local. Its weakness is a very large or active user/tenant, and cross-owner “shared with me” queries. Tenant-based partitioning improves isolation but can produce giant-tenant hotspots. File-ID sharding spreads load but requires secondary indexes for folder and owner queries. There is no universally best shard key.

A huge folder should be listed with pagination, not enumerated in a single response. Fan-out notifications and derived indexes should be processed asynchronously through durable jobs. Per-tenant fairness and backpressure protect other users when one tenant generates extreme traffic.

The sync change stream also has independent limits: write throughput, cursor retention, ordering scope, connected-device count, and notification fan-out. Partitioning by user can help localize ordered processing if the underlying stream actually provides that ordering, but it does not replace operation idempotency or atomic version preconditions.

13. Failure scenarios that reveal correctness bugs

Failure	Safe behavior
Upload interrupted	Reconcile storage-confirmed parts; retry missing parts only
Upload succeeds, metadata commit fails	Idempotent completion plus reconciliation; never publish unverified bytes
Presigned URL expires	Check session status and refresh URLs for remaining parts if still valid
Metadata primary unavailable	Fail over safely or reject mutations; don't acknowledge writes buffered only in memory
Storage temporarily unavailable	Bound retries with backoff and jitter; expose degraded byte-transfer state
Sync push unavailable	Poll durable change log on reconnect or schedule
Duplicate/out-of-order sync delivery	Use operation IDs, version preconditions, and safe cursor checkpoints
Device offline beyond cursor retention	Full snapshot reconciliation with a safe change-stream handoff
ACL revoked after URL issuance	Recognize capability expiry window; stricter validation if required
Abandoned multipart sessions	Abort expired sessions and reclaim parts
Garbage collector thinks a live object is orphaned	Grace period, durable deletion state, reference checks, and independent audit
Regional outage	Follow defined RPO/RTO; prevent split-brain writes during failover

The garbage collector deserves special attention. An object may appear unreferenced because a metadata query failed, a replication lagged, or a deduplication reference has not yet been observed. Never hard-delete a blob solely because one transient lookup returns no references. Use retention windows, tombstones, durable GC state, rechecks, and recoverable backups.

Cross-user content deduplication is optional, not a free storage win. It complicates reference counting and deletion, and a naive “does this hash already exist?” endpoint can leak information about another user's private data. Start without global deduplication unless measured storage savings justify its security and operational complexity.

For a scenario-by-scenario breakdown, see File Storage & Sync — Failure & Scale Scenarios.

14. Architectural trade-offs worth defending

Direct transfer or application proxy? Direct signed upload/download reduces application bandwidth at scale. Proxying remains appropriate for mandatory inline inspection, transformation, or real-time authorization controls.

Native multipart or application-managed chunk store? Native multipart simplifies reliable large uploads. A content-addressed chunk store can support cross-version reuse and deduplication but introduces reference-counting, garbage-collection, and privacy complexity.

Push or polling? Push reduces notification delay for connected clients. Polling is a necessary fallback. Neither replaces the durable change log.

Strong consistency or eventual consistency? The acceptance of a new current version needs a correct atomic precondition. Other devices may observe that committed version later. These are separate consistency boundaries.

Single-region or multi-region? A single-region primary with tested backups and a defined disaster-recovery plan is a defensible baseline. Cross-region replication can improve recovery objectives but requires handling replication lag and preventing two primaries from accepting conflicting writes during a partition. Object durability is not the same as an RPO or RTO guarantee.

For alternatives and decision criteria, see File Storage & Sync — Trade-offs & Decisions.

15. System design interview follow-ups

Why not keep file bytes in PostgreSQL? Small blobs may be fine there. At the stated multi-GB-file and petabyte-scale ingest workload, separate object storage better fits byte capacity, transfer, and lifecycle requirements.

What happens when a 50 GB upload fails at 90%? Recover the durable multipart session, reconcile parts confirmed by storage, refresh expired upload capabilities if the session is valid, and retry only missing parts.

Can the client trust ETag as an MD5? No. Provider, multipart, and encryption behavior affect its meaning. Specify explicit checksum verification.

How do devices discover missed changes? Fetch from the durable change log using a cursor. Push is a hint; polling and reconnect reconciliation provide recovery.

Does SHA-256 identify changed blocks? A single whole-file SHA-256 does not. Delta algorithms need block-level or rolling/content-defined signatures.

What if two offline devices edit the same file? Both carry the base version they edited. Atomic compare-and-set accepts one current version; the divergent edit is handled by an explicit conflict policy.

Can you instantly revoke an issued presigned URL? Not by changing only the file ACL. The already-issued capability may remain usable until expiry unless the access path supports further revocation checks or invalidation.

What if a device reconnects after six months? If its cursor is older than retained history, take a consistent snapshot, reconcile queued local changes, and rejoin the change stream at a safe checkpoint.

How do you prevent sync from losing a metadata mutation? Persist the mutation and its change-log/outbox record atomically or use an equivalent recoverable publication protocol. A best-effort queue publish after commit can lose events.

How do you recover from accidental object deletion? Recovery depends on version retention, backups, and provider replication capabilities. A design that promises recovery must test its restore procedure and define acceptable loss and recovery time.

A larger question bank is available in File Storage & Sync — Interview Perspective.

Final architecture

                         ┌──────────────────────────┐
Laptop / Phone / Web ───> │ API / Auth / File Service│ ──> Metadata DB
        │                └──────────┬───────────────┘     file IDs, ACLs,
        │                           │                     versions, folders
        │                    signed upload/download
        │                           │
        └──────── file bytes ───────┴───────────────> Object Storage
        │                                           immutable version blobs
        │
        └── Sync client ──> Sync API ──> Durable change log / outbox
                               │                    │
                               └── cursor reads <───┘
                                      │
                           Push hints / polling fallback

              Background workers: scanning, previews, indexing,
                    session reconciliation, retention / GC

The important insight is not the number of boxes. It is the boundaries between guarantees:

Object storage protects and serves bytes; metadata identifies which bytes belong to which logical file version.
A verified upload is not visible until its version is safely published.
An accepted edit is conditionally committed; its arrival on other devices is asynchronous.
Push improves freshness, but durable replay preserves synchronization correctness.
Authorization gates capability issuance; signed URLs have explicit lifetime and revocation limitations.
Garbage collection must not destroy data that another component still references.

A good file storage and sync design makes those boundaries explicit. That is what turns a basic upload feature into a system that can survive interrupted transfers, offline edits, permission changes, retries, and operational failures without silently corrupting user state.

References and further reading

Designing Search & Autocomplete: Inverted Index, Trie, Ranking & Scale

MANGESH MANDLIK — Sat, 10 Oct 2026 04:10:33 +0000

The difficult part of search isn't finding a word. It's returning the right results quickly while documents change, users type, and parts of the search infrastructure fail.

Imagine an e-commerce catalog with millions of products. A user types iph, sees suggestions, completes iphone case, applies a price filter, and expects relevant results almost immediately.

A database query can support this at small scale. At larger scale, the system must solve several distinct problems: text retrieval, suggestion generation, ranking, freshness, filtering, and distributed execution. Treating all of them as one database lookup makes the design harder to reason about.

This article develops a search and autocomplete system from requirements through production failure modes. The example is a product catalog, but the underlying techniques also apply to document search, content discovery, and entity search.

1. Requirements: What Are We Actually Building?

Functional requirements

Search products by keywords across titles, descriptions, and selected attributes.
Show autocomplete suggestions as users type.
Rank results by relevance, with optional popularity and recency signals.
Support filters (brand, category, price, availability) and facets (counts by brand or category).
Handle pagination and, where useful, typo tolerance.
Reflect product additions, edits, and deletions within an agreed freshness window.

Suggestions can be query completions (iphone 15 case), entity suggestions (a particular product), or a mixture. This decision affects indexing and ranking; a trie of words is not automatically a complete autocomplete product.

Non-functional requirements

For discussion, assume:

Dimension	Illustrative target or assumption
Catalog	10 million searchable products
Full search	2,000 requests/second at peak
Autocomplete	10,000 requests/second at peak
Full-search latency	P99 below 200 ms
Autocomplete latency	P99 below 100 ms
Index freshness	Most updates visible within several seconds
Availability	Continue serving when individual nodes fail, subject to product correctness rules

These are design assumptions, not universal search-industry benchmarks. The actual requirements depend on corpus size, languages, query mix, hardware, and product expectations.

A crucial distinction: search correctness, ranking quality, freshness, and availability are different properties. A query can return valid matches in the wrong order; an index can be available but stale; a partial result can be fast but incomplete.

2. Why Not Just Use SQL `LIKE`?

The simplest implementation queries the transactional database directly:

SELECT id, title
FROM products
WHERE LOWER(title) LIKE '%iphone%'
LIMIT 10;

A leading-wildcard pattern such as %iphone% generally cannot use a conventional B-tree index as an efficient prefix range lookup. On a large table, it may require expensive scanning.

But the correct conclusion is not that relational databases cannot do full-text search. PostgreSQL, for example, supports full-text indexes and trigram-based approaches. Database-native search may be the right choice for a smaller catalog or modest query volume.

A dedicated search engine becomes attractive when we need several capabilities together: inverted indexes, language-aware analysis, flexible ranking, fuzzy matching, faceting, and independent scaling of read-heavy search traffic. The trade-off is another data system and an eventually consistent copy of source data.

Design decision: keep the transactional database authoritative and maintain a separate, query-optimized search index for this example.

For a deeper comparison, see Database vs Dedicated Search Index.

3. High-Level Search & Autocomplete Architecture

                        WRITE / INDEXING PATH

  Product Service
        |
        v
  Source Database  (source of truth)
        |
        v
  CDC / Reliable Change Events
        |
        v
  Durable Stream / Queue
        |
        v
  Indexing Workers ----> Transform + Analyze + Version Check
        |                              |
        +------------------------------+
        |                              |
        v                              v
  Full-Text Search Index       Autocomplete Suggestion Index
  (inverted index)            (prefix / n-gram / precomputed)

                        READ / QUERY PATH

  Browser / App
        |
        v
  Search API / Gateway
        |
        +---- /suggest ---> Suggestion Cache ---> Suggestion Service
        |                                             |
        |                                             v
        |                                      Prefix Candidates
        |                                             |
        |                                      Suggestion Ranking
        |
        +---- /search ----> Result Cache ----> Search Coordinator
                                                  |
                                      +-----------+-----------+
                                      |           |           |
                                   Shard A     Shard B     Shard C
                                      |           |           |
                                      +-----------+-----------+
                                                  |
                                          Merge / Rank Top-K
                                                  |
                                             Search Results

The diagram separates indexing from serving. The search engine does not need to query the source database for every request. That separation protects transactional workloads and allows search capacity to scale independently.

The diagram also shows a dedicated suggestion index. This is a choice for our assumed low-latency, high-autocomplete-QPS workload, not a requirement for every implementation. A unified search engine with prefix queries can be simpler and entirely sufficient.

4. How an Inverted Index Works

A full-text search engine typically uses an inverted index, mapping terms to the documents containing them.

Suppose the catalog contains:

D1: "Wireless iPhone case"
D2: "iPhone fast charger"
D3: "Wireless charging pad"

After suitable text analysis, a simplified index is:

"iphone"   -> [D1, D2]
"wireless" -> [D1, D3]
"case"     -> [D1]
"charger"  -> [D2]
"charging" -> [D3]

A query for wireless iphone can retrieve candidate documents by intersecting or combining the posting lists, depending on AND/OR query semantics. The search engine then scores the candidates.

Real posting lists may also store term frequencies and positions. Positions allow phrase queries to distinguish wireless iphone case from the same words appearing far apart.

Term lookup can be fast, but processing the posting list is not constant-time regardless of corpus size. A common term can appear in millions of documents. Compression, skip structures, and top-k pruning help control that cost.

Text analysis must be compatible

Indexing and querying must use compatible analyzers:

Raw text -> Tokenize -> Normalize -> Optional stemming/synonyms -> Terms

Normalization: case folding and, when appropriate, accent handling.
Tokenization: splitting text into meaningful units; language-dependent.
Stemming/lemmatization: optional normalization of related word forms.
Synonyms: domain-specific mappings such as tv and television.
Field-specific behavior: product titles may be analyzed; SKUs and IDs usually require exact matching.

Applying aggressive stemming to an SKU or brand field can reduce correctness rather than improve it. Likewise, changing an analyzer can require reindexing existing documents, because the stored terms were generated under the previous rules.

For the internals, see Search & Autocomplete — Architecture Deep Dive.

5. Autocomplete System Design: Generating Suggestions

Autocomplete is a different workload from full search. A user may issue several suggestion requests before submitting one search, and the usefulness of a response drops quickly as they type the next character.

The serving pipeline is:

Typed prefix -> Candidate generation -> Suggestion ranking -> Top suggestions

There are several legitimate candidate-generation approaches.

Approach	Strength	Main trade-off
Search-engine prefix query	Reuses existing search infrastructure	Shares resources with full search; performance depends on engine and load
Edge n-grams	Prefixes are indexed as terms for fast lookup	More index storage and write amplification
Trie / FST-like structure	Purpose-built prefix traversal; can precompute top suggestions	Additional update, memory, and operational complexity
Precomputed query-log suggestions	Efficient for frequently searched queries	Requires freshness, spam filtering, and safe handling of query logs

Trie example

For the terms car, card, and care, a conceptual trie looks like:

(root)
   |
   c
   |
   a
   |
   r  ("car")
  / \
 d   e
("card") ("care")

Looking up car finds the node for that prefix. Retrieving the best suggestions can still require traversing many descendants unless the structure maintains precomputed top-k suggestions at nodes. Prefix lookup and ranking are separate operations.

Edge n-grams example

For iphone, index prefixes such as:

ip, iph, ipho, iphon, iphone

Then a query for iph can match the corresponding prefix term. In practice, minimum and maximum prefix lengths should be configured deliberately: indexing every single-character prefix may be wasteful, and very short prefixes often produce noisy suggestions.

Ranking suggestions

Prefix matching only produces candidates. Suggestions can then be ordered by:

Historical search or selection frequency.
Recent trending activity.
Locale, language, and category context.
Optional personalization.
Safety, policy, and abuse controls.

Never treat raw query-log popularity as automatically trustworthy. Bots, spam, inappropriate queries, and old trends can distort the list. Query logs may also contain sensitive user input, so aggregation and retention need appropriate privacy controls.

For this example, a dedicated suggestion index helps isolate autocomplete latency from expensive full-text searches. It remains a workload-driven decision, not a universally superior architecture.

6. Preventing Autocomplete Request Storms

Without client-side controls, a user typing iphone could issue six requests in quick succession. Across many active users, this is a meaningful traffic multiplier.

Use a combination of:

Debouncing: wait a short, tuned interval after typing before requesting suggestions.
Minimum prefix length: avoid expensive, low-value one-character requests where the product allows it.
Cancellation or stale-response suppression: prevent a slow response for ip from replacing newer results for iphone.
Server-side rate limits: protect shared resources from abusive clients or bursts.
Caching hot prefixes: popular prefixes often have excellent reuse.

Cancellation is an optimization, not a correctness guarantee: a request may already have reached the server. The client should still ignore responses that do not correspond to its latest input.

A practical API might be:

GET /suggest?q=iph&locale=en-US&limit=8
GET /search?q=iphone%20case&category=accessories&sort=relevance&limit=20

The service should bound input length, expansion complexity, and returned suggestion count. The cache key must include any locale, market, or other context that changes the response.

7. Search Ranking: Retrieval Is Not Relevance

An inverted index answers which documents might match. It does not, by itself, determine which matches are most useful.

A practical ranking pipeline is:

Query
  |
  v
Candidate retrieval + eligibility filters
  |
  v
First-stage lexical ranking (for example, BM25)
  |
  v
Optional re-ranking of a bounded top-N set
  |
  v
Final eligibility / business rules + top-K results

BM25 is a common lexical baseline. It rewards matches on distinctive terms, considers term frequency with diminishing returns, and normalizes for document length. It is more nuanced than simply counting keyword occurrences.

An optional second-stage ranker can incorporate popularity, recency, category relevance, or personalization. Running an expensive model on a bounded candidate set controls re-ranking cost, not the cost of retrieving candidates from the index.

Business rules need care. If an out-of-stock product must never be shown, filtering it only after selecting the top 10 could leave too few results even when valid products exist lower down. Apply hard eligibility constraints early enough to preserve the requested result count, or retrieve enough additional candidates to satisfy them.

When the optional re-ranker fails, the system can fall back to lexical ranking, preserving search availability with reduced relevance quality.

8. Filters, Facets, and Typo Tolerance

These features are related but not interchangeable.

Filters constrain eligibility: brand = Apple, price <= 500, or in_stock = true. They generally should not change textual relevance scores unless that is a deliberate ranking rule.

Facets aggregate the matching set: for example, how many eligible products fall into each brand or price range. Facets may be expensive on large result sets or high-cardinality fields. Their semantics must be defined carefully, especially whether a facet's own filter is included when computing its counts.

Typo tolerance expands potential matches. Common techniques include edit-distance matching, n-grams, and spelling correction. They can improve recall, but broad fuzzy matching on a two-character query may expand into an enormous set of candidate terms.

Use bounded edit distance, minimum query length, and query-complexity limits. Exact-match identifiers such as SKUs should generally not receive the same fuzzy treatment as natural-language titles.

9. Keeping the Search Index Up to Date

The source database is authoritative; the search index is a derived view. Product updates need to reach the index reliably.

A typical pipeline is:

Committed DB change
        |
        v
CDC or reliably published event
        |
        v
Durable log / queue
        |
        v
Indexer: validate -> transform -> version-check
        |
        v
Search index (+ suggestion index when applicable)
        |
        v
Refresh / visibility to queries

CDC is one option, not the only one. Application-emitted events and scheduled imports can also work. If an application writes the database and separately publishes an event, it must address the dual-write failure gap (for example, through a transactional outbox).

Duplicate and out-of-order updates

Consider the same product receiving:

Update v11: price = 120
Update v12: price = 100

If v12 is applied first and a delayed v11 is then applied without checking versions, search shows the wrong price. The indexer must reject stale updates based on a trustworthy, monotonically increasing per-document version. A wall-clock updated_at timestamp alone may not be reliable enough if timestamps collide or clocks are inconsistent.

Deletes require the same care. A delayed pre-delete update must not resurrect a deleted product. The system needs version-aware tombstone handling or an equivalent mechanism that prevents stale resurrection.

Indexing lag is not just queue lag

Even after an indexer successfully writes a document, the search engine may not expose it to normal queries until a refresh. Observed freshness includes both pipeline delay and index visibility delay, plus any applicable result-cache staleness.

Reducing refresh intervals can improve freshness but increases indexing/segment-management work. A freshness target should therefore be an explicit product requirement.

Full reindexing without taking search offline

Changing analyzers, mappings, or transformations can require rebuilding the corpus. A safer approach is:

Create a new versioned index.
Bulk-load a consistent source snapshot.
Apply and catch up on changes made during the rebuild.
Validate counts, sample documents, deletes, and search quality.
Switch the serving alias or routing layer to the new index.
Retain the old index briefly for rollback.

The snapshot and change stream need a well-defined handoff position; otherwise updates made during the rebuild can be missed. Do not destructively rebuild the only live index and assume rollback will be easy.

10. Caching Search Results Correctly

Popular queries repeat, making caching attractive. But a cache key based only on the raw query string is often incorrect.

Two requests for iphone can produce different responses depending on:

Filters and sort order.
Locale, region, or tenant.
Pagination cursor or page.
Ranking/index version.
Personalization or user-specific eligibility.

Cache only across requests that are genuinely allowed to share the same response. Authorization or tenant boundaries must never be lost through cache-key normalization.

Cache key = normalized query + filters + sort + locale
          + page/cursor + index/ranker version
          + relevant access/personalization context

Choose TTLs according to query reuse and freshness requirements; there is no universally correct 60-second TTL. For personalized queries with little full-response reuse, caching candidates or reusable components may work better.

Cache stampede

Suppose the cache absorbs 80% of incoming search requests. The index normally receives 20%. If the cache disappears and incoming demand stays the same, index-facing traffic can rise by approximately:

1 / (1 - 0.80) = 5x

That is an example derived from an assumed hit rate, not a universal multiplier. Request coalescing, TTL jitter, stale-while-revalidate where acceptable, capacity headroom, and load shedding can limit the damage.

11. Sharding, Replication, and Scatter-Gather

When one search node cannot efficiently hold or serve the corpus, distribute documents across shards.

Hash-based document sharding

A document ID hash can spread documents relatively evenly. A broad keyword query may then need to reach every relevant shard, because matching documents could be anywhere.

Routed sharding

Routing by tenant, region, or category can reduce query fan-out when the query specifies that routing dimension. It can also create skew: one very large tenant may produce a hot shard.

Scatter-gather query execution

             Search Coordinator
             /       |        \
          Shard A  Shard B   Shard C
          top-K     top-K     top-K
             \       |        /
              Merge + Global Top-K

Each shard produces its best local candidates; the coordinator merges them. For a straightforward global top-K under a consistent comparable scoring function, retrieving K from each shard is sufficient before global merging. Additional candidates may be required by a subsequent global re-ranking stage, pagination strategy, or other engine-specific behavior; oversampling is not inherently necessary just because results are sharded.

Distributed search is sensitive to tail latency: a single slow shard can delay the response. Replicas improve availability and read capacity, but do not guarantee uninterrupted or perfectly fresh service. If no replica of a required shard responds, the system must explicitly choose to fail the query or return clearly marked partial results.

For a product catalog, partial results may be acceptable. For legal discovery or financial reporting, silently omitting a shard's results could be unacceptable.

12. Pagination: Why Deep OFFSET Becomes Expensive

Simple pagination is easy to explain:

-- Conceptual syntax; actual search-engine APIs differ.
OFFSET 10000 LIMIT 20

In a distributed search, deep offset pagination may require shards to collect many more results than the 20 shown to the user, increasing work with page depth. Changes between requests can also cause duplicates or omissions.

Search-after/cursor pagination carries the sort position of the last result, including a deterministic tie-breaker such as document ID:

{
  "query": "iphone case",
  "limit": 20,
  "search_after": [8.42, "product_991823"]
}

The example assumes a score-descending, ID-tiebroken sort; the actual cursor and comparison rules must match the engine's configured ordering. A cursor avoids repeatedly skipping earlier results, but it does not freeze the underlying result set. If stable multi-page results matter, combine it with point-in-time/snapshot semantics supported by the engine.

13. Production Failure Scenarios

A design is incomplete if it only describes the happy path. These are the failures most likely to change our architectural choices.

Failure	User-visible impact	Response
Search shard unavailable	Errors, higher latency, or incomplete results	Route to healthy replica; otherwise follow explicit fail/partial policy
Cache outage	Search cluster receives sudden extra traffic	Request coalescing, headroom, load shedding
Indexer backlog	Newly changed products remain stale in search	Monitor oldest event age, retain backlog, scale catch-up capacity
Lost CDC events / retention gap	Index may permanently diverge	Reconcile against source; rebuild when replay is impossible
Bad analyzer/index deployment	Wrong matches or degraded relevance despite healthy servers	Versioned indexes, validation, alias rollback
Old update arrives after delete	Deleted product reappears	Source-version checks and durable tombstone semantics
Hot prefix or expensive fuzzy query	CPU spikes, increased tail latency	Cache hot prefixes; bound expansions, deadlines, and query complexity
Ranking dependency fails	Search relevance degrades	Fall back to first-stage lexical ranking
Autocomplete index lags	Search finds a product that suggestions omit	Separate freshness monitoring and defined suggestion-update cadence

Monitor more than availability

A search service can return HTTP 200 while its results are wrong. Useful measurements include:

Search and autocomplete P50/P95/P99 latency, error rate, and QPS.
Per-shard latency, timeouts, load, and replica health.
Cache hit rate, hot-key pressure, and upstream request coalescing.
Indexing backlog, oldest unprocessed event age, and end-to-end freshness.
Source/index divergence checks, delete correctness, and failed indexing events.
Zero-result rate, relevance regression tests, and other quality signals interpreted in context.

For more failure walkthroughs, see Search & Autocomplete — Failure & Scale Scenarios.

14. Major Architecture Trade-offs

Decision	Simpler option	When added complexity may be justified
Search engine	Database-native full-text search	High QPS, sophisticated ranking/facets, independent scaling
Autocomplete	Search-engine prefix queries	Dedicated suggestion index for latency isolation or specialized ranking
Prefix representation	Edge n-grams	Trie/FST or precomputed suggestions for specific memory/update/latency goals
Change propagation	Batch import or polling	CDC or reliable domain events for tighter freshness
Ranking	Lexical relevance only	Re-rank bounded candidates for richer relevance signals
Cache	Cache common final responses	Cache candidates/components when full responses have low reuse
Sharding	Hash by document ID	Routing by tenant/domain when queries can exploit it
Pagination	Shallow offset	Search-after for deep pages; snapshot for stable multi-page views
Shard failure	Fail the query	Marked partial results when the product tolerates incompleteness

The correct design is the simplest architecture that meets the stated requirements. A trie, CDC pipeline, separate ranking service, and multiple caching layers should each have a reason to exist. Adding all of them by default is not a sign of a stronger design.

See the full Trade-offs & Decisions discussion for the alternatives behind these choices.

15. Search & Autocomplete System Design Interview Questions

Why not use the primary database for search?

For small workloads, we may. At higher scale, a dedicated index provides search-specific structures and independent scaling. Avoid claiming SQL databases cannot perform full-text search.

What's the difference between an inverted index and a trie?

An inverted index maps terms to matching documents. A trie organizes prefixes for efficient completion lookup. They answer different questions, and autocomplete does not necessarily require a trie.

How do you handle typos?

Use bounded fuzzy matching, n-grams, or spelling correction where appropriate. Keep exact identifiers exact and prevent broad short-prefix expansions.

How does a new product become searchable?

A committed source change reaches an indexer through CDC or reliable events, is applied with version checks, and becomes visible after the search engine's refresh. The process is usually eventually consistent.

What happens if indexing stops for 30 minutes?

Search can continue serving stale results. Recovery requires the missing events to remain available and the indexer to catch up faster than new events arrive. If event history is lost, reconciliation or reindexing is necessary.

Why use BM25 and then a re-ranker?

BM25 gives an efficient lexical baseline. A bounded re-ranking stage can use richer signals without scoring the entire corpus with an expensive model.

How do you prevent autocomplete from overwhelming the backend?

Debounce, ignore stale responses, enforce minimum prefix lengths where suitable, cache common prefixes, and apply server-side rate and complexity limits.

What happens when a shard is down?

Try a healthy replica. If no copy is available, either fail the query or return results explicitly marked as partial, depending on whether incompleteness is acceptable.

How do you change analyzers without downtime?

Build a new index from a snapshot, replay intervening changes, validate, switch the serving alias, and retain the old version for rollback.

Why not use OFFSET for page 500?

Deep offset work grows with depth and results can shift between requests. Search-after avoids repeated skipping; point-in-time semantics can provide a stable view when required.

For a broader question bank, see Search & Autocomplete — Interview Perspective.

Final Takeaways

A reliable search system separates the source of truth from the search-optimized view, and separates candidate retrieval from ranking. Autocomplete introduces another latency-sensitive workload whose candidate generation, ranking, and request controls deserve explicit design decisions.

The most important production details are easy to overlook: compatible text analyzers, version-aware indexing, safe delete handling, refresh lag, correct cache keys, bounded expensive queries, shard failure semantics, and rebuilds that do not destroy the live index.

None of these mechanisms is a universal prescription. Start with the actual search experience, traffic, freshness, and correctness requirements; introduce complexity only where those requirements justify it.

References and Further Reading

Designing a Notification System: Queues, Retries, Rate Limiting & Scale

MANGESH MANDLIK — Thu, 08 Oct 2026 10:44:37 +0000

The difficult part isn't sending an email. It's making sure a slow provider, a viral campaign, or a retry storm cannot take down the services that need to send one.

Imagine an e-commerce platform just completed an order. The order service needs to send a confirmation email, the payment service needs to send a receipt, and a security service may need to send an urgent login alert.

Now imagine the email provider starts taking eight seconds to respond.

If those services call the provider synchronously, an external notification dependency suddenly becomes part of the checkout and login latency budget. If a promotional campaign simultaneously produces millions of messages, critical security alerts can end up behind a mountain of marketing traffic.

A production notification system has to solve a more interesting problem than sending messages: accept notification intent durably, then deliver it independently at the rate each downstream provider can sustain.

Let's build that architecture from first principles, and examine where seemingly sensible implementations fail.

1. Requirements: What Are We Designing?

Our service accepts requests from many producer applications and delivers notifications through push, email, and SMS.

It should support:

Transactional messages such as order confirmations and payment updates.
Time-sensitive messages such as security alerts and one-time passcodes.
Promotional campaigns, potentially targeting millions of recipients.
Recipient preferences, opt-outs, quiet hours, and valid destination checks.
Delivery status, retries, and investigation of failed messages.

The important non-functional requirements are fast producer acknowledgment, durable processing, channel isolation, preferential latency for urgent traffic, and observability.

One distinction defines our API contract:

ACCEPTED does not mean DELIVERED. It means the system has durably recorded the notification for asynchronous processing. Delivery may later succeed, fail permanently, expire, or be suppressed by policy.

That contract is the foundation of the Notification System Complete Design.

2. Capacity Estimation: Ingestion and Delivery Are Different Rates

Consider an illustrative traffic burst:

Metric	Example workload
Incoming notifications	5,000/second
Provider's sustainable send capacity	800/second
Excess arriving during the burst	4,200/second
Backlog after a 60-second burst	252,000 notifications

If incoming traffic drops to zero and the provider continues delivering at 800/second, that backlog alone takes 315 seconds, or about 5 minutes 15 seconds, to drain.

In reality, new traffic may continue arriving, reducing the spare capacity available to clear old messages.

This leads to a rule that is easy to overlook:

A queue absorbs a temporary difference between arrival and delivery rates. It does not manufacture provider capacity.

If 5,000 messages arrive every second indefinitely and the provider can only deliver 800, the backlog grows indefinitely. We eventually need more provider capacity, slower campaign ingestion, traffic shedding, or an explicitly longer delivery window for less urgent notifications.

3. Start With the Simplest Architecture — and Watch It Break

The first implementation might look like this:

Order Service
     |
     v
Notification API
     |
     v
Email / SMS / Push Provider
     |
     v
Response to Order Service

It's simple, but the calling service is now waiting for a provider it does not control.

If the provider slows to an eight-second p99 response time, the producer's request may inherit that delay. If the provider is unavailable, notifications may fail alongside otherwise healthy business operations.

The first meaningful architectural change is not adding ten microservices. It's putting a durable asynchronous boundary between notification creation and delivery.

Producer
   |
   v
Notification API
   |
   v
Durable Queue  ----->  Delivery Workers  ----->  Provider
   ^                         |
   |                         v
Fast acceptance         Retry / failure handling

Now the producer does not wait for the provider. Workers can process messages at a controlled pace, and a temporary provider outage becomes a backlog rather than an immediate failure of every calling application.

But the phrase “durable queue” hides a subtle consistency problem.

4. Durable Ingestion: The Database–Queue Dual-Write Problem

A typical notification API must record the notification's state in a database and publish work to a message broker.

Suppose we implement:

Insert notification into the database.
Publish the notification to Kafka, SQS, or RabbitMQ.
Return success.

What if step 1 succeeds but step 2 fails?

The database says the notification exists, but no worker ever receives it.

Reversing the order doesn't solve it. The broker may accept the message while the database write fails, leaving workers processing a notification with no durable tracking record.

Transactional outbox

A practical solution is to write the notification record and an outbox event in the same database transaction:

Notification API
       |
       v
Single DB transaction
  +------------------------+
  | Notification record    |
  | Outbox event           |
  +------------------------+
       |
       v
Outbox Publisher  --->  Message Broker  --->  Workers

The API can acknowledge once that durable transaction commits, while the outbox publisher continues retrying publication if the broker is temporarily unavailable.

This closes the dual-write gap, but does not guarantee exactly-once processing. A publisher can send an event successfully and crash before marking it published, causing it to publish again after restart. Consumers still need idempotent handling.

The exact failure windows and guarantees are covered in Notification System Architecture Deep Dive — Durable Ingestion.

5. API Design: Accept Intent, Not Delivery Promises

A minimal API might expose:

POST /notifications
Content-Type: application/json
Idempotency-Key: order-8421-confirmation

{
  "recipient_id": "user-123",
  "channel": "email",
  "priority": "normal",
  "template_id": "order-confirmed",
  "template_variables": {
    "order_id": "8421"
  }
}

A successful response returns a notification ID and ACCEPTED status — not DELIVERED.

{
  "notification_id": "notif-9271",
  "status": "ACCEPTED"
}

Other useful endpoints include:

GET /notifications/{id}/status — inspect the lifecycle state.
POST /notifications/batch — accept a campaign definition and return a batch ID without synchronously expanding millions of recipients.

The producer-supplied idempotency key matters. If the producer times out after the API successfully commits, it may retry. The same key should return the existing logical notification instead of creating another one.

6. High-Level Architecture: Isolate Channels and Priorities

A single queue with one pool of workers looks attractive until the email provider fails while push notifications are healthy.

We want the channels to fail and scale independently.

                       Producer Services
                              |
                              v
                       Notification API
                              |
                        Durable Ingestion
                              |
                              v
                   Policy / Channel Router
                              |
                  +-----------+-----------+
                  |           |           |
               Push Queue  Email Queue  SMS Queue
                  |           |           |
               Workers     Workers     Workers
                  |           |           |
             Rate Limiter Rate Limiter Rate Limiter
                  |           |           |
                FCM/APNs   Email Provider SMS Provider
                  \           |           /
                   \          |          /
                    Delivery State & Reconciliation

Each channel can have its own backlog, retry policy, worker concurrency, provider adapter, and rate-limit budget.

The architecture also needs priority-aware scheduling within or across channel queues. We'll address that next.

7. Priority Queues: Security Alerts Must Not Wait Behind Campaigns

Imagine a promotional campaign enqueues 10 million notifications. A security alert arrives one second later.

If everything shares one FIFO queue, that alert could wait behind enormous amounts of lower-value traffic.

Separate priority classes help:

Priority	Typical traffic	Desired behavior
High	Security alerts, payment failures	Preferential low latency
Normal	Order confirmations, shipping updates	Predictable progress
Low	Promotions, newsletters	Can tolerate delay

But a naive rule — always drain high priority before touching normal or low — creates starvation. If high-priority traffic never fully stops, lower-priority notifications may never be served.

Production alternatives include weighted polling, reserved capacity, and aging.

For example, a scheduler might use an illustrative ratio of eight high-priority messages, two normal, and one low per round. The exact weights are workload decisions, not universal defaults.

The objective is to protect urgent traffic while guaranteeing that every eligible class makes progress.

See Trade-offs & Decisions — Strict vs Fair Priority for the alternatives and their costs.

8. Recipient Preferences, Push Tokens, and Templates

Before a notification reaches a provider, the router must decide whether it is eligible to send.

That means checking the recipient's category preferences, channel permissions, opt-out status, quiet hours, and whether the destination is valid.

A promotional email should not be sent simply because it was queued before the user unsubscribed. For delayed messages, cached preferences may need to be checked again close to delivery time.

Push adds another complication: users have devices, and devices have tokens. One user may have several active tokens. A provider-reported invalid token should deactivate that specific destination, not trigger endless retries or disable the user's other devices.

Templates also need versioning. If an order confirmation sits in a queue for an hour and the template changes, should the message use the original or the new content? Referencing a specific template version at creation time avoids silently changing in-flight notifications.

These concerns are explored separately in the Architecture Deep Dive, including recipient eligibility, token lifecycle, localization, and render-before-queue versus render-at-delivery trade-offs.

9. Rate Limiting and Backpressure: More Workers Won't Fix a Provider Quota

Suppose workers are capable of making 5,000 provider requests per second, but the provider account allows only 1,000.

Adding workers does not raise that account limit. It may simply generate more 429 Too Many Requests responses.

The system needs provider-level rate limiting shared across distributed workers.

Two common designs are:

Centralized token bucket: All workers coordinate through a shared rate limiter. This provides a global quota view but adds network calls and contention.
Leased quota: Workers receive bounded slices of the total quota and spend them locally. This reduces coordination overhead but can leave unused quota stranded with idle workers.

When a provider returns 429 or Retry-After, the limiter should adjust rather than repeatedly sending at the same rejected rate.

The queue provides backpressure absorption while workers respect the provider's sustainable throughput. It does not remove the need for admission control when traffic stays above capacity.

For a detailed comparison, see Notification System Trade-offs & Decisions.

10. Retries: Why Exponential Backoff Needs Jitter and Deadlines

A provider outage often triggers a second incident caused by the notification system itself.

Picture thousands of workers receiving failures at the same moment. If they all retry one second later, then two seconds later, then four seconds later, the recovering provider gets hit by synchronized waves of traffic.

This is a retry storm.

Exponential backoff reduces the frequency of retries. Jitter randomizes retry timing so workers don't all return simultaneously.

Without jitter:
  Workers fail together
       | 1s
       +----> retry spike
       | 2s
       +----> retry spike

With jitter:
  Workers fail together
       |  randomized delays
       +--> --> ---> ----> spread-out retries

Not every failure should retry:

Failure	Action
Temporary timeout or provider 5xx	Retry with backoff and jitter
Provider 429	Honor retry guidance; lower send rate
Invalid push token	Deactivate token; do not retry
Malformed destination	Permanent failure or appropriate terminal handling
Maximum retries exhausted	Move to DLQ when appropriate
Notification deadline passed	Expire instead of delivering stale content

Retries should be scheduled using delayed queues or retry topics, not by keeping worker threads asleep.

And every time-sensitive notification needs a deadline. A one-time passcode delivered six hours late is not a successful user experience, even if the provider eventually accepts it.

11. Dead-Letter Queues: Failure Must Be Visible and Recoverable

After a bounded number of attempts, unresolved failures can be moved to a dead-letter queue (DLQ) for diagnosis.

The DLQ is not a trash can. It is an operational tool that preserves context such as the notification ID, error, attempt count, and relevant timestamps.

A growing DLQ should be monitored by channel, provider, error category, and message age, not only by one global threshold.

Replaying messages requires care:

Fix the underlying failure first.
Re-check expiry and recipient preferences.
Replay at a controlled pace.
Retain prior attempt history and observe whether the problem recurs.

Dumping the entire DLQ back into the main queue can recreate the original overload.

See Failure & Scale Scenarios — DLQ Growth and Backlog Recovery.

12. Idempotency: Can We Guarantee Exactly-Once Notifications?

No — not universally across an external notification provider.

There are three different boundaries to protect:

Producer → API: A retried API call should not create a second logical notification. Use a producer idempotency key.
Queue → Worker: Brokers may redeliver messages. Make worker state transitions idempotent.
Worker → Provider: The external side effect can become ambiguous after a timeout or crash.

Consider this sequence:

Worker sends notification to provider
                  |
                  v
Provider accepts request
                  |
                  v
Worker crashes before recording success
                  |
                  v
Broker redelivers the message
                  |
                  v
New worker cannot tell whether it was sent

If the new worker retries, the user might receive the notification twice. If it doesn't, the notification might never arrive.

A pre-send deduplication check cannot resolve this ambiguity because the successful external action was never recorded.

Where supported, provider-side idempotency keys reduce the risk. Persisting provider message IDs and reconciling against callbacks or status APIs also helps.

The honest contract is durable at-least-once processing with strong duplicate reduction, not an unsupported promise of exactly-once end-user delivery.

The full crash-window analysis is in Notification System Architecture Deep Dive — Idempotency & Duplicate Suppression.

13. Provider Failover: A Timeout Is Not a Rejection

Suppose the primary SMS provider times out. Should we immediately send through a backup provider?

Not necessarily.

The primary may have already accepted the request but lost the response. Sending through a second provider can produce two real messages.

A known rejection is different from an ambiguous outcome. Failover policy should consider the signal, message urgency, duplicate tolerance, and the backup provider's capacity.

For a security alert, accepting some duplicate risk might be reasonable to improve the chance of timely delivery. For a promotional notification, conservative retry or reconciliation may be preferable.

Provider failover is a trade-off, not a universal on/off switch. The Trade-offs & Decisions page compares failover versus retrying the primary in detail.

14. Delivery Tracking: SENT Is Not DELIVERED

The system should maintain an observable notification lifecycle:

ACCEPTED → QUEUED → PROCESSING → PROVIDER_ACCEPTED
                                     |
                                     v
                                  DELIVERED

Alternative outcomes:
  RETRY_SCHEDULED | FAILED_PERMANENT | EXPIRED
  SUPPRESSED | BOUNCED | UNDELIVERABLE | DLQ

PROVIDER_ACCEPTED means the provider took the request. It does not prove that the recipient received it.

Providers may send delivery-status webhooks later, but those callbacks can arrive duplicated, delayed, out of order, or not at all.

A robust reconciliation pipeline should:

Authenticate provider callbacks.
Deduplicate callback events.
Apply idempotent, monotonic state transitions.
Prevent a late SENT callback from overwriting DELIVERED.
Query provider status APIs when callbacks are missing and such APIs are available.

Not every channel provides a reliable end-device delivery receipt. The status model must describe what the system actually knows, rather than claim more than a provider can prove.

15. Production Failure Scenarios: Where the Architecture Gets Tested

A notification system's quality becomes clear when something breaks.

Email provider outage

Email workers back off and accumulate a durable backlog. Push and SMS continue independently. Recovery is paced according to provider capacity and message deadlines.

Broker outage

With a transactional outbox, the API can continue durably recording notification intent while the broker is down, subject to database capacity and the API's acceptance contract. Publishing resumes when the broker recovers. Without a durable fallback, the API should not falsely report successful acceptance.

Massive promotional campaign

A campaign targeting millions of recipients should be expanded into bounded fanout chunks, paced to sustainable capacity. Transactional and security traffic must retain protected capacity. The campaign should support cancellation and expiry.

Priority starvation

A strict high-first scheduler can leave normal and low-priority notifications waiting forever. Monitor oldest-message age by priority, not just queue depth, and use fair scheduling or reserved capacity.

Delivery tracker outage

A well-isolated tracker failure should degrade status visibility rather than stop all delivery. But provider IDs and state events must be preserved for later reconciliation; simply losing them creates a permanent audit gap.

Backlog recovery storm

After a long outage, releasing every queued notification at once can overwhelm a recovering provider. Drain gradually, respect priority, and discard or expire messages that are no longer useful.

These are part of the 14 concrete incidents covered in Notification System Failure & Scale Scenarios.

16. How Would We Evolve This Architecture?

A good system design grows in response to evidence, not because a diagram looks more impressive with more components.

Stage	Architecture	Problem it solves
1	Simple notification service	Basic low-volume delivery
2	Durable ingestion and asynchronous workers	Provider latency and failure isolation
3	Per-channel queues, preferences, priority scheduling	Multi-channel correctness and urgent-message latency
4	Shared provider rate limits, retry scheduling, DLQ	Throttling, transient failures, and observability
5	Idempotency, reconciliation, campaign pacing	Duplicate reduction, trustworthy state, and large-scale fanout

The precise order depends on product requirements. For example, recipient opt-outs and basic idempotency may be mandatory from the beginning rather than later enhancements.

17. Notification System Design Interview Questions

Why use a queue in a notification system?

To decouple the rate of notification creation from the rate of provider delivery, keep producer latency independent of provider health, and buffer temporary bursts durably.

Can Kafka or SQS guarantee exactly-once notification delivery?

No. Internal broker semantics cannot eliminate the ambiguous external provider-call window. Use at-least-once processing, idempotent internal transitions, provider idempotency where available, and reconciliation.

How do security alerts bypass promotional notifications?

Use priority classes with protected capacity or fair scheduling, so urgent traffic gets preferential latency without starving other classes indefinitely.

What happens if the provider allows only 1,000 messages per second?

Coordinate a rate-limit budget across workers. Additional workers cannot bypass the provider's account-level ceiling. Pace incoming campaigns and monitor backlog age.

Why do retries need jitter?

Exponential backoff alone can leave workers synchronized. Jitter spreads attempts across time and reduces retry spikes during recovery.

Should we fail over to a second provider after a timeout?

Only after considering whether the outcome is ambiguous, how urgent the notification is, and whether duplicate delivery is acceptable.

How do we send 100 million promotional notifications without delaying OTPs?

Accept the campaign quickly, enumerate recipients asynchronously in bounded chunks, pace fanout, and reserve downstream capacity for time-sensitive traffic.

What is the difference between SENT and DELIVERED?

SENT or PROVIDER_ACCEPTED means the external provider accepted the request; DELIVERED requires a stronger provider or downstream confirmation, where supported.

For more advanced follow-ups on outbox consistency, webhook ordering, push-token invalidation, and dedup-store failures, see the Notification System Interview Perspective.

Final Thoughts

The hardest part of a notification system isn't choosing Kafka over RabbitMQ or integrating an SMS SDK.

It's recognizing that notification creation, delivery, provider acceptance, and confirmed receipt are different events with different guarantees.

A durable queue protects producers from provider slowness. Channel isolation contains failures. Fair scheduling protects critical traffic. Rate limiting respects real provider ceilings. Backoff, jitter, deadlines, and DLQs make recovery controlled rather than chaotic. Idempotency and reconciliation reduce duplicate risk without pretending external side effects can be made universally exactly-once.

The architecture becomes easier to reason about when each component exists to solve a specific failure mode.

That's the deeper lesson of notification system design: reliability comes from making failure boundaries explicit, not from adding more infrastructure.

References & Further Reading

These SeeItFlow resources explore the same system in more detail:

Notification System — Complete Design — Requirements, API contract, queue-based evolution, channel routing, priorities, retries, DLQ, and final architecture.
Notification System — Architecture Deep Dive — Transactional outbox, three idempotency boundaries, fair scheduling, recipient policy, push tokens, templates, distributed rate limiting, provider adapters, webhooks, campaign fanout, and delivery-state storage.
Notification System — Trade-offs & Decisions — Sync versus async, queue topology, retry policies, rate-limit coordination, provider failover, and template-rendering decisions.
Notification System — Failure & Scale Scenarios — Fourteen operational scenarios covering provider outages, retry storms, queue failures, throttling, duplicate events, starvation, and backlog recovery.
Notification System — Interview Perspective — Interview answer structure, common mistakes, and advanced follow-up questions.
Notification System — Visual Journey.

Explore more free, visual explanations of distributed systems, backend engineering, and system design at SeeItFlow.

Designing a URL Shortener: Architecture, Base62, Redis Caching & Scaling

MANGESH MANDLIK — Thu, 08 Oct 2026 07:51:19 +0000

A URL shortener sounds like one of the simplest backend systems to build.

Take a long URL:

https://example.com/articles/distributed-systems/introduction

Generate a short code:

https://short.ly/aB7xK2p

Store the mapping, and redirect anyone who visits the short URL.

Three operations. One database table. A few API endpoints.

So why is URL Shortener System Design such a useful distributed systems interview problem?

Because generating the short URL is rarely the difficult part.

The interesting engineering begins when a link gets shared with millions of people, thousands of requests hit the same database record, a cache restarts during peak traffic, or the analytics pipeline starts slowing down redirects.

A production-grade URL shortener is fundamentally a problem of read-heavy architecture, uniqueness, caching, consistency, and failure isolation.

Let's design one from first principles.

1. What Are We Actually Building?

Before choosing Redis, Kafka, or a particular database, we need to understand what the system must do.

Functional requirements

Our URL shortening service should support:

Creating a unique short URL from a long URL.
Redirecting visitors to the original destination.
Optional custom aliases, such as short.ly/summer-sale.
Expiring links after a specified duration.
Tracking click analytics, including counts and trends.

We will keep authentication, billing, QR codes, and advanced link-management features outside the initial scope.

Non-functional requirements

The system should provide low-latency redirects, high availability, durable URL mappings, and the ability to absorb sudden traffic spikes.

One requirement is especially important: redirect availability matters more than URL creation availability.

If users cannot create new links for a few minutes, that is inconvenient. If existing links stop redirecting, every application, email, or document containing those links can be affected.

That difference will influence how we design the entire system.

2. Capacity Estimation: Why Reads Dominate the Architecture

Let's establish an illustrative workload.

Metric	Design assumption
Stored URL mappings	100 million
Daily redirects	100 million
Read-to-write ratio	Approximately 100:1
Average mapping storage	500 bytes including overhead
Peak traffic	Up to 100× average redirect rate
Redirect latency	Low milliseconds on the cache-hit path

These are assumptions for our design exercise, not universal URL-shortener benchmarks.

How much storage do we need?

With 100 million mappings at approximately 500 bytes each:

100,000,000 × 500 bytes ≈ 50 GB

That is a manageable amount of mapping data for a modern database deployment, although indexes, replicas, backups, and future growth require additional capacity.

Storage is not necessarily our first scaling problem.

How many redirects must we serve?

100 million redirects per day translates to approximately:

100,000,000 / 86,400 ≈ 1,157 redirects/second

Now imagine a major traffic event pushes peak demand toward 100,000 redirects per second.

If every redirect queries the database, the database repeatedly performs lookups for the same popular links.

The system isn't doing complicated work. It is doing simple, repetitive work far too often.

That distinction matters.

The first architectural optimization should target repeated reads, not immediately introduce database sharding.

For a more complete capacity model, see URL Shortener — Complete System Design.

3. Start With the Simplest Possible Architecture

Our first version needs only three components:

Client
   |
   v
URL Shortener API
   |
   v
Relational Database

The application exposes two core operations:

POST /shorten
Content-Type: application/json

{
  "long_url": "https://example.com/article"
}

Response:

{
  "short_url": "https://short.ly/aB7xK2p"
}

And for redirects:

GET /aB7xK2p

The service looks up the mapping and returns an appropriate HTTP redirect response with a Location header.

Our initial database schema can be remarkably small:

CREATE TABLE url_mappings (
    short_code VARCHAR(50) PRIMARY KEY,
    long_url TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    expires_at TIMESTAMPTZ
);

The primary-key index supports fast lookups by short code and enforces uniqueness.

This architecture is perfectly reasonable for an early-stage product.

There is no reason to operate Redis clusters, distributed ID generators, or multiple database regions before the workload requires them.

But once popular URLs receive large numbers of repeated requests, our database begins doing unnecessary work.

That is the first bottleneck we should solve.

4. How Do URL Shorteners Generate Unique Short Codes?

The short code is the identity of a URL mapping.

We need it to be compact, unique, and ideally difficult to guess if enumeration is a concern.

There are several approaches.

Approach A: Auto-increment IDs with Base62

Suppose the database generates an integer ID.

We convert that integer into Base62, using:

0-9, a-z, and A-Z

This produces compact, URL-safe representations.

Advantages include straightforward uniqueness and minimal collision handling.

The disadvantage is predictability. Sequential identifiers can be enumerated, even when encoded in Base62.

Approach B: Hash the original URL

We could hash the long URL and use part of the resulting value as the short code.

This can support deterministic mappings, where the same input produces the same output.

But truncated hashes can collide.

It also creates a product question: should two different users shortening the same destination receive the same code?

For campaign tracking, ownership, or independent expiry policies, they may need different codes.

Approach C: Random Base62

Generate a random code from the Base62 alphabet.

The available namespace grows exponentially with code length.

Code length	Possible codes
6 characters	56.8 billion
7 characters	3.52 trillion
8 characters	218 trillion

A seven-character code offers a large namespace for our illustrative workload.

But large does not mean collision-free.

With 100 million occupied codes in a seven-character namespace, the probability that the next uniformly random attempt collides is approximately:

100,000,000 / 62^7 ≈ 0.00284%

That is a small per-attempt probability, but collisions will still occur over a sufficiently large number of creations.

The solution is not to pretend collisions are impossible.

It is to make them harmless.

Use a database uniqueness constraint, and retry with a new random code when a collision occurs.

Generate random Base62 code
           |
           v
Attempt database INSERT
           |
      +----+----+
      |         |
   Success   Collision
      |         |
    Return   Generate again
              (bounded retries)

The retry count must be bounded. An unexpected increase in collisions should be observable rather than producing an infinite loop.

Approach D: Snowflake-style distributed IDs

A Snowflake-style generator combines information such as a timestamp, worker identifier, and sequence counter to generate unique IDs without coordinating with the database on every allocation.

It can be useful for very high creation throughput or independent generation across many nodes.

However, it introduces clock-handling and worker-ID management requirements.

And importantly, Base62-encoding a Snowflake ID does not make it unpredictable.

Our choice: For this workload, random Base62 with a database uniqueness constraint and bounded retries is a sensible starting point.

Snowflake is an alternative when decentralized ID allocation becomes a genuine requirement, not an automatic upgrade merely because multiple application servers exist.

For collision mathematics, namespace sizing, and distributed ID mechanics, see URL Shortener — Architecture Deep Dive.

5. Choosing the Database: SQL or NoSQL?

A common system design question is:

Should a URL shortener use PostgreSQL, MySQL, DynamoDB, or another distributed key-value store?

The answer depends on the workload.

Our dominant query is:

SELECT long_url, expires_at
FROM url_mappings
WHERE short_code = 'aB7xK2p';

We need exact-key lookups, durable writes, and reliable uniqueness enforcement.

A relational database such as PostgreSQL or MySQL handles these requirements well.

A distributed key-value database can also be appropriate, especially when horizontal write scaling, global distribution, or multi-region availability becomes a stronger requirement.

But choosing NoSQL solely because the application might eventually serve millions of redirects is not sufficient reasoning.

If most redirects are served from cache, the database never sees most of that traffic.

We should scale the database based on its actual workload, not the public request rate.

There is another important trade-off: database replicas.

Routing cache misses to read replicas protects the primary's write capacity. However, asynchronously replicated data may be temporarily stale.

A short URL created moments ago might exist on the primary but not yet on a replica.

If the cache misses during that window, the system could incorrectly return 404.

Eager cache population after a successful write, read-after-write routing, or stronger replication guarantees can mitigate this, depending on the consistency requirement.

The detailed alternatives are compared in URL Shortener — Trade-offs & Decisions.

6. Redis Caching: The Most Important Scaling Decision

Now we address our main bottleneck.

Without caching, every redirect requires a database lookup.

With Redis, popular mappings can be served from memory.

The request path becomes:

Client
   |
   v
Load Balancer
   |
   v
URL Service
   |
   +----> Redis
   |        |
   |     Cache Hit ----> Redirect
   |
   +----> Cache Miss
              |
              v
         Database
              |
              v
        Populate Redis
              |
              v
           Redirect

This is the cache-aside pattern.

On URL creation, we can also populate Redis immediately after the database successfully commits the mapping.

That reduces the chance that a newly created link experiences a cold-cache miss when its first visitors arrive.

How much database traffic can caching eliminate?

Assume 100,000 redirects per second and a measured or estimated cache-hit ratio of 92%.

Then:

Database reads = 100,000 × (1 - 0.92)

Database reads ≈ 8,000 per second

The cache absorbs approximately 92,000 reads per second.

This illustrates why caching can dramatically reduce database pressure. The actual hit ratio must be measured; it depends on traffic distribution, memory capacity, and expiry policies.

The subtle problem: cache expiry correctness

Suppose a short URL expires in 10 minutes.

But Redis stores the mapping with a fixed TTL of 24 hours.

After 10 minutes, the authoritative mapping is expired. Yet Redis may continue redirecting users for almost another day.

This is a correctness bug, not just a caching inefficiency.

A safer TTL rule is:

cache_ttl = min(
    configured_cache_ttl,
    expires_at - current_time
)

For non-expiring mappings, use the configured cache TTL.

The redirect handler should also check logical expiry when that metadata is available.

Explicit updates or deletions require cache invalidation rather than relying exclusively on TTL expiry.

The complete cache hierarchy, expiry rules, and invalidation mechanisms are covered in the Architecture Deep Dive.

7. What Happens When a Short URL Goes Viral?

Imagine a short link appears in a major social-media post.

Within seconds, hundreds of thousands of people click it.

A warm Redis cache helps because most requests avoid the database.

But a different problem can appear.

Every request for the same short code may hit the same Redis key.

Even in a sharded Redis cluster, that key is generally owned by one shard.

Adding more shards distributes different keys. It does not automatically distribute reads for one extremely popular key.

This is called a hot-key problem.

How do we solve it?

One option is a small process-local cache inside each URL service instance.

                  Load Balancer
                       |
          +------------+------------+
          |            |            |
       Service A    Service B    Service C
          |            |            |
       Local Cache  Local Cache  Local Cache
          |            |            |
          +------------+------------+
                       |
                     Redis
                       |
                    Database

If each application instance caches the extremely popular mapping locally, many requests no longer reach Redis.

Other options include serving reads through cache replicas or using CDN edge caching for genuinely immutable redirects.

Each choice adds different consistency and invalidation considerations.

What if Redis restarts during the viral spike?

Now the cache is empty.

Thousands of requests for the same URL can simultaneously miss and query the database.

That is a cache stampede, also called a thundering herd.

Request coalescing solves an important part of the problem: concurrent misses for the same code share one in-flight database lookup rather than issuing duplicate queries.

TTL jitter and proactive warming of known hot keys can reduce the risk further.

A viral URL, a Redis hot key, and a cold-cache stampede are related but distinct operational scenarios.

See URL Shortener — Failure & Scale Scenarios for the incident-by-incident breakdown.

8. 301 vs 302 Redirect: A Small Decision With Big Consequences

Should a URL shortener return 301 Moved Permanently or 302 Found?

There is no universally correct answer.

A permanent redirect communicates that the destination is intended to remain stable.

A temporary redirect communicates that the destination may change.

The choice affects browser caching, infrastructure load, and analytics visibility.

For example, if a browser caches a permanent redirect, later visits may bypass the URL shortener entirely.

That reduces load on our infrastructure.

But those visits may also become invisible to origin-side click analytics.

If a short URL's destination can be edited, long-lived client caching becomes more problematic because visitors may continue reaching an old destination.

Temporary redirects can keep requests visible to the service, provided the caching headers and client behavior are configured accordingly.

Remember that HTTP caching depends on status code, response headers, and client behavior together.

It is incorrect to assume that every 301 is cached forever or that a 302 can never be cached.

For mutable destinations or products prioritizing click visibility, a temporary redirect posture is often easier to manage.

For immutable, heavily accessed links, permanent redirects with appropriate caching policies may reduce origin traffic significantly.

This decision is explored further in Trade-offs & Decisions — Permanent vs Temporary Redirect.

9. Click Analytics Should Not Slow Down Redirects

A URL shortener may need to record every click for dashboards, reporting, or campaign analytics.

The naive approach is to insert an analytics record before returning each redirect.

But that makes redirect latency depend on the analytics database.

At 100,000 redirects per second, we could generate 100,000 analytics events per second.

Worse, if the analytics store slows down, the redirect path slows down too.

These are two workloads with different requirements:

Redirect serving needs low latency and high availability.
Analytics processing needs high throughput and can often tolerate delayed aggregation.

We should isolate them.

                   URL Service
                   /         \
                  /           \
           Return Redirect   Async Event
                                  |
                                  v
                            Kafka / SQS
                                  |
                                  v
                          Analytics Workers
                                  |
                                  v
                           Analytics Store

The redirect service emits an event through a bounded, asynchronous mechanism.

Workers consume events, batch them, and write them into an analytics store such as ClickHouse.

But asynchronous processing is not automatically safe.

What happens if Kafka or SQS becomes unavailable?

The service must have an explicit policy: drop, sample, or temporarily buffer events within strict limits, depending on whether analytics loss is acceptable.

The redirect should not block indefinitely waiting for analytics.

If the product requires exact, lossless click counts, the design needs stronger event durability and deduplication guarantees.

Another subtle scaling issue: analytics partitioning

If we partition events by short code, all clicks for one viral URL may land on the same partition.

That creates a hot partition even if the rest of the event-streaming cluster has available capacity.

If the analytics workload primarily computes counts and aggregates, distributing events more broadly and merging counts downstream may be preferable.

Strict per-URL event ordering is not always necessary.

The Architecture Deep Dive covers event-delivery guarantees, partitioning alternatives, and duplicate-count handling.

10. The Production Architecture

We can now assemble the architecture based on the bottlenecks we have actually identified.

                     Clients
                        |
                        v
                   Load Balancer
                        |
                        v
              Stateless URL Services
              /          |         \
             /           |          \
       Local Cache     Redis     Analytics Events
             |           |              |
             +-----+-----+              v
                   |                 Kafka / SQS
                   v                     |
             Database Layer             v
              /         \         Analytics Workers
          Primary     Replicas           |
                                        v
                                  Analytics Store

Each component has a specific responsibility.

The load balancer distributes requests across stateless application instances.

The URL services handle validation, short-code generation, cache lookups, expiry checks, and redirects.

Redis absorbs repeated mapping reads, while optional local caches protect extremely hot keys.

The relational database remains the authoritative source of URL mappings and uniqueness.

The event pipeline processes analytics independently.

This is a logical architecture, not a prescription for a fixed number of servers, replicas, or cache nodes.

The goal is not to add as many distributed components as possible. It is to introduce each component when it solves a demonstrated problem.

To understand how the simple database-only design evolves into this architecture, explore the interactive URL Shortener Visual Design Journey on SeeItFlow.

It walks through the architecture's evolution with animated, narrated explanations rather than relying only on a static diagram.

11. Production Failures That Reveal Whether the Design Is Correct

A system design is incomplete until we understand what happens when dependencies fail.

Scenario A: Redis becomes unavailable

All requests that previously hit Redis may fall through to the database.

Redirects can continue if the database has sufficient capacity.

But if it cannot absorb the sudden read increase, a cache outage can cascade into a database outage.

Circuit breakers, bounded database concurrency, request coalescing, and local caching help contain the damage.

Scenario B: The database primary fails

New URL creation becomes unavailable until a writable primary is restored or a replica is promoted.

Existing redirects may continue through Redis and healthy read replicas.

The exact impact depends on the replication topology and the freshness of available cached data.

Scenario C: A newly created URL returns 404

The write succeeded on the primary, but a cache miss reached a lagging read replica.

The replica has not yet received the new mapping.

This is a read-after-write consistency problem.

Eager cache population, primary reads for recently created mappings, or stronger replication guarantees can mitigate it.

Scenario D: Two users request the same custom alias

Both requests check availability and initially find the alias unused.

Both attempt to create it.

An application-level availability check cannot safely resolve this race.

The database uniqueness constraint must arbitrate the conflict atomically. One request succeeds, while the other receives a conflict response.

Scenario E: The analytics broker fails

If event publishing blocks indefinitely, analytics infrastructure can take down redirect serving.

A bounded, explicitly defined producer failure policy keeps the two availability domains separated.

These are only some of the important cases.

The full Failure & Scale Scenarios reference covers Redis failures, cache stampedes, viral spikes, replica lag, namespace pressure, expired mappings, broker outages, and abuse traffic.

12. How Would This Architecture Evolve Over Time?

We should not deploy the final architecture on day one.

A more sensible progression is:

Stage	Architecture	Reason to evolve
1	Application + relational database	Initial product with manageable traffic
2	Add Redis cache	Repeated redirect reads begin stressing the database
3	Stateless services, replicas, async analytics	Application throughput and analytics isolation become important
4	Local hot-key caching and additional capacity	Concentrated viral traffic exceeds shared-cache or application capacity
5	Multi-region reads or writes	Geographic latency, availability, or independent regional write requirements justify the complexity

The trigger for each transition should be a measured bottleneck or a new product requirement.

Multi-region writes, for example, introduce global uniqueness, replication consistency, and cross-region invalidation problems.

A single write region with geographically distributed reads may be sufficient for many products.

Adding active-active writes merely because the system is large would create complexity without necessarily improving the user experience.

13. URL Shortener System Design Interview Questions

Here are some of the most important questions to prepare for.

Why is Redis useful in a URL shortener?

Because a read-heavy workload with skewed access patterns repeatedly requests the same mappings. Redis can serve popular mappings from memory, reducing database load and redirect latency.

How do you prevent duplicate short URLs?

For randomly generated codes, enforce uniqueness at the database layer and retry generation on a conflict. If the product requires deterministic deduplication of destination URLs, that is a separate rule with its own identity and concurrency requirements.

Is Snowflake required for distributed ID generation?

No. Random Base62 with database uniqueness enforcement or a database sequence can work across multiple application instances. Snowflake becomes useful when decentralized generation offers a concrete advantage.

What happens if Redis fails?

Reads can fall back to the database, but the additional load must be bounded. Otherwise, the cache failure can trigger a cascading database overload.

Should the URL mapping database be SQL or NoSQL?

A relational database is a strong starting choice for exact-key lookups and uniqueness constraints. A distributed key-value store becomes more attractive when global distribution or horizontal write scalability justifies it.

How do you handle millions of clicks on one short URL?

Keep the mapping cached, identify hot-key concentration, consider process-local caching or read replicas, and ensure the application tier and analytics producer can handle the request volume. The first bottleneck depends on the actual deployment.

How do you ensure expired links stop redirecting?

Enforce logical expiry on reads and ensure cache entries cannot outlive the authoritative mapping's expiry. Background row deletion alone is insufficient.

For a much larger collection of follow-up questions and reasoning strategies, see URL Shortener — Interview Perspective.

Final Thoughts

A URL shortener teaches an important lesson about distributed systems.

The difficult part is not generating a seven-character string or writing a database query.

It is understanding how a small operation behaves when repeated millions of times under unpredictable conditions.

The strongest architecture emerges from a sequence of questions:

Why is the database doing the same lookup repeatedly?

What happens if our cache disappears?

Can a recently created URL temporarily return 404?

Does our analytics pipeline have the power to break redirects?

Are we solving a real scaling problem, or adding infrastructure because it looks sophisticated?

Answering these questions leads to better architecture than memorizing a diagram.

And that reasoning applies far beyond URL shorteners — to API gateways, caching systems, distributed databases, event-driven platforms, and other read-heavy services.

References & Further Reading

The following SeeItFlow resources explore the design in greater depth:

URL Shortener — Complete Design — Requirements, capacity estimation, ID generation, caching, data model, final architecture, and production evolution.
URL Shortener — Visual Design Journey — Interactive, narrated architectural evolution from a simple database-backed service to a distributed redirect system.
URL Shortener — Architecture Deep Dive — Base62 namespace sizing, collision handling, Snowflake IDs, cache hierarchy, hot-key protection, database replication, and analytics partitioning.
URL Shortener — Trade-offs & Decisions — Alternative approaches and selection criteria for databases, redirects, caching, ID generation, and analytics.
URL Shortener — Failure & Scale Scenarios — Twelve production failure and scaling scenarios, including triggers, detection, mitigation, and recovery.
URL Shortener — Interview Perspective — Interview preparation, architecture reasoning, common mistakes, and advanced follow-up questions.

All six resources are part of SeeItFlow, a free visual software-engineering learning platform covering system design, distributed systems, backend engineering, databases, networking, and modern AI systems.