Part of the SeeItFlow series: System Design — Beyond the Architecture Diagram.
How does a recommendation system select 20 relevant videos, products, or posts from a catalog containing millions of items—without scoring the entire catalog on every request?
The answer is not simply “use machine learning.” A production recommendation system is a low-latency retrieval and ranking platform supported by event ingestion, user profiles, feature pipelines, caches, experimentation, and safeguards against stale or ineligible results.
This article develops that architecture from a simple trending feed to a personalized, production-ready design. The examples and capacities are illustrative engineering assumptions, not claims about the internal systems of YouTube, Netflix, Amazon, or any other company.
For interactive diagrams and the full six-part walkthrough, see Recommendation System Design on SeeItFlow.
1. Requirements: What are we actually building?
Suppose we are designing recommendations for a video or commerce platform. Given a user, a recommendation surface (home feed, related items, next-watch), and request context, return a bounded, ordered list of relevant, eligible items.
The system should:
- Personalize results when meaningful behavioral history exists.
- Work for anonymous users, new users, and new catalog items.
- React to recent user actions without rebuilding the entire long-term profile synchronously.
- Avoid showing deleted, restricted, unavailable, or otherwise ineligible items.
- Log which items were returned and which were actually exposed, not just which were clicked.
- Degrade to safe, useful recommendations when personalization services fail.
Important nonfunctional requirements include latency, availability, freshness, relevance, cost, privacy, and the ability to explain quality regressions even when HTTP error rates are zero.
A recommendation system is surface-specific. Home-feed recommendations, “similar products,” and next-video autoplay may use different signals, objectives, freshness budgets, and eligibility rules. Do not assume one universal ranking policy is appropriate for every surface.
2. Estimate the workload before choosing components
Consider an example with 60 million daily active users, each requesting recommendations 25 times per day:
Daily recommendation requests = 60M × 25 = 1.5B
Average QPS = 1.5B / 86,400 ≈ 17,361
Illustrative 10× peak ≈ 174,000 QPS
If each uncached request scores 400 candidates, the theoretical peak demand is approximately 69.6 million candidate scores per second across the fleet. Caching, cheaper pre-ranking, and partial fallbacks can reduce actual main-model scoring volume; the multiplication is a sizing upper-bound scenario, not an assertion that all requests invoke the full ranker.
Now suppose the application ingests 120 behavioral events per active user per day:
Behavior events/day = 60M × 120 = 7.2B
Average events/sec ≈ 83,333
If those 120 events exclude rendered impressions, the impression stream must be sized separately. For example, a feed returning 20 items does not mean all 20 were visible to the user. Returned items and actually rendered/exposed items must not be counted interchangeably.
These numbers immediately explain the architecture: retrieve hundreds, not millions; batch feature reads; avoid unnecessary model execution; and ingest feedback asynchronously.
3. High-level architecture
CLIENTS (web / mobile / TV)
|
Recommendation API
|
+--------------+--------------+
| |
Response Cache User / Session
| Context Lookup
| |
+--------------+--------------+
|
Candidate Retrieval
/ | \
Popularity Collaborative Embedding/ANN
\ | /
+------ Merge & Dedup -----+
|
Early Eligibility Filter
|
Batched Feature Hydration
|
Pre-ranker (if needed)
|
Main Ranker
|
Re-ranking / Diversity Rules
|
Final Eligibility Recheck
|
Response + Continuation
Events / impressions -> Durable log -> Stream + batch jobs
| |
Session overlay Profiles / features
| |
Online serving <- Published indexes/models
The diagram separates two kinds of work. Offline and streaming pipelines prepare reusable data. The online serving path combines that data with the current request and produces the final result within a deadline.
4. Start simple: trending recommendations
A first version can return regional or category-level popular items from a precomputed table. This is cheap to cache, easy to operate, and useful when the system knows nothing about a user.
But a trending-only feed has clear limits: it cannot distinguish a sports fan from someone interested in databases, and it may repeatedly expose the same popular items while hiding the long tail.
Treat popularity as a candidate source and fallback, not as a substitute for personalization.
5. Event ingestion: personalization begins with trustworthy signals
Clients and backend services emit events such as impressions, clicks, watch duration, skips, saves, hides, and purchases. A typical event envelope contains:
{
"eventId": "evt_9012",
"userId": "user_42",
"itemId": "video_81",
"eventType": "watch_complete",
"surface": "home_feed",
"requestId": "rec_730",
"eventTime": "2026-10-27T10:15:00Z",
"schemaVersion": 2
}
The collector authenticates where applicable, validates the schema, applies abuse controls, and durably records accepted events before acknowledging their ingestion. Downstream consumers can then process events asynchronously.
At-least-once delivery and duplicate events
Network retries, broker redelivery, and operational replay can deliver the same logical event multiple times. Assign a stable event ID and make consumers idempotent. Deduplication must cover the required replay horizon, or aggregates must be rebuildable from deduplicated durable history. A short-lived recent-ID cache alone cannot guarantee correctness across an arbitrarily old replay.
Do not claim exactly-once business effects simply because the message broker offers exactly-once-related features. The end-to-end effect still depends on consumer state updates, transactional boundaries, and external stores.
Event time is not processing time
A mobile device can report a watch event minutes late. Time-window aggregations should distinguish when the action occurred from when it arrived, with watermarks or explicit late-event policies and a defined correction strategy. There is no free global event ordering across users or devices.
Returned is not rendered
A recommendation API may return 20 items, but the user might only scroll far enough to see four. Maintain distinct signals for:
- Generated: retrieved by a candidate source.
- Returned: included in the API response.
- Rendered/exposed: actually shown under the product's defined visibility criteria.
- Consumed/acted upon: clicked, watched, purchased, or otherwise engaged with.
A server response cannot prove an impression occurred on screen. Client impression reports may themselves be lost or duplicated, so track reporting quality and avoid treating missing client telemetry as proof that no exposure happened.
Without exposure data, “no click” cannot be interpreted reliably: the item might never have been visible. This is a core source of selection bias in recommendation training and evaluation.
6. Build compact user profiles, not unbounded histories
Raw event logs are unsuitable for synchronous scanning on every recommendation request. Instead, maintain a compact user profile containing long-term category affinities, bounded recent activity, and preference summaries.
Separate it from a short-lived session overlay. If a user suddenly watches several cooking videos, a real-time processor can update session intent for the next request without immediately rewriting every long-term preference feature.
A useful design is:
Durable behavior log
|--> Batch / stream aggregates --> Long-term user profile
|--> Fast signal processor -----> Session overlay (short TTL)
Request-time context + profile + overlay --> Retrieval and ranking
Session signals should have bounded influence and expiry. One accidental click should not permanently redefine a user's interests. Also distinguish ordinary relevance preferences from privacy, consent, policy, and eligibility state: the latter cannot safely depend solely on an old profile snapshot.
7. Candidate generation: millions of items become hundreds
The candidate stage is a retrieval problem. Its job is to find a manageable set of plausible items quickly, not to make the final ranking decision.
Typical sources include:
Popularity and trending. Regional or category-level items with recent engagement. Strong for cold start and fallbacks, but biased toward widely exposed items.
Collaborative retrieval. Items associated with similar interaction patterns among users or sessions. Useful when behavioral history is rich; weaker for new users and items.
Content-based retrieval. Items similar in topic, attributes, text, creator, or other metadata. Useful for item cold start when behavioral history is missing.
Embedding and approximate nearest-neighbor (ANN) retrieval. Encode users, sessions, or queries and items into vector representations; retrieve nearby item vectors using a versioned index. ANN trades exact-neighbor recall for faster retrieval at scale. It is one retrieval strategy, not the entire recommender.
Sources can run concurrently with independent deadlines and quotas. Each source should return item IDs, source provenance, index version, and optionally a source-local score. Raw scores from different retrieval models are not inherently comparable. Merge by deduplicating IDs, applying source quotas or calibrated fusion logic, and preserving provenance for downstream features.
Example:
Popularity source: 120 candidates
Collaborative source: 200 candidates
Embedding/ANN source: 250 candidates
Content source: 100 candidates
----------------
Before deduplication: 670 entries
After merge and dedupe: bounded candidate set
The exact counts are examples. They are not guaranteed to sum to the final unique count because sources overlap and filters remove items.
Failure rule: one slow retrieval source should not hold the entire feed hostage. Enforce timeouts and continue with healthy sources or an approved fallback.
8. Eligibility filtering must happen before and after ranking
An item can be relevant and still be forbidden to show. Examples include deleted content, age-restricted media, blocked creators, region restrictions, inventory unavailability, or items the user is not entitled to access.
Perform an early filter to avoid wasting ranking work on obviously ineligible items. Then perform a final eligibility check before returning results, including cached recommendations.
Why twice? Candidate indexes and cached lists may lag behind catalog updates. A product can go out of stock after candidate generation; a video can be removed while an older index still references it.
The final check must consult a source with freshness and consistency guarantees appropriate to the policy. If that source is unavailable, fail closed for sensitive restrictions rather than silently treating an unknown state as allowed. Where strong real-time guarantees are required, the serving system needs an authoritative or appropriately synchronized eligibility path; merely lowering a cache TTL is not sufficient.
A final recheck narrows the stale-data risk but does not magically make all concurrent changes impossible. Define the required consistency boundary and ensure the action endpoint (such as purchase or playback) enforces its own authorization and availability rules.
9. Feature store: the bridge between offline learning and online serving
Ranking needs more than user IDs and item IDs. Example features include recent category affinity, item popularity in a time window, item age, user-item similarity, device context, and recent negative feedback.
A feature platform usually spans three related concerns:
- Definitions and metadata: names, types, transformations, ownership, freshness, and schema versions.
- Offline feature history: reproducible features for training, backfills, and evaluation.
- Online feature serving: low-latency access to the features needed for inference.
A feature store is not necessarily one physical database, and centralizing definitions does not automatically eliminate training-serving skew.
Point-in-time correctness
Suppose a model learns from a click that occurred on Monday. Its training example must not include an item-popularity count that incorporates Tuesday's clicks. That would leak future information into training and inflate offline evaluation results.
Point-in-time joins must select feature values that were available at the prediction time, accounting for both event time and feature publication/availability time where relevant. Version and timestamp the data used for reproducibility.
Batch feature hydration
Fetching features one candidate at a time creates excessive network fan-out. Batch or multi-get the feature bundles needed for the bounded candidate set, use well-defined defaults for missing features, and check that feature versions are compatible with the selected model.
10. Ranking: why a retrieval score is not the final answer
Retrieval identifies plausible items; ranking estimates their usefulness for a particular user, surface, and moment.
A basic ranking pipeline:
400 retrieved candidates
|
Batch feature fetch
|
Optional cheap pre-ranker
|
Expensive main ranker
|
Diversity / repetition controls
|
Final eligibility check
|
Top 20
At smaller scale, one ranker may be sufficient. Introduce a cheaper pre-ranker when the main model's cost or latency makes scoring all retrieved candidates impractical. The trade-off is that the pre-ranker may discard items the richer model would have liked.
The ranking objective may combine engagement, satisfaction, conversion, long-term retention, or other product outcomes, subject to policy constraints. These objectives can conflict. Optimizing only short-term clicks may reward sensational or repetitive results while reducing long-term satisfaction.
A model score is not automatically a calibrated probability. Depending on the training objective, it may be a relative ordering signal, an uncalibrated logit, or a calibrated estimate of a defined outcome. Do not compare scores across model versions as if they share a universal scale unless calibration and comparability are established.
Post-ranking and diversity
After model scoring, re-ranking can cap repeated creators, diversify categories, suppress recently consumed items, and reserve a controlled number of exploration slots. Such rules should be measured: more diversity can improve discovery but may reduce immediate predicted engagement.
11. Cold start: new users and new items are different problems
New user: no meaningful personal history. Use region-appropriate popularity, contextual information, optional onboarding preferences, and exploration. Personalization can improve as reliable events accumulate.
New item: little or no interaction history. Use metadata/content-based retrieval and carefully controlled exploration to create opportunities for exposure. An item may also receive an embedding derived from content before any interactions occur; therefore, it is incorrect to assume that every new item must wait for behavioral history or a scheduled embedding rebuild.
Track new-item exposure and catalog coverage. Otherwise, the feedback loop can permanently favor already-popular items because those are the only ones receiving training data.
12. Real-time recommendations without recomputing everything
A session overlay lets the system react quickly to recent actions. For example, after a user skips three similar videos, the next feed request can reduce that category's short-term weight.
This is usually near-real-time, not a hard guarantee that every action is reflected instantly. Processing lag, network delay, caching, and concurrent requests can all affect visibility.
Use targeted cache invalidation on strong signals, short-lived overlay state, and freshness SLOs appropriate to each surface. Keep slower long-term aggregation in batch or streaming pipelines based on its actual business need—not because all features must update on every event.
13. Caching and pagination: fast results must still be correct
Cache regional trending lists, hot item metadata, frequently requested features, and, where useful, short-lived personalized recommendation results.
Personalized cache keys must be scoped to the right user/session, surface, locale, relevant context, policy version, and model/index version. Omitting a user-specific dimension can cause a cross-user cache leak, which is a privacy issue, not just a relevance issue.
Use request coalescing and soft TTL refresh to prevent many simultaneous misses from overwhelming retrieval and ranking services. Never let a cache hit bypass the final eligibility check.
Infinite scroll needs session consistency
Page 1 and page 2 may be generated seconds apart, during which item popularity, user activity, or model versions can change. Naively rerunning retrieval can show duplicate items or skip previously ranked ones.
A continuation token can bind requests to a short-lived recommendation snapshot, candidate-set reference, or seen-item state. The token should be tamper-resistant and expire appropriately. It does not have to contain the entire candidate list; it can reference server-side state.
14. Model and index rollouts: silent failures are the real risk
Recommendation failures often return HTTP 200. A new model can be fully available and still show worse content.
For a model rollout, use offline validation, compatibility checks, shadow evaluation where useful, a controlled canary or A/B experiment, progressive expansion, and rollback thresholds based on both system health and recommendation quality.
For an index rollout, build a new version alongside the old one, validate its integrity and coverage, then publish it through a versioned pointer or explicitly controlled gradual rollout. Retain the prior good version for rollback. An atomic pointer update in one metadata store does not imply every serving process worldwide instantly switches at the same moment; use version pinning and rollout visibility to manage mixed versions safely.
A/B assignment should be sticky and based on stable identifiers where appropriate, and the assigned variant must be propagated into exposure and outcome logs. Guard against sample-ratio mismatches, missing telemetry, and nonrandom logging loss before trusting experiment conclusions.
15. Failure scenarios worth designing for
| Failure | User-visible or silent impact | Safe response |
|---|---|---|
| Ranking service overloaded | Slow feed, timeouts | Reduce candidates, use cheaper model, serve safe cached fallback |
| One candidate source times out | Lower retrieval coverage | Respect deadline; merge healthy sources |
| Feature store unavailable or incompatible | Generic or incorrectly scored items | Use known-compatible defaults/snapshots; never bypass hard policy checks |
| Event ingestion lag | Recent actions do not influence feed | Track event age; use bounded session state when available |
| Bad model or index rollout | Worse relevance with HTTP 200 | Detect via quality metrics; roll back version |
| Deleted or restricted item in stale cache | Correctness or policy breach | Mandatory current-state eligibility check; fail closed for sensitive rules |
| Duplicate events | Inflated popularity/profile signals | Stable IDs, idempotent updates, replay-safe reconstruction |
| Trending item creates hot key | Counter and cache pressure | Pre-aggregate, shard hot counters, coalesce reads |
| One source dominates retrieval | Narrow, repetitive feed | Monitor source share; quotas and diversity controls |
| Cross-user cache-key collision | Privacy leak | Strict key scoping, tests, purge and incident response |
Quality monitoring must go beyond p95 latency and error rate. Track candidate counts, per-source contribution, feature freshness, fallback rate, rendered impressions, exposure concentration, new-item coverage, repeat-item rate, and experiment outcomes. A system that is fast and always returns 200 can still be failing its users.
For more detailed incident walk-throughs, see Recommendation System Failure & Scale Scenarios.
16. Key architectural trade-offs
Popularity vs. personalization: popularity is cheap and resilient; personalization improves relevance but needs behavioral state and more online work. Use a deliberate blend.
Precomputation vs. freshness: compute reusable embeddings, features, and indexes offline; reserve request-time work for context-dependent decisions. Use streaming only where freshness earns its cost.
Single-source vs. multi-source retrieval: one source is simple; several sources improve coverage and resilience but introduce merge complexity and extra fan-out.
Exact search vs. ANN: exact vector search can be appropriate for small catalogs or offline evaluation; approximate retrieval becomes attractive when exact search exceeds the online budget. Benchmark recall, latency, memory, and update behavior for the chosen workload.
Single-stage vs. multi-stage ranking: a pre-ranker reduces expensive model load but may eliminate valuable candidates too early.
Relevance vs. exploration: serving only the current highest-scoring items reinforces exposure bias; exploration creates information about underexposed content, at a measurable short-term cost.
Eventual freshness vs. hard correctness: most relevance signals tolerate bounded staleness; policy, authorization, deletion, and inventory-sensitive decisions need stronger checks appropriate to their semantics.
For the reasoning behind these choices, read Recommendation System Trade-offs & Decisions.
17. How to explain this in a system design interview
Start with the surface and the objective. A “recommendation system” is not one fixed product requirement.
Then evolve the architecture in this order:
- Trending baseline: deliver a useful list without personalization.
- Event collection: record durable actions and exposure context.
- Profiles: summarize history into fast-serving state.
- Candidate generation: narrow the catalog using multiple bounded retrieval sources.
- Ranking: score only a manageable candidate set.
- Feature store: make online inference and offline training consistent and reproducible.
- Real-time overlay: respond to current-session intent.
- Production safeguards: enforce eligibility, caching discipline, versioned rollouts, experimentation, and degraded modes.
Interviewers often ask:
Why not rank every item? Because the cost of expensive per-item inference across a huge catalog exceeds the latency budget. Retrieval narrows the search space first.
Why log impressions rather than just clicks? Because an unclicked item may never have been exposed. Without exposure data, negative labels and offline evaluation can be biased.
What happens when the feature store is down? Fall back to known-compatible features or a cheaper model where safe; independently enforce policy and eligibility using an appropriate current-state source.
How do you handle new items? Metadata/content retrieval, content-derived embeddings where available, and controlled exploration rather than relying solely on historical engagement.
How do you detect a bad ranking model? Monitor controlled experiments and quality metrics, not only API errors and latency; retain a fast rollback path.
See the Recommendation System Interview Perspective for a larger question bank.
18. Final takeaway
The core of recommendation system design is a sequence of bounded decisions:
Collect reliable signals → build compact state → retrieve plausible candidates → hydrate compatible features → rank and diversify → recheck eligibility → log actual exposure → learn and improve.
Machine-learning models matter, but so do event semantics, feature freshness, latency budgets, cache isolation, model compatibility, exposure bias, and safe degradation. Neglecting any of those can produce recommendations that look technically healthy while quietly becoming irrelevant—or incorrect.
Explore the complete visual walkthrough on SeeItFlow
- Complete Design — requirements, sizing, and architecture
- Visual Design Journey — step-by-step evolution
- Architecture Deep Dive — event contracts, features, ranking, and rollouts
- Trade-offs & Decisions — engineering choices and consequences
- Failure & Scale Scenarios — production incident analysis
- Interview Perspective — questions and strong answer directions
SeeItFlow is free to use, with interactive visual explanations and no sign-in required.
Top comments (0)