You cannot score ten million videos for every user on every page load. That single constraint explains almost the entire architecture of a modern recommendation system. The solution is a funnel: cheaply narrow ten million items down to a few hundred, then spend real compute ranking only those.
The core problem
A good recommendation is personal and fresh, which means it has to be computed close to request time. But a heavy ranking model that considers hundreds of features might take a millisecond per item. Run that over your full catalog and a single page load takes hours. So recommenders split the work into two stages with very different budgets.
Stage one: candidate generation
The first stage answers a loose question: out of everything, which few hundred items are even plausibly relevant to this user? It must be fast and it is allowed to be approximate. You are not trying to get the order right, you are trying to not miss good items while throwing away the obvious junk.
The dominant technique is embedding-based retrieval. You train a model that maps each user and each item into the same vector space, so that a user vector sits near the item vectors they are likely to enjoy. Recommending then becomes a geometry problem: find the item vectors closest to this user's vector.
Doing that exactly means comparing the user vector against every item vector, which is the expensive scan we were trying to avoid. This is where approximate nearest neighbor search comes in. An ANN index, built with an algorithm like HNSW (a navigable graph of vectors) or IVF (clustering vectors into buckets you search selectively), finds the closest vectors without checking all of them. You trade a small amount of accuracy for a massive speedup, turning a full scan into a few milliseconds. In practice you also run several candidate generators in parallel, one from embeddings, one from what is trending, one from items similar to the user's recent activity, and merge their outputs.
Stage two: ranking
Now you have a few hundred candidates and a much larger per-item budget. The ranking model is where the expensive, accurate scoring happens. It takes rich features (user history, item metadata, context like time of day and device, and interactions between them) and predicts the probability of the outcome you care about, often the chance the user clicks, watches, or buys.
Because the candidate set is small, you can afford a heavy model here, frequently a gradient-boosted tree or a deep network. The output is a single score per candidate, and you sort by it. Many systems add a third light re-ranking pass on top to enforce business rules and diversity, so the final list is not ten near-identical items.
The trade-off that shapes everything
The split exists because retrieval optimizes for recall (do not lose good items) and ranking optimizes for precision (order the survivors correctly). Trying to do both in one stage forces an impossible compromise: either the model is too slow to run over the catalog, or too weak to rank well. Separating them lets each stage use the right tool. The cost is a two-model system to train, serve, and keep in sync, and a subtle failure mode where a great item never gets ranked because retrieval dropped it.
Freshness is the other constant tension. Item embeddings can be precomputed in batch, but a brand-new item has no interaction history, the cold-start problem. Systems handle it by falling back to content features (the item's own attributes) until enough behavior accumulates to learn a good embedding.
How the real systems do it
YouTube's recommender is the canonical example of this two-stage design, with a candidate generation network feeding a separate ranking network. Instagram, TikTok, and Spotify all follow the same funnel: cheap approximate retrieval over the full catalog, then heavy precise ranking over a few hundred survivors. The vector index at the retrieval stage, whether FAISS, ScaNN, or a managed vector database, is what makes serving personalized results at scale possible at all.
I wrote the full breakdown, with diagrams and the data model, here: https://www.systemdesign.academy/interview/design-recommendation-system
Top comments (0)