Most explanations of recommendation systems start with a model and stop there. Collaborative filtering, matrix factorization, maybe a neural network, and a hand wave at the end about how it all gets served. That framing is why so many engineers can describe embeddings fluently and still have no idea what happens between a user opening an app and twenty items appearing on their screen a tenth of a second later.
The interesting engineering is not the model. It is the fact that you have ten million candidate items, a hard latency budget of roughly a hundred and twenty milliseconds, and a model good enough to rank well that is far too expensive to run more than a few hundred times per request. Everything about the architecture, from the two-tower split to the vector index to the feature store, exists to resolve that single contradiction. Once you see it that way, the design stops being a pile of machine learning components and becomes a fairly ordinary distributed systems problem with an unusual constraint.
A recommendation system is not a model with some infrastructure around it, it is a funnel where each stage buys the next stage the right to be expensive.
This piece works the problem the way a design review should. We start with the numbers, because the latency budget and the scoring rate settle the architecture before any model choice does. Then we build the funnel: candidate generation from several independent retrievers, approximate nearest neighbour search over the embedding space, feature retrieval, ranking with a multi-task model, and slate-level reranking. Along the way we deal with the three subproblems that actually separate a working system from a demo: training-serving skew, biased labels, and the feedback loop a system creates by learning from its own output. Every number below is an industry-typical figure chosen to drive the sizing, not a measured metric from any specific company.
Start with the budget, not with the model
A recommendation request arrives when a user opens a feed. The whole response has to be back inside a budget that, for a feed or a homepage, is usually somewhere between one hundred and two hundred milliseconds of server time. Take a hundred and twenty as the working number and split it into the stages that will need it.
Ranking gets the largest slice of a hundred and twenty milliseconds, and the sizing math on the right explains why the vector index fits in memory while the ranker cannot see the whole catalog.
Now put the scale next to it. A catalog of ten million items, two hundred million monthly active users, four hundred million feed requests per day. Four hundred million requests a day averages about forty six hundred per second, and peak traffic runs roughly double that, so call it nine thousand requests per second. If the ranking model scores three hundred items per request, the ranker alone is doing about two point seven million model evaluations per second at peak. That is the number that kills the naive design. If you tried to rank the full catalog, you would need ninety billion evaluations per second, which is not a tuning problem, it is a different universe.
The storage math points the other way and is unexpectedly friendly. Ten million item embeddings at a hundred and twenty eight float dimensions is ten million times a hundred and twenty eight times four bytes, about five point one gigabytes. The entire searchable representation of your catalog fits in the memory of one ordinary machine, with room for the index structure on top. That asymmetry, an enormous scoring cost against a tiny representation cost, is exactly what the architecture exploits. Cheap comparisons against a compact representation cut the field down, and only then does the expensive model get involved.
A funnel that trades breadth for precision
The resolution is a cascade. Each stage takes the output of the previous one, applies a more expensive method to fewer items, and hands a shorter list forward. Cost per item goes up at every step and the count comes down faster, so the total work stays flat while the quality of the decision keeps improving.
Ten million candidates become twenty delivered items across six stages, each one allowed to be more expensive than the last because it sees fewer items.
Concretely, candidate generation reduces ten million to roughly sixteen hundred using methods cheap enough to behave like a lookup. Filtering removes items the user has already seen, items blocked in their region, items that fail eligibility or policy rules, and near-duplicates, taking the list to around nine hundred. That step is bookkeeping, not modelling, and it is where a surprising number of production bugs live. A pre-ranking stage, a small model over cheap features, narrows to about three hundred when candidate volume is high enough to justify it. At three hundred items you can usually skip pre-ranking entirely, and many systems do. The full ranking model then scores those three hundred with hundreds of features each and produces an ordering. Finally a reranking stage applies constraints that only make sense over the list as a whole and delivers twenty items.
The arithmetic that makes this work is worth stating plainly. Divide each stage's time budget by the number of items it sees and you get the cost it is allowed to spend per item. Ranking gets about forty five milliseconds for three hundred items, so it can spend roughly a hundred and fifty microseconds each, which is enough for a real forward pass over hundreds of features. A pre-ranking stage carved out of that same budget gets maybe five milliseconds for nine hundred items, so about five microseconds each, enough for a small model on a handful of cheap features and nothing more.
Run the same division on retrieval and it breaks. Twenty milliseconds spread over ten million items is two nanoseconds each, and no method scores anything in two nanoseconds. That is the real reason retrieval cannot be a scoring stage at all. It has to be a search that never looks at most of the catalog, which is exactly what an approximate index does: it examines tens of thousands of vectors and ignores the rest. Each stage exists precisely because the next one cannot afford its input, and retrieval exists because nothing can afford the full catalog.
Three clocks, one system
The other structural decision is temporal. The pieces of a recommendation system run on wildly different clocks, and the cleanest way to organize the whole thing is by clock speed rather than by feature area.
The online band has a latency budget, the nearline band keeps things fresh in seconds, and the offline band produces the artifacts both of the others consume.
The online band is everything inside the request. Routing, candidate generation, feature fetch, ranking, reranking. It has a deadline, so it contains no training, no aggregation, and no scans. Every component here either reads a precomputed value or runs a bounded forward pass.
The nearline band runs in seconds to minutes. It consumes the event stream of impressions, clicks, dwell times, and hides, and it turns them into things the online band can read cheaply: rolling counters written into the online feature store, refreshed embeddings for new and updated items, and periodic rebuilds of the vector index. Nothing in this band blocks a request, but everything in it determines how stale the online band's inputs are.
The offline band runs in hours to days. It joins events into training data, trains the retrieval and ranking models, evaluates them on held-out periods, and publishes versioned artifacts to a registry. Every arrow between these bands points from slow to fast. The offline band produces artifacts, the nearline band keeps them fresh, and the online band only ever reads. That direction is what guarantees no request ever waits on a batch job, and it is a discipline worth enforcing explicitly rather than assuming.
Candidate generation is many small recall systems
Candidate generation is where most people expect a single clever model and instead find a committee. The reason is that relevance has several independent causes, and no single retriever captures all of them.
Six independent retrievers run in parallel, each optimizing a different notion of relevance, and a blender unions them under per-source quotas.
A learned embedding retriever finds items semantically close to the user's current interest vector. An item-to-item collaborative signal finds things that co-occur with what the user just engaged with, which catches short-term intent that a slow-moving user embedding misses entirely. A social or graph source returns items from accounts the user follows, which is a relevance signal no content model can infer. A freshness source returns recent items with popularity decay, which is the only way anything published in the last hour ever gets seen. A geographic source returns items popular in the user's locale. And an exploration source deliberately samples items with low exposure.
Each of these is allowed to have poor precision on its own. That is the whole point of putting a ranker downstream. What matters at this stage is recall, meaning the probability that an item the user would have loved appears somewhere in the union. An item that no retriever surfaces cannot be recovered later, no matter how good the ranking model is. This is the single most common silent failure in recommendation systems, and it is invisible in every downstream metric because you can only measure performance on candidates you actually produced.
The sources run in parallel, so the latency cost is the slowest source rather than the sum. A blender unions the lists, drops duplicates, and enforces a per-source quota so that one prolific generator cannot crowd out the others. Those quotas are a product lever, not a model output, and they deserve to be tuned deliberately and reviewed like any other policy.
The matrix is almost entirely holes
Underneath the learned retrievers sits a structural fact about the data. Represent the world as a matrix with users as rows and items as columns, and a cell filled in whenever a user interacted with an item. In any real system that matrix is more than ninety nine point nine percent empty.
Users interact with a vanishing fraction of the catalog, so the modelling problem is filling in the unknown cells rather than memorizing the observed ones.
That sparsity is the defining property of the problem, and it rules out entire families of approach. You cannot memorize the observed cells, because they cover almost none of the space you need to score. Whatever method you choose has to generalize across both users and items.
The classical answer is factorization. Approximate the huge sparse matrix as the product of two much smaller dense matrices, one holding a vector per user and one holding a vector per item, chosen so their dot products reproduce the observed interactions. Scoring a user against an item becomes a dot product between two dense vectors of, say, a hundred and twenty eight dimensions. That reduction is what makes everything downstream possible, because a dot product is cheap enough to run at retrieval scale and, more importantly, is a geometric operation that specialized indexes can accelerate.
Pure factorization has a hole in it, and the hole has a name. A brand new item has no interactions, so it has no learned vector, so it cannot be scored at all. The same is true of a brand new user. Cold start is not an edge case in a system where the catalog turns over constantly, it is the daily condition of a large fraction of your inventory, and the fix is to learn vectors from content features rather than from identity alone. Which leads directly to the model that dominates modern retrieval.
Two towers that meet only at a dot product
The two-tower model takes the factorization idea and makes it learnable from features. One network, the user tower, maps user features and request context to a vector. A second network, the item tower, maps item features to a vector in the same space. The two are trained jointly, but they never share weights and they never see each other's inputs. The only place they touch is the dot product at the end.
The user and item networks are deliberately kept apart so that every item vector can be computed offline and only the user vector has to be computed per request.
That separation looks like an arbitrary restriction and is in fact the entire design. Because the item tower depends only on item features, you can run it offline over the whole catalog and store the results. Because the user tower depends only on user features, you run it once per request. Retrieval then reduces to finding the item vectors with the largest dot product against one query vector, which is a geometric search problem with well understood solutions.
Compare that to a model that concatenates user and item features at the input and passes them through a joint network. That model is strictly more expressive, because it can learn arbitrary interactions between the two sides. It is also completely unusable for retrieval, because scoring requires one forward pass per candidate, and you have ten million candidates. The expressive model is exactly right for ranking, where you only have three hundred items. The constrained model is exactly right for retrieval. The same trade appears in almost every large system: you give up expressiveness in exchange for the ability to precompute one side of the comparison.
Training deserves a note. The standard approach uses sampled softmax with in-batch negatives, meaning the other items in the same training batch act as negative examples. This is cheap and effective, but it introduces a bias, because popular items appear in batches more often and therefore get pushed down more often as negatives. The correction is to subtract the log of each item's sampling probability from its logit, which is a small change with a large effect on how much the model over-favours the head of the catalog.
Approximate search is a recall dial
With ten million item vectors stored, retrieval means finding the top few hundred by dot product against the query vector. Exact search means ten million dot products, which at a hundred and twenty eight dimensions is more than a billion multiply-add operations per request. That is too slow, so every production system uses approximate nearest neighbour search, and it is worth understanding what "approximate" costs you.
Both index families expose the same underlying knob, and turning it toward speed quietly removes candidates you will never know you lost.
There are two dominant families. HNSW builds a layered proximity graph where the top layers are sparse and the bottom layer contains every vector. A search enters at the top, greedily hops toward the query, and descends layer by layer, refining as the graph gets denser. It delivers high recall at very low latency, typically above ninety five percent recall in single-digit milliseconds, but the graph links cost memory on top of the vectors, typically adding somewhere between a third and a half again on top of the raw vector data depending on how many neighbours each node keeps. Its search-time parameter, usually called ef_search, controls how many candidates the greedy search keeps alive.
The IVF family partitions the vector space into cells around learned centroids and stores each vector in its nearest cell. A query finds the nearest few centroids and scans only those cells. The parameter, nprobe, is the number of cells to scan. Combined with product quantization, which compresses each vector into a handful of bytes by quantizing subvectors against learned codebooks, IVF makes billion-scale indexes affordable at the cost of some precision in the distance computation.
The important point is not which family you pick. It is that both expose the same underlying trade and that trade is invisible from downstream metrics. An index running at seventy percent recall against exact search is silently discarding almost a third of the candidates it should have returned, and no click-through metric will ever explain that to you, because the missing items were never shown. Measure index recall directly, against exact search, on a held-out set of real query vectors, and treat it as a monitored quantity rather than a launch-day check.
One feature definition, two very different pipelines
Once candidates exist, ranking needs features, and feature infrastructure is where recommendation systems most often fail in ways that are difficult to diagnose. The problem is that the same feature has to be computed twice, in two completely different environments, and the two computations have to agree exactly.
The offline path reconstructs features as of the label's timestamp while the online path serves them in milliseconds, and any disagreement between the two poisons training.
In training, features are computed in bulk over historical events. In serving, the same features are read one row at a time under a few milliseconds of budget. A feature store exists to make those two paths produce identical values from a single definition, generating both the batch job and the streaming job from the same source.
The failure this prevents is called training-serving skew, and the most damaging version of it is temporal. Suppose a feature is "number of clicks this item received in the last twenty four hours." At serving time, that obviously means the twenty four hours before now. If the backfill job computes it by grouping the day's events, it will include clicks that happened after the impression it is attached to. The model then trains on a feature that partially encodes the answer. Offline metrics look excellent, because the leakage is genuinely predictive in the training set. Online performance is flat or worse, because at serving time that information does not exist yet.
The fix is a point-in-time join. Every event carries a timestamp, every feature value carries the time it became valid, and the training join takes the value that was current at the moment the label was produced, never a later one. This is not an optimization, it is a correctness requirement, and it is the reason offline feature stores keep full history with event timestamps rather than just current values. When someone reports an offline improvement that does not reproduce online, skew is the first thing to check and usually the thing you find.
Ranking is one model with several opinions
The ranking model is where the expensive computation goes. It sees a few hundred candidates and a wide feature vector for each, and it produces the scores that determine the ordering.
Hundreds of features feed a shared trunk with several prediction heads, and the product decides afterwards what each predicted behaviour is worth.
The features cluster into groups with different characteristics. User features describe the person: account age, historical engagement rates, topic affinities built from long-run behaviour. Item features describe the content: topic, creator, age in hours, media type, length. Context features describe the moment: device, hour of day, position in the session, network quality. Counter features are rolling aggregates over one hour, twenty four hour, and seven day windows, and they are the ones with the tightest freshness requirements. Embedding features carry the user and item vectors plus their dot product, which lets the ranker reuse retrieval's learned representation.
Then there are cross features, which combine an attribute of the user with an attribute of the item: this user's historical engagement with this creator, or with this topic, or with this media format. These consistently deliver the largest gains, because they hand the model the interaction it would otherwise have to discover from scratch across a sparse space.
Modern rankers are multi-task. Rather than predicting one number, a shared trunk feeds several heads, each predicting a different behaviour: probability of click, probability of a long dwell, probability of an explicit positive like a save or a like, probability of a negative like a hide or a report. Sharing the trunk lets rare labels benefit from the representation learned on abundant ones. It also separates two things that should be separate. The model predicts what will happen. The product decides what those outcomes are worth, by blending the head outputs with weights and subtracting a penalty for predicted negatives.
That separation has a practical requirement: the head outputs must be calibrated, meaning a predicted probability of zero point three actually corresponds to a thirty percent rate in reality. Neural network outputs are frequently not calibrated, particularly after training on downsampled negatives, so a calibration step such as isotonic regression sits between the model and the blend. Without it, the weighted combination is arithmetic on incomparable quantities, and tuning the weights becomes guesswork.
Your labels are the model's real architecture
The hardest problem in a recommendation system is not learning from the data. It is that the data is generated by the system itself, and therefore encodes the system's past decisions as much as it encodes user preference.
Implicit feedback is noisy, position distorts what a click means, and easy negatives teach the model nothing it did not already know.
Start with implicit feedback. You do not have ratings, you have clicks, watch times, saves, shares, follows, and hides. A click is weak evidence of interest and gets weaker the more clickbait exists in your catalog. A non-click is barely evidence at all, because the overwhelming majority of non-clicks are items the user never saw, never scrolled to, or scrolled past without registering. Treating every non-click as a negative teaches the model that most of the catalog is bad, which is not what the data says. The practical response is to weight labels by strength, treating a long dwell or a completion as much stronger evidence than a click, and to be careful about which non-events you are willing to call negatives.
Then there is position bias. The top slot of a feed collects clicks nearly regardless of what occupies it. If you train on raw clicks, the model learns the ranking policy that produced the logs rather than the underlying preference. The standard correction is inverse propensity weighting: estimate the probability that a position would be examined at all, and divide each observed click by that probability, so a click at position twelve, where the user rarely looks, counts for far more than a click at position one. Estimating those propensities is itself work, and the cleanest source is a small slice of randomized traffic where the ordering is shuffled, which gives an unbiased view of examination probability at the cost of some engagement on that slice.
Negatives round out the set of problems. A random item drawn from ten million is trivially irrelevant, so the model learns to separate it from a positive with no effort and no useful gradient. The informative negatives are the hard ones: items the current model ranked highly that the user nonetheless skipped. Mining those is what pushes a retrieval model past mediocre. And as noted earlier, in-batch negatives systematically over-represent popular items, so a sampling correction is not optional if you want the model to learn relevance rather than popularity.
Offline metrics filter ideas, they do not decide them
Given all of the above, evaluation is harder than it looks, and the gap between offline and online results is a permanent feature of this domain rather than a sign that something is broken.
Offline lift and online lift agree often enough to be useful and disagree often enough that only the A/B test decides.
Offline evaluation splits into two questions. For retrieval, the metric is recall at k: of the items the user actually engaged with, what fraction did candidate generation surface at all. This is measurable and meaningful, and it is the right way to compare retrieval strategies. For ranking, the metric is usually NDCG or a similar position-weighted measure computed over the logged candidate set. That metric has a built-in ceiling, because it is scored against a candidate list the previous system chose. It can tell you whether you order those candidates better. It cannot tell you anything about items the old system never showed.
This is why offline and online results diverge in specific, predictable ways. A change that improves ordering within the old system's candidate set usually improves both. A change that retrieves a genuinely different set of items often looks flat or negative offline while winning online, because the offline metric penalizes disagreeing with the logs. And a change that improves offline while hurting online is usually overfitting to the logged distribution, which is what deeper retrieval into a poorly calibrated ranker tends to do.
The operational conclusion is to use offline evaluation as a filter and never as a verdict. It is cheap, it runs in minutes, and it correctly kills most bad ideas. The A/B test is the measurement that ships or blocks a model, and it needs to run long enough to get past novelty effects, which typically means at least two weeks and a guardrail set that includes more than the primary metric.
Six stores, each chosen by its access pattern
There is no single database in a recommendation system, and trying to build one is a reliable way to end up with something bad at everything.
Every component here has a query shape narrow enough that the storage engine picks itself.
The vector index is the only component queried by similarity rather than by key, which is why it is a specialized engine rather than a table. It holds the item embeddings, lives in memory, and is sharded by item so that each shard searches its own partition and the results are merged.
The online feature store sits directly on the critical path. It is read by primary key, one row per user and one row per item, with a tail latency requirement of a few milliseconds. That is a memory-resident key-value workload with no scans and no joins, which points at an in-memory store rather than anything general purpose.
The offline feature store answers a completely different question: what was this feature's value at this moment in history. That requires full retention with event timestamps and efficient scans over date ranges, which is a columnar lake partitioned by day.
The event log holds raw impressions, clicks, dwell events, and hides. It is append-only and needs replay, so it is a partitioned durable log rather than a database. The model registry holds versioned model and index artifacts read at deploy time, which is object storage plus a metadata table. And the seen-item filter, which stores the last couple of thousand item ids per user, is deliberately approximate: a Bloom filter in the key-value store, where a false positive simply hides an item you might have shown anyway and the memory saving is worth it many times over.
Reranking scores the list, not the item
The ranker evaluates each item independently. That is what makes it cheap enough to run, and it is also what makes it blind to a whole class of problems. It cannot notice that the top five items are all from the same creator, or that three of them are the same news story from different outlets, or that the user has seen this topic in every session this week.
Constraints that span positions, from creator caps to a reserved exploration slot, can only be applied by a stage that sees the whole list.
Reranking is the stage that operates on the slate. The order of its steps matters. Calibration and blending come first, so that everything downstream is working with one comparable value per item. Near-duplicate collapsing comes next, because if duplicates survive into the diversity step they each consume a slot and the diversity constraint appears satisfied while the user sees the same thing three times. Then come the diversity constraints, expressed over windows of positions: at most two items per creator in any ten, at most three per topic. Then policy and eligibility rules that demote borderline content or enforce advertiser and legal requirements. Then, finally, exploration.
That exploration slot is worth defending explicitly, because it will be the first thing someone proposes cutting. Reserving a position for an item with low exposure costs measurable short-term engagement, and it is the only reliable source of unbiased training data you have. Without it, everything the model learns tomorrow is conditioned on what the model chose to show today, which brings us to the failure mode.
The failure mode is a loop, not a crash
Recommendation systems rarely fail loudly. They fail by slowly optimizing themselves into a smaller and smaller world while every dashboard stays green.
Nothing in this loop is a bug. Each step is correct behaviour, and the composition is a system that recommends a shrinking catalog.
The mechanism is straightforward. The model has more observed data about popular items, so it predicts their engagement with more confidence and ranks them higher. Higher rank means more impressions, which means more engagement data, which means those items are even better represented in tomorrow's training set. Meanwhile items that were never shown generate no data at all and never get the chance to prove themselves. Each retraining amplifies the tilt. Nothing errors, nothing alerts, and engagement metrics typically stay flat or improve throughout, because the model is genuinely getting better at the objective you gave it.
The damage shows up in quantities nobody is watching. Catalog coverage, the fraction of items that received any impressions in a period, falls quarter over quarter. The exposure distribution across creators concentrates. New items take longer to find their audience, so creators publish less, so the catalog itself degrades. By the time this surfaces as a business problem it has been running for a year.
The defenses are all about measurement and injection. Measure catalog coverage and the Gini coefficient of the exposure distribution as first-class metrics, reviewed alongside engagement. Guarantee exploration traffic so that some fraction of impressions is not chosen by the model. Correct for popularity in the training objective rather than only in the serving policy. And separate a new item's cold-start period, where it is scored on content features and given deliberate exposure, from its steady state, where it is scored on its own observed performance. None of these are free. Each one costs engagement in the short run and buys a healthier system in the long run, which is exactly the kind of trade that needs an explicit owner or it never gets made.
The trade-offs, and what transfers
Every choice above bought something and cost something. The multi-stage funnel bought a tractable latency budget and cost you the ability to recover a candidate that early stages dropped. The two-tower split bought precomputable item vectors and cost you the expressiveness of early feature interaction. Approximate search bought single-digit millisecond retrieval and cost you a recall percentage you have to monitor deliberately. The multi-task ranker bought shared representation across sparse labels and cost you a calibration step and a set of blend weights that someone has to own. The feature store bought consistency between training and serving and cost you a second pipeline for every feature.
Four decisions from this design, stage by cost, precompute one side, distrust your logs, and watch what your objective ignores, transfer to every ranked retrieval system.
Four lessons transfer beyond recommendations, and you will meet all of them again in search, in advertising, and in any system that ranks a large corpus under a deadline.
First, stage by cost rather than by concern. When a large candidate set meets a fixed budget, split the work into stages where each one is cheap enough for the size of its input and precise enough for the size of its output. The stages are not arbitrary layers, they are the direct consequence of the arithmetic.
Second, constrain the model so that one side of every comparison can be precomputed. The two-tower dot product is a specific instance of a general move, and it is the move that makes real-time retrieval over millions of items possible at all. Whenever you find yourself needing to compare one thing against many, look for the factorization that lets you prepare the many in advance.
Third, assume your data is biased, because it is. Logs record what the previous system chose to show, at what position, to which users. Position bias, exposure bias, and selection bias are properties of the data rather than defects you can train away, and correcting them is usually worth more than the modelling improvement you were going to ship instead.
Fourth, watch what your objective ignores. Any system trained on data it produced itself needs guardrail metrics that look at coverage, diversity, and freshness, because the primary metric is measured on a distribution the system controls. A metric that only ever goes up is not evidence that nothing is wrong.
Get those four right and the specific model you use matters far less than it seems to from the outside. Get them wrong and no architecture will save you.
I teach system design this way, from first principles with real diagrams and the trade-offs that only surface in production, as an interactive course at systemdesign.academy. The foundation lessons are free and need no signup.
Read the free lessons: https://systemdesign.academy
Top comments (0)