DEV Community: RONI DAS

Machine Learning System Design: The Framework That Wins Interviews

RONI DAS — Mon, 20 Jul 2026 02:34:23 +0000

Ask most people to design a machine learning system and they start with the model. They reach for a transformer, or a gradient-boosted tree, or whatever architecture is fashionable that quarter, and they spend the interview tuning it in their heads. That instinct is exactly what the question is built to expose, because in a real ML system the model is the easy part. It is a small box in the middle of a much larger diagram, and it is usually the most replaceable component in it.

A machine learning system design interview is not asking you to invent a new model. It is asking whether you can take a vague business goal, something like show people more relevant posts, and turn it into a system that reliably produces good predictions on live traffic, keeps producing them as the world changes, and does so inside a latency budget measured in tens of milliseconds. The interesting work is almost never the architecture. It is how you frame the task, what you choose to optimize, where the labels come from, how features stay identical between training and serving, how you evaluate offline and confirm online, and how you notice when the whole thing quietly decays.

This piece is the long version of that framework. We will walk it in the order a strong candidate actually walks it: the two loops that make up any ML system, how to frame the problem and pick metrics, the feature store and the skew it exists to kill, how data and labels are where interviews are won or lost, the two-stage candidate-and-ranking pattern, the gap between offline evaluation and online A/B testing, the two ways to serve, and finally the monitoring that stops a live system from rotting in silence. Any specific latency, throughput, or catalog-size number here is an industry-typical range or an illustration for reasoning, not a measured figure from a particular company.

Two loops, one shared clock problem

The backbone of a good answer is seeing that an ML system is two loops that share artifacts but run on completely different clocks.

The offline training loop and the online serving loop share one feature store.

The offline training loop runs on a schedule. It reads raw logged events, impressions, clicks, watches, purchases, from a data lake or warehouse, joins them with the features as they were at the moment each event happened, builds a labeled training set, trains a model, evaluates it offline, and if it passes, registers it in a model registry. This loop is allowed to be slow. It runs hourly, or daily, or weekly, and nobody is waiting on it in real time.

The online serving loop runs on every single request. It takes a live context, fetches the relevant features, generates candidates, ranks them, applies business and policy filters, returns the result, and logs everything it just did. This loop lives on the critical path of a user-facing response, so it is measured in milliseconds.

The two loops meet in the logs. Everything the serving loop shows becomes tomorrow's training data for the offline loop. That is elegant and it is also the source of the subtlest bug in all of ML, which we will get to. For now the point to internalize is that keeping these two loops straight, and keeping their feature computation identical, is what separates an answer that sounds like production from an answer that sounds like a notebook.

Framing: the honest first question is whether to use ML at all

Before any of that, you turn a fuzzy business goal into a precise ML task. And the first honest question is whether machine learning is even warranted.

If the problem can be solved with a rule, show newest first, block known-bad IPs, a heuristic baseline is the correct starting point. It is cheaper, easier to debug, trivial to explain, and it gives you the bar the ML model has to beat to justify its complexity. Saying this in an interview is a strength, not a hedge. It signals that you reach for ML because it beats the baseline on a metric that matters, not because a model is the more impressive thing to talk about.

When ML is warranted, you frame it as a concrete prediction task. Is this classification, will the user click? Regression, how long will they watch? Ranking, order these items by value? Getting this framing explicit shapes everything downstream, from the loss function to the evaluation metric.

Then you separate two kinds of metrics that beginners constantly conflate.

Business goal to ML task to the split between the offline metric and the online metric.

The offline metric is what you train and evaluate against on logged data: AUC, log loss, precision at k, NDCG for ranking. It is fast to compute, repeatable, and it lets you iterate quickly. The online metric is what the business actually cares about: engagement, watch time, revenue, retention. It can only be measured on live traffic. These two do not move together perfectly. A model can improve offline AUC and do nothing, or even harm, the online metric.

The discipline that separates strong answers is refusing to declare victory on an offline metric alone, and always naming the online metric you would confirm the win with. Google's Rules of Machine Learning makes exactly this point: get the infrastructure and the metrics right before you get clever with the model, and be honest about the gap between the metric you can optimize and the outcome you actually want.

The feature store and the skew it exists to kill

If I had to name the single most important piece of ML-specific infrastructure in this design, it would not be the model. It would be the feature store, and it earns that place by solving one specific, damaging bug.

The feature store gives one feature definition to both training and serving, killing train-serve skew.

Train-serve skew is the most common and most expensive failure in production ML. The setup is innocent. A data scientist computes a feature one way in a training pipeline, say average session length over the last 30 days, written in SQL over the warehouse. An engineer reimplements that same feature another way in the serving code, computed on the fly from a slightly different source with a slightly different window. The two definitions drift apart. The model is now trained on one distribution and served another, and quality drops in a way that is completely invisible to offline evaluation, because offline evaluation uses the training-time computation.

There is a subtler cousin, time-travel leakage: joining an event to a feature value that was only known after the event happened. That inflates your offline metrics beautifully and then collapses in production, because at serving time the future value does not exist yet.

A feature store fixes both by making one definition of each feature the source of truth. It has an offline store, usually columnar and in the warehouse, that serves point-in-time-correct values for training, meaning the value as it actually was at event time. And it has an online store, usually a low-latency key-value store, that serves the identical value to the ranker at request time. The training pipeline reads from the offline side, the serving path reads from the online side, and both read the same feature computed from the same definition.

When an interviewer hears you volunteer that you would use a feature store specifically to avoid train-serve skew, it reads as someone who has operated an ML system rather than only trained models. It is one of the cleanest signals in the whole interview.

Data and labels: where interviews are actually decided

The model is only as good as the labels behind it, which is why the data section is usually where these interviews are won or lost. There are three things a strong candidate always covers.

First, where the labels come from. Explicit signals, a purchase, a rating, are clean but sparse. Implicit signals, clicks, dwell time, are abundant but noisy and biased. A click can be a misclick, and a non-click can simply mean the item was never seen. That leads directly to a rule: log impressions, not just clicks. If you only train on what got clicked, your training set is biased toward items the system already favored and contains no real negatives.

Second, leakage. Any feature that encodes information not available at prediction time, a value updated after the event, an aggregate computed over the future, will make offline metrics look great and fail in production. Features must be computed point-in-time-correct, which is again exactly what the offline feature store gives you.

Third, imbalance. Click-through and conversion problems have far more negatives than positives, and a naive model can score well by predicting the majority class while being useless. The fixes matter: negative downsampling with calibration correction, class weighting, and choosing metrics like precision-recall and calibration rather than raw accuracy. Just as important, keep the evaluation set at the true class distribution so you are not fooling yourself.

One more thing that separates people who have shipped from people who have studied: the train/test split for anything time-dependent should be by time, not random. You evaluate on the future the way the model will actually be used, not on a random slice that leaks tomorrow into today.

Candidate generation, then ranking

For recommendation, feed, and search, which is the most common ML design prompt by far, you cannot run a precise model over millions of items inside a latency budget. So the standard solution is two stages with opposite goals.

Two-stage design: cheap high-recall candidate generation, then precise ranking on the shortlist.

Candidate generation optimizes recall and speed. From a catalog of millions, it produces a few hundred items that plausibly belong, using cheap methods: approximate nearest neighbor search over learned embeddings, retrieval from several sources like recent, popular, and similar-to-what-you-engaged-with, and lightweight rule-based filters. It is allowed to be imprecise, because the next stage sorts things out. What it must not do is drop the genuinely good items, because ranking can only reorder what generation hands it.

Ranking optimizes precision. It takes only those few hundred candidates, assembles a rich feature vector for each one, blending user, item, and context features, scores each with a heavier model, and orders them. Crucially, mature systems rank against a blended objective rather than a single click probability, because pure click optimization leads straight to clickbait. Some systems add a third re-ranking stage for business rules, diversity, and freshness.

The reason to volunteer this structure early is that it directly resolves the tension the interviewer is testing: how do you consider everything and still rank precisely inside a tight budget? You cut the millions down cheaply, then spend your compute only on the shortlist. This is the pattern documented publicly in YouTube's recommendation paper, and it shows up in some form across most large ranking systems.

Offline evaluation gates, online A/B decides

Offline evaluation and online testing answer different questions, and treating one as a substitute for the other is a classic mistake.

Offline evaluation filters what you test; the online A/B test decides what you ship.

Offline evaluation, on a held-out set of logged data, tells you whether a model is promising. It is fast, cheap, repeatable, and safe, so you use it to filter many candidate models down to the one or two worth trying on real users. But offline metrics are computed on data the old system generated. You only observe outcomes for items the old system chose to show, so a better offline metric does not guarantee a better user outcome.

Online A/B testing answers the question that actually matters: does this model improve the business metric on live traffic? You route a fraction of users to the new model, hold the rest on the control, run long enough to reach statistical significance, and compare the online metric with proper attention to variance. You also watch guardrail metrics, so you do not celebrate a lift in clicks while quietly hurting retention.

Two rollout patterns bridge the gap safely. Shadow mode runs the new model in parallel without acting on its output, so you can compare predictions with zero risk. Gradual ramp-up limits the blast radius while you build confidence. The rule to state plainly is short: offline evaluation gates what you test, and only an online A/B test decides what you ship.

Serving: batch, online, and the latency budget

There are two ways to serve predictions, and the choice is a genuine trade-off, not a detail.

Batch inference precomputes predictions on a schedule and stores them for lookup. It is cheap, simple, and has no serving-time model latency, which makes it a fine fit when the input space is small and the prediction does not need to reflect the immediate request: daily product recommendations, precomputed user segments. Its weakness is staleness and the inability to react to real-time context.

Online inference computes the prediction on the request, so it reflects the current user, query, and session. The cost is a hard serving-time latency budget, a model that has to be small and fast enough to fit it, and infrastructure to fetch features quickly. Most feed and search ranking is online, because relevance depends on immediate context. In practice many systems compromise: they precompute the expensive parts, embeddings and candidate pools, in batch, and combine them with a light online model. You get freshness where it matters without paying the online cost everywhere.

The latency budget shapes everything upstream. It caps model size, limits how many features you can fetch and assemble, and it is the whole reason the online feature store is a low-latency key-value store rather than the warehouse. When you quote a latency number, frame it as the budget for the model step inside a larger request budget, not as a guarantee. Every added feature is a real tension here: it can lift model quality, but it costs time to fetch and assemble, and that time comes straight out of the ranking budget. Past a point, more features buy diminishing accuracy while eating latency you do not have.

Monitoring: an unwatched model rots in silence

An ML model is trained on a snapshot of the world, and the world moves. Every deployed model decays. The only questions are how fast, and whether you notice before your users do.

Monitor three layers: input distributions, prediction distributions, and the business metric.

Data drift, also called covariate shift, is when the input distribution changes: new users, new items, a marketing campaign that shifts the traffic mix, so the model now sees inputs unlike its training data. Concept drift is when the relationship between inputs and the label changes: the same input now leads to a different outcome because of seasonality, a shift in taste, or a competitor's move. Both degrade quality even though your code never changed.

So monitoring has to watch three layers. Input feature distributions, to catch drift and broken pipelines feeding nulls or defaults. Prediction distributions, to catch the model's output shifting. And the downstream business metric, which is the ground truth but arrives late. Because labels are delayed, drift detection on inputs and predictions is your early warning, the thing that fires before the business metric visibly drops.

The mitigation is retraining on fresh data at a cadence matched to the drift rate. The interview-grade point is that retraining cadence is not a fixed schedule you pick arbitrarily. It is set by how fast your data actually drifts, which is precisely why you monitor rather than guess. Fast-moving signals like trending content may need hourly or streaming updates; stable relationships can be retrained weekly. Chip Huyen's writing on distribution shifts and monitoring lays this out in production terms, and it is worth reading before any senior ML interview.

Two failure modes deserve a special mention because they catch strong candidates off guard. Cold start is making good predictions with little or no history, and it comes in three flavors: new user, new item, and brand-new system, each of which pushes you toward content features, popularity fallbacks, and an exploration budget. And feedback loops are the quiet killer: a ranking system's own output becomes its next training data, so if it only shows what it already scores highly, it only ever learns about those items, and it narrows over time toward a shrinking set of popular content. The fix is to log impressions and non-clicks, reserve an exploration budget, and correct for position bias. Interviewers probe this because it separates people who see ML as a static prediction problem from people who understand a deployed ranking system is a closed loop that shapes the very data it learns from.

The patterns that transfer

Step back from recommendations specifically and the framework generalizes to almost any ML system design prompt.

The transferable patterns: baseline first, separate your metrics, kill skew, evaluate then confirm, monitor for decay.

Start with a baseline before anything complex, so you have a bar to beat. Separate the offline metric you optimize from the online metric you care about, and never confuse a proxy win for a real one. Treat the feature store as the thing that guarantees train-serve consistency, because a feature computed two ways is a silent correctness bug, not a nicety. Use offline evaluation to filter and online A/B tests to decide. Serve within a budget, and let that budget discipline your model size and feature count. And monitor input, prediction, and business drift, because an unmonitored system degrades quietly until the metric everyone watches finally drops.

This is the consistent message across the most cited public sources on production ML. The Hidden Technical Debt in Machine Learning Systems paper is the canonical warning that model code is a tiny box in a much larger system, and that the surrounding infrastructure, data dependencies, feature management, configuration, monitoring, is where the cost and the risk concentrate. Google's Rules of Machine Learning opens by telling engineers to do ML like a good engineer, not like an ML researcher. Both point at the same conclusion the interview is testing for.

Which is the honest reason I keep coming back to the platform side. The model is the part you can swap in an afternoon. The parts that decide whether a system works on live traffic, feature stores, low-latency serving, drift monitoring, safe retraining, are the parts that take real engineering, and they are exactly what a senior interviewer probes hardest. Candidates who stand out treat the model as the easy part and the platform around it as the hard part, because in production that is precisely where the difficulty and the value live. Getting fluent in that platform layer is what turns a model that works in a notebook into a system that keeps working on live traffic, and it is the difference between a plausible interview answer and a production-ready one.

If you want to build that platform intuition properly, I teach ML systems and system design from first principles, with real diagrams and the trade-offs that only surface in production. The foundation lessons are free and need no signup at systemdesign.academy.

Designing a Time-Series Database: Compression, Cardinality, and Retention

RONI DAS — Mon, 20 Jul 2026 02:25:57 +0000

Ask an engineer to design a database and they reach for the tools they know: rows, a B-tree index, a mix of reads, updates, and deletes on individually addressable records. That instinct is correct for most systems and wrong for this one. A time-series database is built around a workload so lopsided that almost every general-purpose assumption inverts. Writes arrive constantly and in enormous volume, they almost always carry a timestamp near now, and once a point lands it is essentially never updated. Reads are heavily skewed toward recent data. Queries are range scans and time-bucketed aggregations, not point lookups. The whole design bends around those facts, and every interesting decision in it falls out of taking them seriously.

This is the long version. We will start from the shape of the workload, build up the series data model and why it matters, walk the write path from the write-ahead log through the in-memory head to immutable on-disk blocks, take apart the compression that makes months of history affordable, be precise about the one failure mode that actually kills these systems in production, and then work through downsampling, retention, and the query engine that turns a tag filter into a fast aggregation. By the end you should be able to defend, in front of a skeptical staff engineer, why a row store struggles here and where a purpose-built engine earns its keep. One note on numbers up front: every throughput, cardinality, and ratio figure here is an industry-typical range or an illustration, with one exception. The Gorilla figures of roughly 1.37 bytes per point and about a 12x reduction come from Facebook's published paper, and I cite them as such.

The workload is the whole argument

Start with what the data actually looks like. A point is tiny: a timestamp and usually a single float, sometimes with a little metadata. A single fleet of servers or IoT devices can push millions of these per second, each one describing CPU usage, request latency, a sensor reading, or a price at an instant. The points arrive in a relentless stream, almost always in timestamp order, and they are immutable once written. Nobody goes back and edits the CPU reading from four minutes ago.

Time-series workloads are append-heavy on writes, hot on recent data for reads, and immutable once a point lands.

Reads have their own bias. Dashboards refresh every few seconds and query recent windows, so the vast majority of read traffic touches the newest slice of data. When someone does ask a long-range question, they almost never want every raw point; they want a trend, a rate, a percentile bucketed into intervals. The dominant read is not "give me this one value" but "give me p99 latency per service over the last six hours, grouped into one-minute buckets," answered in well under a second.

Hold those three facts together, because the rest of the design is just their consequences. Writes vastly outnumber and outpace reads. Data is immutable. Access is overwhelmingly recent and aggregated. A storage engine tuned for exactly this can make choices a general database cannot, and that is the entire reason these systems exist as a separate category.

The series is the unit, and cardinality is the price

The fundamental object in a time-series database is not a row, it is a series. A series is identified by a metric name plus a set of key-value tags, and each series is an ordered stream of points. So http_requests_total with tags service=checkout, region=us-east, and status=500 is one series, and changing any tag value gives you a different one. This model is powerful because it lets you slice and aggregate a metric along any labeled dimension at query time. It is also a loaded gun, and understanding why is the difference between someone who has operated these systems and someone who has only read about them.

A series is a metric name plus a set of tags; the number of distinct tag combinations is the cardinality, and it is the real scaling limit.

The number of active series is called cardinality, and it equals the product of the cardinalities of all the tags on a metric. A metric with a service tag of 50 values and a region tag of 10 values has 500 series, which is nothing. Add an endpoint tag with 200 values and you are at 100,000. Now attach a tag whose values are unbounded, a user id, a full URL with query parameters, a request id, a container name that changes on every deploy, and cardinality does not grow, it explodes, because every new value mints a brand-new series that must be tracked for the whole retention window. Keep this word in mind. We will come back to it as the central failure mode, because it is the one thing this workload punishes hardest.

Why a relational row store fights you

Before building the specialized thing, it is worth being precise about why the general thing struggles, because the answer names every design decision that follows.

A relational engine stores data row by row and indexes it with a B-tree. That layout interleaves the timestamp, the value, and every tag column together on disk. A query that only wants the value over a time range still drags every other column through memory and cache, wasting I/O on data it will discard. The B-tree itself suffers under sustained high-rate inserts, because a stream of new series and new points keeps touching and rewriting index pages, producing write amplification and fragmentation exactly when you need ingestion to stay cheap. Values are stored in fixed-width, uncompressed form, which leaves enormous compression on the table, and we are about to see just how compressible this data really is. And retention in a row store means deleting billions of individual rows, which generates dead tuples, vacuum pressure, and more index churn, rather than simply dropping a file.

You can bolt time-series behavior onto a relational engine. TimescaleDB does precisely this, turning a PostgreSQL table into a hypertable that automatically partitions into time-bounded chunks and adds columnar compression on older chunks. That it works is real proof the relational base can be adapted. That it requires exactly time partitioning, columnar layout, and compression to be added on top is also the whole argument for why a native design has those properties from the start.

The write path: WAL, head block, immutable blocks

Here is the pattern at the heart of nearly every modern time-series database, and it is worth internalizing because it recurs everywhere. Recent data lives in fast, mutable memory. Historical data lives in tightly packed, read-only, time-bounded files. The write path is the machinery that moves data from the first state to the second with a durability guarantee in between.

Every point is appended to a write-ahead log, buffered in an in-memory head block, then flushed to immutable on-disk blocks.

When a point arrives at a storage node, the first thing that happens is a cheap sequential append to a write-ahead log. This is the durability contract: if the process crashes before anything else, the point is not lost, because on restart the node replays the WAL to rebuild its in-memory state. Only after the log append does the point get inserted into the in-memory head block, a mutable structure holding the most recent window of samples for every active series. The head is what serves the freshest reads, and it is where incoming points accumulate before they are ever compressed.

The head does not grow forever. On a time boundary, Prometheus for example cuts on roughly two-hour boundaries, the accumulated data is compressed and written out as an immutable block covering that fixed time range. Once that block is safely on disk, the WAL segments it covered are redundant and get truncated. This separation is what buys you both properties at once. The head absorbs a high, effectively random write rate in memory with a fast durability guarantee from the log, while historical data settles into files that are cheap to scan, trivially cacheable because they never change, safe to replicate, and cheap to delete wholesale later.

There is one wrinkle worth naming because interviewers probe it. The head and the block encoders are optimized for near-now, in-order writes. A point that arrives far behind the current time does not fit that assumption cleanly, so many systems historically rejected out-of-order data outright, and those that support it route it through a separate, more expensive path. This is not an accident or a missing feature so much as a deliberate trade: optimizing for the common case, ordered and recent, is where the throughput and compression wins come from.

Compression is a design goal, not an optimization

In most databases compression is a knob you turn later. In a time-series database it is load-bearing, because it is the entire difference between keeping a week of data and keeping a year for the same money. And the data compresses astonishingly well once you exploit its structure, which is exactly what Facebook's Gorilla paper set out to do.

Timestamps compress with delta-of-delta encoding; float values compress with XOR against the previous value, the Gorilla scheme.

Two properties do the work. First, timestamps within a series are usually regularly spaced, say every fifteen seconds. Storing each 64-bit timestamp raw is absurd when they are so predictable, so instead you store the delta between consecutive timestamps, and then the delta-of-delta, the change in that delta. For perfectly regular sampling the delta-of-delta is zero, which encodes into a single bit, and small jitter encodes into a handful of bits. A long run of evenly spaced timestamps collapses to almost nothing.

Second, consecutive values within a series usually change slowly, and many metrics are near-constant or trending gently. Gorilla encodes each float by XORing it with the previous value. When two values are identical the XOR is zero, one bit. When they are close, the XOR has long runs of leading and trailing zero bits, so you store only the meaningful middle bits plus a short header saying where they sit. Put the two encoders together and, on real Facebook production data, points went from 16 bytes each down to about 1.37 bytes, roughly a 12x reduction. That is the number that makes months of retention economically possible.

The caveat is the same one from the write path. This bit-packed encoding assumes in-order, append-only writes within a block, which is another reason these systems prefer near-now ordered ingestion and treat backfill as a special case. The compression and the write-path design reinforce each other.

Cardinality is the failure mode that actually kills you

Now we return to the word to remember, because if you take one thing from this piece into an interview, make it this. High cardinality is the single most important thing to get right, and the failure it causes is the one that most cleanly separates people who have run these systems from people who have not.

High cardinality, driven by unbounded tag values, is the central failure mode; per-series state grows with series count, not point rate.

Recall that cardinality is the number of distinct active series and equals the product of the cardinalities of all the tags. The damage concentrates in exactly the parts of the system that hold per-series state. The in-memory head keeps structures for every active series. The inverted tag index, which we will meet properly in a moment, stores a posting list entry for every label value. Both of those grow with the number of series, not with the number of points. That is the crucial inversion: a time-series database scales primarily on active series, not on points per second. You can push more points into a bounded set of series more or less forever, but every new series costs memory that does not come back until the data ages out.

So a cardinality explosion does not show up as a gradual slowdown. It shows up as the ingestion node running out of memory and crashing, or as a query that used to touch a thousand series now having to touch ten million and grinding to a halt. The standard defenses all target cardinality directly and they are worth reciting because they are the answer to "how would you keep this stable." Never put unbounded or high-churn values, ids, raw URLs, emails, timestamps, into tags. Aggregate or drop high-cardinality labels at ingest through relabeling. Enforce per-metric and per-tenant series limits so one bad deploy cannot take down the cluster. And move genuinely high-cardinality dimensions into logs or traces, where per-event detail belongs, rather than forcing them into a metrics model that charges you forever for each distinct value. A candidate who volunteers the cardinality trap unprompted is signaling real operational scars, and that is exactly what the question is designed to find.

Downsampling and retention make the economics work

Raw high-resolution data is both expensive to keep and expensive to query over long ranges. A query for the last year does not want to read billions of raw points per series, and you do not want to pay to store them at full resolution forever. Two coordinated mechanisms solve both problems.

Downsampling rolls raw points up to 1-minute and 1-hour aggregates, and tiered retention keeps each resolution for a different length of time.

Downsampling precomputes lower-resolution versions of the data. A background rollup job reads raw points and, for each series and each coarser bucket, one minute and then one hour, stores a small set of aggregates: typically min, max, sum, count, and last, often with a percentile sketch alongside. A year-long query then reads a few thousand hourly rollup points per series instead of billions of raw ones, which is what keeps long-range dashboards fast. Retention then governs how long each resolution tier lives, for example raw for fifteen days, one-minute rollups for ninety days, one-hour rollups for two years. Because data is partitioned into time-bounded blocks, enforcing retention is a cheap file-level delete of whole expired blocks, not a row-by-row scan.

Two subtleties matter and both make good interview material. First, the aggregation choice has to be mathematically correct. You cannot average pre-averaged data across buckets and get the right answer, which is exactly why rollups store sums and counts, so an average can be recombined correctly, and why percentiles demand mergeable sketches rather than a single stored p99 value that cannot be meaningfully combined. Second, downsampling is lossy by design. Once raw points age out you can no longer investigate a spike at full resolution or compute an aggregate you never precomputed. The trade is therefore permanent and worth tuning per metric: pay to keep raw data for ad hoc high-resolution forensics, or pay far less to keep only rollups and give up that option.

The query engine: inverted index plus scatter-gather

The last piece is the read path, and it starts from a problem. Queries select series by tag matchers, not by primary key, so the engine needs to turn a predicate like service=checkout and status matches 5xx into an exact set of series quickly, without scanning sample data for series it does not need. The right structure is borrowed straight from search engines: an inverted index.

An inverted index turns tag matchers into a series set, then the query fans out to shards and gathers partial aggregates.

For every label key-value pair, service=checkout, status=500, region=eu-west, the index keeps a posting list, the sorted set of series ids that carry that pair. A tag query resolves by looking up the posting lists for the matching pairs and intersecting or unioning them to produce the precise series set to read. Notice that this index is another place cardinality bites first: its size grows with the number of distinct label values, so a high-cardinality label bloats the index before it bloats anything else.

Once the series set is known, the engine reads the relevant time range from the head block and any overlapping on-disk blocks, choosing the coarsest resolution tier that still satisfies the requested step so it does the least work possible. On top of the raw samples it applies the operations a generic SQL engine does not have built in: group by time to bucket points into fixed intervals, rate and increase to turn monotonically increasing counters into per-second rates while correctly handling counter resets, aggregations like sum, avg, min, and max across series, and quantile functions computed from mergeable sketches so they can be combined across shards and rollups.

That last point is what makes distribution work. In a sharded deployment, series are spread across nodes by hashing series identity, and a read runs as scatter-gather: a coordinator sends the query to every shard that owns matching series, each shard computes a partial aggregate locally to minimize how much data crosses the network, and the coordinator merges the partials into the final result. Pushing the aggregation down to where the data lives, rather than shipping raw points to a central node, is the same principle that shows up in every well-designed distributed query system.

The trade-offs worth arguing about

A good design answer is not a list of components, it is a set of defended choices, and this problem is rich in the kind where there is no single right answer.

Push versus pull ingestion is the classic one. In a pull model the database scrapes a metrics endpoint on each target on a fixed schedule, which gives the server control over load, makes a target being down an immediate and explicit signal because the scrape fails, and makes redundant scrapers trivial, but it cannot easily reach targets behind NAT or capture short-lived jobs, so pull systems add a push gateway for batch work. In a push model clients send data in, which handles ephemeral and event-like producers naturally and works through any network path, but it exposes the database to uncontrolled write bursts and turns "is the client alive" into an ambiguous question. Pull fits controlled, service-oriented fleets; push fits heterogeneous or ephemeral producers; large platforms often support both.

The engine choice is another. You can reuse a log-structured merge tree, the family behind Cassandra and RocksDB, which gives you a proven, general engine and existing operational knowledge quickly, at the cost of weaker time-series compression because it does not know timestamps are monotonic and values change slowly. Or you build a purpose-built engine with per-series columnar layout and delta-of-delta and XOR encoding, which is bespoke code to maintain but wins the roughly 10x storage advantage. Dedicated systems almost always pick the specialized path because the compression win is large enough to pay for the engineering. And underneath it all sits the durability trade: acknowledging each point only after quorum replication and fsync maximizes safety but caps throughput, so because one lost fifteen-second sample rarely matters, most time-series databases lean hard toward throughput in a way a transactional database never would.

Where to go from here

The thing worth keeping is the method, not the trivia. A time-series database is fast and cheap because it refuses to be general. It gives up arbitrary updates, cross-series joins, and unbounded cardinality, and in exchange it gets columnar per-series layout, compression that turns 16 bytes into a fraction of one, time partitioning that makes retention a file delete, and a query engine that answers recent-range aggregations in milliseconds. Every one of those is the same move: take the lopsided shape of the workload seriously and let the design fall out of it. Understand which of those properties your own problem actually needs, and the build-versus-adapt decision stops being a guess.

I teach time-series databases and the rest of system design this way, from first principles with real diagrams and the trade-offs that only surface in production, as an interactive course at systemdesign.academy. The foundation lessons are free and need no signup.

Read the free lessons: https://systemdesign.academy

API Design: REST vs GraphQL vs gRPC, Versioning, Idempotency, and Errors

RONI DAS — Mon, 20 Jul 2026 02:25:53 +0000

Most parts of a system can be rewritten. You can swap the database, rename the tables, restructure the services, and as long as the behavior on the outside stays the same, nobody outside your team ever knows. An API is the one part where that stops being true. The moment the first external client integrates against it, every field name, every status code, every error shape, and every pagination style is frozen. Other people have written code against your decisions, and then they have stopped touching that code, which means your mistake is now their production dependency for years.

That is what makes API design a distinct discipline from the architecture behind it. A schema is private and you migrate it on your own schedule. A public API surface is a promise you signed on behalf of your future self. So the interesting questions are almost never about raw throughput. They are about how you model resources so they still make sense in three years, how you evolve the contract without breaking existing callers, how you make a retried request safe to run twice, and how you tell a client the difference between bad input, insufficient permission, and a genuine server fault.

This is the long version. We will start with the idea of the API as a frozen contract, then work through the three dominant styles and when each fits, the gateway that carries the cross-cutting concerns, versioning, pagination, idempotency, and the error format that decides whether an API is one people love or one they dread. Any latency or size figures I give are industry-typical ranges for illustration, not measured numbers from a specific vendor.

An API is a frozen contract others build on: internal code can change freely, the surface cannot.

The contract is the product

Here is the mental shift that separates people who wire up endpoints from people who design APIs. Internally, your data model is yours. You can normalize it, denormalize it, shard it, and move it between stores, and none of that is anyone else's business. The API is the translation layer between that private world and the public one, and its entire job is to stay stable while the private world churns underneath.

This is why the single most important habit in API design is separating the external contract from the internal model. When a service serializes a response, it is not dumping its database rows onto the wire. It is producing the agreed representation, deliberately, so that when you refactor storage next quarter the callers never notice. Every time an internal detail leaks through the contract, a database column name that becomes a JSON field, an enum value that mirrors an internal state machine, you have quietly handed your integrators a dependency on something you thought was private, and you will discover it the day you try to change it.

Good API design leans hard toward backward compatibility, predictability, and least surprise, for one blunt reason. The cost of a breaking change is paid by every integrator at once, and it cannot be undone unilaterally. That asymmetry should sit behind every decision that follows.

Three styles, three different questions

People frame REST, GraphQL, and gRPC as competitors. They are better understood as answers to different questions, and a strong design picks by looking at the consumer, not at fashion.

REST models the domain as resources addressed by URLs, uses HTTP methods to state intent and status codes to state the outcome, and in exchange rides the entire HTTP ecosystem for free: caching, proxies, CDNs, and universal client support in every language and tool. It is the right default for public, resource-oriented APIs and anything that benefits from HTTP caching. Its weaknesses are the mirror of its simplicity. An endpoint over-fetches when it returns more than a given screen needs, and a client under-fetches when one screen needs three resources and therefore makes three round trips.

GraphQL answers the wish to let the client ask for exactly the fields it wants. One endpoint, a typed schema, and a query language, which is genuinely powerful for mobile apps and rich frontends where different screens need wildly different slices of data. The cost is real and often understated. HTTP caching no longer applies cleanly because most requests are POSTs to a single URL, you have to defend the server against expensive deeply nested queries with depth and complexity limits, and the N+1 problem in resolvers has to be solved with batching or you will hammer your database.

gRPC is contract-first binary RPC over HTTP/2 using protobuf: compact, fast, strongly typed, with streaming and code generation across many languages. It shines for internal service-to-service communication where both ends are yours and you want speed and a strict contract. It is awkward for browsers, which need a proxy, it is not human-readable on the wire, and it is not something you usually expose as a public API.

REST vs GraphQL vs gRPC: pick by consumer, not by fashion.

The staff-level answer is not to crown a winner. Public and cache-friendly leans REST. Diverse client-driven data needs lean GraphQL. Internal high-performance calls lean gRPC. And plenty of real systems run gRPC between their own services with a REST or GraphQL edge facing the outside world, which is a design, not a contradiction.

The gateway carries the cross-cutting concerns

A request does not go straight to the service that owns the data. It first reaches an edge tier, typically a load balancer in front of an API gateway, and the gateway is where the shared parts of the contract live. It terminates TLS, authenticates the caller by validating an API key or a bearer token or a request signature, enforces rate limits and quotas per caller, and routes to the service that owns the resource.

Putting authentication, throttling, and routing at the gateway is not just tidy. It keeps those policies consistent across every backend, so you are not reimplementing rate limiting in five services and getting it subtly different in each. It also keeps the backend services focused on business logic instead of auth and throttling boilerplate. The gateway is the natural place to attach a request ID when the client did not supply one, so that every downstream log line and error response can be tied back to a single request.

The client hits a gateway that does auth and rate limiting before routing to the owning services.

Behind the gateway sit the services that implement the operations. A service validates the request body against its schema, checks authorization (this caller may read this resource but not write it), executes the operation, and serializes the response in the agreed format. There is a clean division worth stating plainly. Authentication, who you are, usually resolves at the gateway. Authorization, whether this specific caller may touch this specific resource, often has to happen in the service, because only the service knows ownership. And authorization must always be enforced server-side, never inferred from the client, because anything the client controls, an attacker controls too.

The gateway also leans on supporting infrastructure: a cache tier and CDN for safe reads driven by ETags and Cache-Control, a fast rate-limit store such as Redis holding per-caller counters, and an async subsystem for long-running work. The design goal across all of it is that the common path, an authenticated read or a simple write, stays cheap, while the harder guarantees are handled by dedicated mechanisms rather than bolted onto every handler.

Resources, methods, and status codes that mean what they say

The backbone of a usable REST API is consistent resource modeling, and the rule is to model nouns, not verbs. It is /orders and /orders/{id}/refunds, never /createOrder or /getOrderRefund. Use plural collection names, express containment through hierarchy such as /customers/{id}/invoices, and keep case and pluralization consistent across the whole surface, because inconsistency is exactly what makes an API feel untrustworthy even when it works.

Map methods to intent honestly. GET is safe and never mutates. POST creates or triggers actions. PUT replaces a resource wholesale. PATCH applies a partial update. DELETE removes. Idempotency is part of the method contract too: GET, PUT, and DELETE are defined as idempotent, and POST is not, which is precisely why creating with POST needs an idempotency key while PUT does not.

Status codes have to carry real meaning rather than decoration. Use 200 for success with a body, 201 with a Location header for a created resource, 202 for accepted but not yet done, and 204 for success with no body. On the error side, 400 for malformed input, 401 for missing or bad authentication, 403 for authenticated but not permitted, 404 for not found, 409 for a conflict such as a version mismatch, 422 for well-formed but semantically invalid input, 429 for rate limited, and 5xx reserved for genuine server faults. The classic mistake, and it is genuinely destructive, is returning 200 with an error buried inside the body. It breaks every client, proxy, and cache that trusts the status line, which is all of them.

Versioning: the discipline of not needing it

Versioning exists so you can make breaking changes without breaking existing clients, and the first rule is to avoid needing it. Make changes additively whenever you possibly can. Add a new optional field, a new endpoint, a new optional parameter, and build every client to ignore unknown fields, so that adding one never breaks anyone.

When a breaking change is genuinely unavoidable, there are three places to put the version. URI versioning, /v1/orders and /v2/orders, is the most common and most visible. It is trivial to route, trivial to test in a browser, and unambiguous, at the cost of arguably conflating the identity of a resource with its representation, and of proliferating paths. Header versioning, a custom header or Accept-Version, keeps URLs clean and lets the version travel as metadata, but it is invisible in a browser address bar, easy to forget, and harder to cache and debug. Media-type versioning through content negotiation, Accept: application/vnd.company.v2+json, is the most RESTful in spirit because it treats the version as one representation of the same resource, but it is the least approachable and least widely understood by consumers.

URI, header, and media-type versioning: the same choice placed in three different parts of the request.

Most large public APIs choose URI versioning for its bluntness and clarity. Some, Stripe being the widely cited example, use a date-based version pinned per account, so a client keeps its exact behavior until it explicitly upgrades. Whatever you pick, the disciplines are identical: never change the meaning of an existing field, add rather than mutate, and publish a real deprecation policy. When a version or field is retired, announce it, emit Deprecation and Sunset headers on the old surface, warn active integrators, and give a generous window, commonly six to twenty-four months, because forcing a faster cutover breaks integrations that nobody is actively maintaining anymore. The single most damaging move in API evolution is the silent semantic change, where a field keeps its name but quietly starts meaning something different. It passes every schema check and breaks clients in production with no warning at all.

Pagination: offset is a trap at scale

Any endpoint that returns a collection must be paginated, full stop. An unbounded list is a latency hazard, a memory hazard, and a denial-of-service vector all at once, which is why collection endpoints need a sane default page size, commonly twenty to fifty, and a hard maximum, commonly one hundred to a thousand, so that no client can pull an entire table by setting the limit parameter to a large number.

Offset pagination, using limit and offset or page numbers, is simple and lets a client jump to an arbitrary page. It is genuinely fine for small, mostly static datasets. At scale it has two real problems. Deep offsets are slow because the database still has to scan and discard every skipped row to reach page 500. And the results drift when items are inserted or deleted while a client is paging, so the client sees duplicates or silently skips records, which is the kind of bug that surfaces as a furious support ticket months later.

Offset pagination drifts and slows down at depth; cursor pagination stays indexed and stable.

Cursor pagination, also called keyset pagination, fixes both. It encodes a stable position, typically the sort key of the last item seen, into an opaque cursor token that the client passes back to fetch the next page. The query becomes give me the next twenty items after this key, which uses an index directly and stays correct even as rows are inserted and deleted underneath it. The trade is that you cannot jump to an arbitrary page and the cursor is opaque to the client, which is a fair price for correctness and speed on large or live datasets. As a rule: offset for small admin-style lists where users expect page numbers, cursor for large, live, high-throughput collections. Filtering and sorting deserve the same discipline. Expose a defined set of filterable and sortable fields, validate against that list, and reject anything not on it, because an open query language lets one client write an unindexed query that degrades the service for everyone.

Idempotency: making a retry safe

This is the deep-dive that separates a good answer from a great one in an interview, and it is a genuinely load-bearing production concern. Networks fail after the server has already done the work but before the client receives the response. The client, seeing a timeout, retries. Without protection, that retry performs the action a second time, and for a payment or an order, charging a customer twice is not a bug, it is an incident.

Idempotency is the property that performing an operation more than once has the same effect as performing it once. GET, PUT, and DELETE are naturally idempotent by their contract. The dangerous case is POST. The standard solution is the idempotency key: the client generates a unique key, usually a UUID, for the logical operation and sends it in a header. On first receipt the server executes the operation and stores the key together with the result. On any retry carrying the same key, it skips execution and returns the stored result.

An idempotency key lets the server replay the first result instead of running a retried write twice.

Two subtleties matter. First, store a fingerprint of the request, a hash of method, path, and body, alongside the key, and reject a reused key that arrives with a different payload, because that combination means a client bug, not a legitimate retry. Second, handle the race where two requests with the same key arrive concurrently, typically by inserting the key under a uniqueness constraint so the second request detects the in-flight first one and either waits or returns a conflict. Keys are retained for a bounded window, and twenty-four hours is a common choice because retries happen within minutes, not weeks. Done well, this turns retry-on-timeout from a dangerous gamble into a safe default, which is the whole point.

Errors are part of the contract

An error is as much a part of the API contract as a success, and clients need to program against it, not parse prose. The standard is RFC 7807, Problem Details for HTTP APIs, now updated by RFC 9457, which defines the application/problem+json media type and a small set of fields: type, a URI identifying the problem class; title, a short human summary; status, the HTTP code; detail, a human-readable explanation of this specific occurrence; and instance, a URI for this occurrence, plus room for extension members. The value is consistency. Every error across the API has the same shape, so a client writes one handler instead of a special case per endpoint.

RFC 9457 problem+json with a stable error code, a request ID, and status codes that tell the truth.

In practice you add one thing the standard does not mandate: a stable, machine-readable error code, a string like rate_limit_exceeded or card_declined, that clients switch on. The human-readable message will change over time and must never be parsed. Crucially, the error body should carry the request ID, and that same ID should appear in your structured logs and traces. This is the observability payoff. A caller reports a failure and quotes the request ID, and an operator finds the exact log line, trace, and downstream calls in seconds instead of guessing. Errors should also distinguish clearly between a client fault, 4xx, do not retry without changing the request, and a server fault, 5xx, safe to retry with backoff, and for retryable errors a Retry-After header tells the client when to come back. Vague 500s with no code and no request ID are the precise difference between an API teams love and one they dread.

The rest of the surface

A few more decisions round out a serious design. Rate limiting caps request rate per caller over a window; a token bucket allows short bursts while bounding the sustained rate, and when a caller exceeds the limit you return 429 with Retry-After and, ideally, RateLimit headers advertising the limit, remaining, and reset so a well-behaved client self-throttles before it gets rejected. Quotas are the longer-horizon cousin, capping usage per day or month and often tied to a billing tier.

Long-running work should not block a request. For a report, a video transcode, or an infrastructure provision, accept the request, return 202 with an operation resource that has its own URL and a status field, run the work on a background queue, and let the client either poll that operation until it reaches succeeded or failed or register a webhook to be called back. Polling is simple and needs no inbound endpoint on the client; webhooks avoid polling overhead but require the client to expose and secure a receiver and require the server to retry deliveries and sign them.

Caching is the lever for read-heavy APIs. Cache-Control tells clients and intermediaries how long a response may be reused, and ETags enable conditional requests, where the client sends If-None-Match with the ETag it holds and the server replies 304 Not Modified with no body when nothing changed. The same ETags power optimistic concurrency on writes through If-Match, so a client can update a resource only if it has not changed since the version it read, which the server enforces by returning 412 Precondition Failed on a mismatch.

The through-line

Notice what unites all of it. Model nouns, separate the external contract from the internal model, prefer additive change, paginate with a stable cursor, make writes idempotent, and return errors a machine can act on. Every one of these is the same instinct applied in a different place: reduce the surprise you hand to the person on the other end of the contract, because you will not get to take it back. The best API designers I have worked with are the ones who have been the integrator on the receiving end of a bad API, and design as if that person is watching.

The honest interview answer, and the honest design, separates the timeless principles, resources, safe and idempotent methods, backward-compatible evolution, standard errors, from the pragmatic choices where reasonable APIs differ, URI versus header versioning, offset versus cursor pagination, and never claims a single approach is universally correct.

I teach API design and the rest of system design this way, from first principles with real diagrams and the trade-offs that only show up once real clients depend on you, as an interactive course at systemdesign.academy. The foundation lessons are free and need no signup: https://systemdesign.academy

Designing DynamoDB: A Distributed Key-Value Store at Any Scale

RONI DAS — Sun, 19 Jul 2026 17:14:21 +0000

Ask most engineers how DynamoDB scales and you get one answer. It partitions the data. That is true, and it is also the least interesting sentence you can say about it, because sharding a dataset across machines is a decades-old idea and most implementations of it are painful. What makes DynamoDB feel like it does not care how large your table is, returning a point read in single-digit milliseconds whether it holds a megabyte or a petabyte, is a stack of deliberate design decisions layered on top of that partitioning, each one a trade you can learn from and reuse in your own systems.

This piece is the long version, working from why a distributed key-value store is hard at all through consistent hashing, quorums, conflict resolution, the storage engine, secondary indexes, hot partitions, and multi-region global tables. Along the way I will separate the 2007 Dynamo paper, the conceptual ancestor, from production DynamoDB, which made deliberate changes to that design. By the end you should be able to defend both why the system is fast and exactly what you gave up to get that speed. One note on numbers: any specific latency or throughput figure here is an industry-typical range or an illustration, not a measured AWS-internal metric.

Why a distributed key-value store is genuinely hard

Start with the problem, because the whole design is a chain of answers to it. You have more data and traffic than one machine can hold or serve, so you must spread both across many machines. The moment you do, four hard questions appear at once, each with a wrong answer that looks fine until production. Which machine owns which key, in a way that stays balanced and does not reshuffle the whole dataset every time you add or remove a node? What keeps the data available when a machine dies, since at the scale of thousands of nodes something always is? When two copies of the same key disagree, which one is right? And how do you keep a single-key operation fast while the table grows without bound and the system is actively moving data around underneath you?

The partition key decides everything downstream, and the data is always in motion as partitions split and move.

The unifying decision that falls out of these questions is that the partition key is the most consequential thing in the system. It is hashed to decide which partition, and therefore which physical nodes, own an item. Get it right and traffic spreads evenly across the fleet and throughput grows almost linearly as you add hardware. Get it wrong and you concentrate load onto one partition, one node, one CPU, and no amount of provisioned capacity saves you. Everything that follows is built so a well-chosen key gives flat latency at any size, and so no single request ever touches more than a handful of nodes.

The request path stays thin on purpose

A client addresses an item by its primary key and talks to the service through a stateless request layer whose first job is routing. It hashes the partition key and looks up which partition owns that hash range in the partition metadata. That lookup tells it which storage nodes hold the replicas, and it forwards the operation. For a write it targets the partition leader; for a cheap read it can target any replica.

A request router hashes the key, consults the ring map, and forwards to the replica group on the owning storage nodes.

The important property of this tier is that it holds no durable state. Routing is a pure function of the key plus a metadata map, so you can run as many routers as you need and any of them can serve any request. This is the same instinct you see in every system that scales cleanly: keep the fast path simple and local, and push the slow global coordination somewhere it cannot get in the way. The router authenticates the caller, checks throughput, finds the partition, and forwards. That is all.

Underneath the routers sit the storage nodes, grouped into replica sets, one per partition. Underneath all of it sits a control plane the data path never touches directly. It tracks node membership and health, decides when a partition should split or move, drives the data movement while the partition stays online, and updates the metadata map the routers consult. The critical-path get or put never waits on it. In the Dynamo paper this membership spread through gossip; production DynamoDB backs it with a more centralized, consensus-backed metadata service. The principle is identical: the fast path is simple and local, and the expensive global agreement is kept off to the side.

Consistent hashing and why virtual nodes exist

Now to the first hard question, ownership. The naive answer is to hash the key and take it modulo the number of nodes. That works until the node count changes, then fails completely, because changing N remaps almost every key at once, moving almost all of your data. At scale that is not a rebalance, it is an outage.

Keys and nodes both hash onto a ring, and each physical machine appears many times as virtual nodes to smooth the distribution.

Consistent hashing solves this. Place both keys and nodes on a ring, a circular hash space, and let a key be owned by the first node clockwise from its hash. Now adding or removing a node only reassigns the keys in the arc immediately next to it, so roughly one over N of the data moves instead of nearly all of it. That single change is what makes online scaling possible at all.

The basic scheme has two flaws, and the fix for both is the same. First, if each node sits at one random point on the ring, chance alone gives some nodes much larger arcs than others, so load is uneven. Second, when a node fails, its entire arc dumps onto its single clockwise successor, which can then tip over from the doubled load. The Dynamo paper fixes both with virtual nodes: each physical machine is represented by many points on the ring rather than one. This smooths the distribution because the law of large numbers evens out the arc sizes, spreads a failed machine's load across many peers instead of one, and lets a more powerful machine carry more virtual nodes and therefore more data. DynamoDB's partitioning is the managed evolution of this idea, where partitions are the owned arcs and the service moves them for you rather than exposing the ring.

Replication and quorums, and the two flavors of read

Owning a key on one node is not enough, because that node will die, so each partition is replicated, typically three copies across three availability zones. Three copies in three failure domains is the standard durability-versus-cost point: it survives losing any one copy, or one entire zone, while still keeping a majority of two available to make progress.

With N copies, a write waits for W acknowledgements and a read consults R replicas; W plus R greater than N guarantees a read sees the latest write.

The Dynamo paper frames consistency as a quorum with three tunable numbers. N is the number of replicas, W is how many must acknowledge a write, and R is how many must be consulted for a read. The key inequality is that if W plus R is greater than N, any read set and any write set overlap on at least one replica, which guarantees a read sees the most recent write. Tuning these trades latency against consistency: W equal to 1 is fast but risky, W equal to N is durable but slow because the slowest replica gates every write, and a common balance is N equal to 3, W equal to 2, R equal to 2. Dynamo also used sloppy quorums with hinted handoff: if a target replica was unreachable, a healthy stand-in accepted the write and held it as a hint to forward later, keeping writes available during a partition at the cost of temporary inconsistency.

Production DynamoDB simplifies the developer's view by designating one replica per partition as the leader. Writes go to the leader, which appends to a write-ahead log and replicates to the others, acknowledging once a durable quorum has accepted. This gives reads two distinct flavors, one of the most common interview discriminators. An eventually consistent read, the default and cheaper option, can be served by any replica and might return data a few milliseconds stale, because the replica you hit may not have applied the latest write yet. A strongly consistent read is routed to the leader, which holds the latest committed value, so it reflects everything acknowledged before it. Strong reads cost more latency, consume roughly double the read capacity, and are slightly less available during a leader failover. Eventual is the default on purpose, because forcing every read through the leader would cost latency and availability that most workloads, a profile fetch or a product lookup, do not need.

When two writes collide

In a leaderless system, two clients can write the same key at nearly the same instant to different replicas, and those replicas end up holding divergent versions. Something has to reconcile them, and the two answers sit at opposite ends of a spectrum from correctness to simplicity.

Last-writer-wins keeps the higher timestamp and silently drops the other; vector clocks preserve causal history and surface true conflicts to the application.

The original Dynamo attached a vector clock to each version, a set of node-and-counter pairs recording the causal history of the update. When one version's clock strictly descends from another's, the newer wins automatically. When two versions are concurrent, neither descending from the other, Dynamo cannot decide and returns both to the application to merge. The canonical example is the Amazon shopping cart, where merging two concurrent carts by union is safe and losing either one silently is not. Vector clocks preserve correctness but push complexity onto the developer and can grow large.

The simpler alternative is last-writer-wins: attach a timestamp to each write and keep the higher value, discarding the other. It is trivial and keeps the API clean, but it can silently lose a write when two updates race, and it depends entirely on the quality of the clocks. Production DynamoDB keeps this problem away from developers on the single-region path by funneling writes through the per-partition leader, which orders them, so single-region writes never produce sibling versions. It resurfaces only in cross-region global tables, which resolve conflicts with last-writer-wins on a timestamp. That is precisely why multi-region writes carry a real last-writer-silently-wins caveat you have to design around.

The storage engine underneath a partition

Everything above decides which nodes hold an item. On each, the item lives in a local storage engine facing a specific workload: a high write rate, point and short-range reads, and a dataset far larger than RAM. That is what a log-structured merge tree was built for.

Instead of updating data in place on disk, which turns every write into slow random I/O, an LSM tree buffers incoming writes in an in-memory table and appends them to a durable log for crash recovery. When the in-memory table fills, it is flushed to disk as a single immutable, sorted file. Reads check the in-memory table first, then the on-disk files, using per-file indexes and Bloom filters, compact probabilistic structures that let a read skip a file that definitely does not hold the key. Over time the accumulating small files are merged into larger sorted ones by a background process called compaction, which is also where deleted and overwritten records are finally discarded. The trade is clear: writes become cheap and sequential, while a read may consult several files and compaction consumes background I/O. B-trees are the alternative, favoring read-optimized in-place updates at the cost of more expensive random writes. A store that must ingest writes at scale generally leans LSM, as Cassandra and the other Dynamo descendants do.

This explains the write path. The durability boundary is the log append, not the flush: once the mutation is in the durable log and a quorum holds it, the write is safe even though the sorted data file is not written yet.

Secondary indexes are a separate system that lags

A pure key-value store can only find an item by its primary key, too limiting for real applications, so secondary indexes let you query by other attributes. There are two kinds, they behave very differently, and confusing them is a frequent stumble.

A local secondary index shares the base table's partition key but offers an alternate sort key. Because it lives on the same partition as the items it indexes, it can be updated inside the same write and kept strongly consistent, but it inherits that partition's size limits and must be declared when the table is created. A global secondary index uses a completely different partition key, so its entries can live on entirely different partitions and nodes than the base items. Updating it synchronously would require a distributed transaction across the fleet on every write, destroying write latency. So instead the base table's ordered change stream is consumed asynchronously and applied to the index. The consequence is that it is eventually consistent: right after you write an item, a query against that index might not see it yet, usually for a very short window. It also has its own throughput budget, and here is the trap, if that budget is under-provisioned, writes to the base table itself can be throttled because the index cannot keep up.

The hot partition, the limit you cannot buy your way out of

This catches teams by surprise, and it follows from the physics of partitioning. Every partition has a hard throughput ceiling, illustratively a few thousand reads and about a thousand writes per second, because it is backed by a bounded slice of one node's CPU, memory, and disk.

A skewed partition key concentrates traffic on one partition; adaptive capacity lets it borrow idle throughput and can split it out, but cannot exceed one machine.

The subtlety that trips people up is that provisioned throughput is divided across partitions, not held in a global pool. A table provisioned for 10000 writes per second over ten partitions gives roughly 1000 per partition. If your key concentrates traffic, a status flag with three values, today's date, one celebrity account, a disproportionate share of requests lands on one partition, and it throttles at its 1000 write ceiling even though nine-tenths of the table's paid-for capacity sits idle. That is a hot partition, the direct punishment for a low-cardinality or skewed key.

There are two classes of mitigation. On your side, choose a naturally high-cardinality key, or use write sharding, appending a suffix to spread a single hot key across several synthetic keys and therefore several partitions, then reading back across all of them. On the service side, DynamoDB added adaptive capacity, which lets a hot partition temporarily borrow unused throughput from siblings and can isolate a persistently hot item by splitting the partition around it. That softens uneven load and short bursts. What it cannot do is break physics. A single key hotter than one partition can serve is still bounded by one partition, one leader, one machine, because one item cannot be split across machines. This is the ceiling sharding hits everywhere: it distributes keys, not the load on any single key, and the only real fix lives in how the application models that key.

Global tables and the price of multi-region writes

A single-region table survives node and zone failures but not the loss of an entire region, and it cannot give a user on another continent a local low-latency write. Global tables solve both by replicating across multiple regions with active-active, multi-primary writes. Every region accepts local reads and writes and asynchronously ships its changes to the others, typically within a second, though that lag widens with distance and load.

Every region accepts local writes and replicates asynchronously to the others, resolving concurrent updates by last-writer-wins on timestamp.

The benefit is real: local latency everywhere and tolerance of a whole region going dark. The cost is equally real, and it is consistency. Because writes are accepted independently in each region and replicated after the fact, two regions can update the same item concurrently, and the system resolves that conflict with last-writer-wins on a timestamp: the later write survives and the other is silently discarded. There is no cross-region strong consistency and no read-your-write guarantee across regions. An application on global tables must tolerate a brief window where regions disagree and accept that a losing concurrent write vanishes. Where that is unacceptable, a financial ledger, inventory that cannot oversell, you either pin writes for an item to a single region or add application-level reconciliation. This is the same last-writer-wins trade from the conflict section, surfacing at the one layer where DynamoDB could not engineer it away.

The trade-offs, stated plainly

Every design choice above bought something and cost something, and naming both sides separates an answer that recites features from one that understands the system. The per-partition leader traded the last increment of write availability for developer simplicity, since a leader sequences writes so single-region conflicts never arise and strong reads are trivial. Eventually consistent reads as the default traded freshness for cost, latency, and availability, the right call because most reads tolerate a few milliseconds of staleness and you opt into strong consistency only where correctness demands it. Single-table denormalized design traded query flexibility for single-round-trip speed, baking access patterns into the key schema so a new query pattern can require a new index or migration. And the bounded item size with no unrestricted cross-partition transactions is the price of predictable latency and near-linear scale. In every one of these, predictability was chosen over generality.

Where this design is the right tool, and where it is not

DynamoDB fits one class of workload extremely well: high-volume access by a known key, where you need flat latency as the data grows and your access patterns are stable enough to design keys around. Session stores, shopping carts, user profiles, feature stores, metadata services, and event records all sit comfortably here. The store is built for the one operation these share, a small addressable read or write over an item you can name in advance.

It is the wrong tool when your queries are rich and unpredictable and need joins and ad hoc analytics across the dataset, because there is no planner or join engine and paying per partition access punishes that pattern hard. It is wrong when you need strong global consistency with no chance of a silently dropped write, because the multi-region model gives up exactly that. And it is awkward when access patterns are still in flux, because the key schema is expensive to change after the fact. The correct mental model is a precise, predictable instrument for keyed access at scale, used alongside a relational database or an analytics store rather than instead of them.

Where to go from here

If this was useful, the thing worth keeping is the method, not the trivia. A system that scales cleanly usually does so because it refuses to let any single request touch more than a few nodes, and because it pushes the expensive global coordination off the fast path. DynamoDB is one of the clearest examples in the toolbox. Consistent hashing decides ownership without reshuffling everything, quorums buy availability without waiting on the slowest replica, the leader removes single-region conflicts, the LSM engine makes writes sequential, and a well-chosen partition key spreads the load so linearly that the table's size stops mattering. Understand which of those levers your own bottleneck sits on, and the fix stops being a guess.

I teach distributed systems and the rest of system design this way, from first principles with real diagrams and the trade-offs that only surface in production, as an interactive course at systemdesign.academy. The foundation lessons are free and need no signup.

Read the free lessons: https://systemdesign.academy

How RAG Actually Works in Production (The Model Is the Easy Part)

RONI DAS — Sun, 19 Jul 2026 16:59:44 +0000

Ask most people how a retrieval-augmented generation system works and you get an answer that is really about the language model. You put your documents somewhere, the model reads them, and it answers your question. The model is the part everyone talks about, because the model is the part that feels like magic. But if you have ever shipped one of these systems and watched it confidently answer the wrong question, or miss an answer that was sitting right there in your data, you already know the uncomfortable truth. The model is the easy part. RAG is a retrieval problem wearing a language model's clothes, and it lives or dies on retrieval quality long before the model ever gets a turn.

This piece is the long version. We will start with why a raw model is not enough and what RAG actually fixes, then walk the ingestion pipeline that runs offline, chunking and embedding and indexing. Then we will follow a single query through retrieval, reranking, and prompt assembly, and see how the model generates an answer that is grounded in what we found. After that we get to the parts that decide whether the thing survives contact with real users: the failure modes, how you evaluate a system whose two halves fail differently, and how you keep the index from going stale. At the end we pull out the patterns that transfer to systems that have nothing to do with language models. I am not going to quote any vendor's internal numbers. Where a figure matters, chunk sizes, how many passages to retrieve, embedding dimensions, I give the industry-typical range and say so, because the reasoning is what transfers, not a precise number I cannot stand behind.

Why a raw model is not enough

A language model knows what it was trained on, and nothing else. Its knowledge is frozen at a training cutoff, so it has never seen anything that happened after that date. It has never seen your company's internal wiki, your customer's support tickets, your product's current pricing, or the contract a user uploaded thirty seconds ago. None of that was in the training data, and none of it can be, because that data is private, or new, or both.

A raw model answers from frozen training knowledge and invents plausible detail when asked beyond it; RAG grounds the answer in documents retrieved at query time.

Now watch what happens when you ask such a model a question about data it has never seen. It does not stop and say it does not know. That is not how these models work. They are trained to produce fluent, plausible continuations of text, and a confident wrong answer is more fluent than an admission of ignorance. So it fills the gap with something that sounds right, invented citations, a policy that does not exist, a number in the correct shape but wrong. We call this hallucination, and it is not a bug you can patch out. It is the model doing exactly what it was built to do, applied to a question it has no grounds to answer.

RAG addresses this directly. Instead of asking the model to answer from memory, you first retrieve the relevant documents from your own data, then hand those documents to the model along with the question, and instruct it to answer only from what you gave it. The knowledge no longer lives in the model's frozen weights. It lives in a corpus you control, that you can update the moment a document changes, and the model's job shrinks to reading and summarizing text that is right in front of it. That is a much easier job, and a much safer one. The catch, and the whole rest of this article, is that the answer can only ever be as good as the documents you retrieved. If retrieval hands the model the wrong passages, the model will faithfully summarize the wrong passages, and now you have a fluent, well-cited, confidently wrong answer that is harder to catch than the original hallucination.

A fair question at this point is why not fine-tune the knowledge into the model instead. Fine-tuning is for teaching behavior, format, tone, and task, but it is a poor and costly way to inject facts that change, and it cannot cite a source. RAG owns changing knowledge, fine-tuning owns behavior, and mature systems use both.

The pipeline at a glance

Before we go part by part, here is the whole shape, because it is hard to reason about any single stage without knowing where it sits.

Two pipelines: offline ingestion loads, chunks, embeds, and indexes the corpus; at query time the system embeds the query, retrieves, reranks, assembles a prompt, and generates.

There are really two systems here, and they run at different times. The first is ingestion, and it runs offline, ahead of any user. You load your source documents, split them into passages, convert each passage into a vector (a list of numbers) using an embedding model (a network that maps text to such a vector), and store those vectors in an index built for fast similarity search. This is a batch job, or a streaming pipeline, but either way the user is not waiting on it. Its output is a searchable index.

The second system runs at query time, while a user waits, and every millisecond counts. You take the user's question, embed it with the same model you used on the documents, search the index for the passages closest to it, optionally rerank those candidates for precision, assemble the best of them into a prompt that fits the model's context window (the fixed amount of text it can read at once), and finally send that prompt to the model to generate an answer. Retrieval quality is decided almost entirely in the offline pipeline, but it is felt entirely at query time. Most teams pour their attention into the query-time model call, the visible part, and underinvest in ingestion, the invisible part, which is exactly backwards. Get ingestion wrong and there is no prompt clever enough to save you.

Chunking: the decision that quietly sets your ceiling

The first real decision in ingestion is how to cut your documents into pieces. You cannot embed a whole hundred-page manual as one vector, and you would not want to. So you split documents into chunks, passages of a few sentences to a few paragraphs, and each chunk becomes a unit of retrieval. This sounds like plumbing. It is actually one of the highest-leverage decisions in the entire system, and it is where a lot of RAG systems are silently broken from day one.

Chunks too large dilute the embedding and waste context; too small lose meaning; a middle size with overlapping edges keeps each passage coherent at its boundaries.

The tension is easy to state and hard to resolve. Make chunks too large and two bad things happen. The embedding of a big chunk is an average of many ideas, so it points nowhere in particular and matches queries weakly, the signal for any one fact diluted by everything around it. And a large chunk eats your context budget at query time, so you can afford to include fewer of them. Make chunks too small and you get the opposite problem. A single sentence pulled out of its surroundings often loses the meaning that made it useful. "It supports up to 200 connections" is useless if the chunk does not carry what "it" refers to. You retrieve a fragment that matches the words of the query but cannot actually answer it.

The practical middle ground for prose is a chunk on the order of a few hundred tokens, but the exact number matters less than two techniques that address the boundary problem directly. The first is overlap. You let consecutive chunks share some text at their edges, so a sentence that would otherwise be orphaned at a boundary appears in full in at least one chunk. The second, and more important, is structure-aware splitting. Instead of cutting blindly every N characters, you split on the document's own structure, on headings, paragraphs, list items, code blocks, table rows. A chunk that respects a section boundary carries a complete thought; a chunk that cuts a sentence in half carries neither half's meaning. For structured content like Markdown or HTML or source code, splitting on the structure rather than on raw length is often the single biggest quality improvement available to you, and it costs nothing but a better parser.

One more technique decouples the unit you retrieve from the unit you send. You index small precise chunks for matching, but at retrieval time you expand each hit to its surrounding window or parent section, so the model reads the fragment in context rather than alone. This is often called small-to-big or parent-document retrieval.

Embeddings: turning meaning into geometry

Once you have chunks, you need a way to find the ones relevant to a question. Keyword matching alone is not enough, because a user who asks "how do I reset my password" should find a document titled "recovering account access" even though they share almost no words. You need to match on meaning, not on spelling, and that is what embeddings give you.

An embedding model maps text to a vector so that semantic similarity becomes geometric closeness; query and chunks share one vector space.

An embedding model is a neural network trained to map a piece of text to a vector, a list of numbers, typically several hundred to a couple of thousand of them. The training objective is what makes this useful. The model is trained so that texts with similar meaning land close together in this high-dimensional space, and texts with different meaning land far apart. "Reset my password" and "recover account access" end up as nearby points even though the surface words differ, because the model learned that they mean nearly the same thing. Semantic similarity, a fuzzy human notion, becomes geometric distance, a number you can compute.

The crucial discipline here is that you must embed your chunks and your queries with the same model, so that they live in the same space and their distances are comparable. Mixing embedding models, or comparing vectors from different model versions, produces distances that mean nothing. Embedding quality is also domain-specific, so a model strong on general text can be weak on specialized jargon, and many retrieval models encode a short query and a long passage differently, which is part of why query and chunks must go through the same model in the same way. Closeness is commonly measured with cosine similarity, which compares the direction two vectors point rather than their magnitude, so it cares about what a text is about rather than how long it is; note this is a similarity rather than a distance, since cosine distance is one minus it. Dot product and Euclidean distance are also widely used. Once every chunk is a vector and every query is a vector in that same space, "find the relevant passages" reduces to a purely geometric question: which chunk vectors are closest to the query vector? That is a question we know how to answer fast, which is the subject of the next section.

The vector index: why you cannot just compare everything

Finding the closest chunk vectors to a query vector sounds simple, and for a small corpus it is. Compute the distance from the query to every chunk, keep the closest handful, done. This is exact nearest-neighbor search, and its cost grows in direct proportion to the number of chunks. That is fine for a thousand chunks and hopeless for ten million, because you would be doing ten million distance computations on every single query, written O(n), while a user waits.

Exact search compares the query to every vector, O(n) per query; ANN structures like HNSW and IVF trade a little recall for large speedups.

The escape is to give up on finding the exact closest vectors and settle for finding almost-certainly the closest ones, much faster. This is approximate nearest neighbor search, ANN, and it is what every production vector index actually runs. The trade is explicit and tunable: you accept that occasionally the index will miss a true nearest neighbor, in exchange for answering in sublinear time instead of linear. Graph indexes like HNSW get roughly logarithmic behavior; cluster indexes like IVF are sublinear but not that aggressive. The fraction of true nearest neighbors the index actually returns is its recall, and you trade recall against speed: higher recall costs more time, lower recall buys speed.

Two families of structure dominate. HNSW, hierarchical navigable small world graphs, builds a layered graph where each vector links to its neighbors, with sparse long-range links up top for fast traversal and dense links at the bottom for precision. A search enters at the top, greedily hops toward the query, and descends layer by layer, reaching the neighborhood of the answer in a small number of hops rather than scanning everything. IVF, the inverted file approach, first clusters all the vectors into buckets, then at query time only searches the few buckets nearest the query, skipping the rest entirely. Both approaches, and the quantization tricks often layered on top to shrink the vectors in memory, are answering the same question: how do I avoid comparing the query to everything? The answer is always the same in spirit, spend some memory and some recall to buy a large cut in the number of comparisons, and it is the same bargain a database makes when it builds an index instead of scanning a table.

The two families also differ in operations, not just search. IVF must be trained on a representative sample to build its clusters and needs retraining as the data drifts, while HNSW needs no training but is costly to update, with deletes usually tombstoned and the graph degrading under heavy churn until a rebuild. That update cost often decides which index a team can live with.

Query-time retrieval

Now we are at query time, with a built index waiting. A user asks a question, and the first thing that happens is the same thing that happened to every chunk during ingestion: the question is embedded, with the same model, into a vector in the same space.

At query time the question is embedded with the same model, and ANN search returns the top-k chunks by vector similarity.

With the query vector in hand, you ask the index for the top-k nearest chunk vectors, where k is a number you choose, commonly somewhere between five and a few dozen depending on how much context you can afford downstream. The index runs its ANN search and hands back that many candidate chunks, each with a similarity score, ranked closest first. That is the entire first-stage retrieval, and it is fast, typically on the order of milliseconds to low tens of milliseconds even over millions of vectors, depending on how the index is tuned, which is the whole point of the index. In practice the vector search is usually constrained by metadata filters (date, source, language, version, tenant), which both improves relevance and is the hook where access control is enforced. That access control is not optional: in any multi-tenant or permissioned corpus, retrieval must filter by the caller's access rights before the model ever sees a chunk, because a vector index has no notion of who may read what, and a leaked passage becomes a cited, fluent answer.

But be honest about what this score is and is not. The similarity score tells you how close two embeddings are, and embeddings compress a passage down to a few hundred to a few thousand numbers, which necessarily throws information away. Two chunks can score nearly identically against a query while only one actually answers it, because the compression blurred the distinction that mattered. First-stage vector retrieval is very good at pulling a relevant neighborhood out of millions of chunks, and only roughly good at ordering within that neighborhood. That gap between "in the right neighborhood" and "the actual best answer, ranked first" is exactly what the next two sections exist to close. You choose k larger than the number of chunks you ultimately want, precisely so that the real answer is somewhere in the candidate set even if it is not yet ranked first.

Hybrid retrieval: dense search has blind spots

Dense vector search, which matches on meaning, is strong but weak in a specific, predictable way. Because it matches on semantic similarity, it can miss the cases where the exact token is what matters. A part number like "X-4021-B", an error code, a person's surname, a rare technical term the embedding model barely saw in training, these are precisely the queries where "close in meaning" fails you, because the thing you need is an exact string, not a paraphrase.

Dense vector search misses exact keywords, IDs, and rare terms; combining it with sparse keyword search like BM25 and fusing the rankings covers both.

The fix is not to abandon vector search but to pair it with the older technology it was supposed to replace. Sparse keyword search, most commonly BM25, ranks documents by exact term matches weighted so that rare terms count for more and common ones count for less. BM25 is excellent at exactly what dense search is bad at: finding the document that contains this specific string. So you run both. The query goes to the vector index and to a keyword index at the same time, and each returns its own ranked list of candidates. This is hybrid retrieval, and in practice it beats either method alone across a wide range of real queries, because real query logs contain both "explain our refund philosophy," which wants semantic matching, and "what is fee code 30983," which wants exact matching.

The remaining question is how to combine two ranked lists whose scores are not comparable, a cosine similarity and a BM25 score living on different scales. The common answer is reciprocal rank fusion, which throws away the raw scores and combines the lists by rank position instead, rewarding documents that rank highly in either list. It is simple, it needs no score calibration, and it is hard to beat. The lesson underneath is worth keeping: when a single retrieval method has a known blind spot, the cheapest fix is often a second method with the opposite blind spot, fused together, rather than an heroic effort to make the first method perfect.

Reranking: precision on the shortlist

First-stage retrieval, dense or hybrid, is built for one job, to cut millions of chunks down to a few dozen candidates cheaply. It is deliberately not built for precision in ordering, because the thing that would give you precision is too expensive to run over millions of chunks. So you run it in two stages. Cheap and approximate first to get the shortlist, then expensive and precise on just that shortlist. This retrieve-then-rerank pattern is one of the most important structures in production RAG.

A cross-encoder rescores each candidate together with the query for far better precision than the first-stage score, at higher cost, so it runs only on the shortlist.

The reason first-stage retrieval cannot be precise is architectural. To make the index fast, you embed each chunk independently, ahead of time, with no knowledge of the query. The query and the chunk never actually meet; you only compare their two pre-computed vectors. A reranker throws out that separation. It is a cross-encoder, a model that takes the query and one candidate chunk together, as a pair, and reads them jointly to produce a single relevance score. Because it sees both texts at once, it can judge whether this chunk actually answers this query, catching relevance that the blurred, independent embeddings missed. It is far more accurate.

It is also far more expensive, and that is why it cannot run over your whole corpus. A cross-encoder has to run a full model forward pass for every query-chunk pair, so its cost scales with the number of candidates. Running it on ten million chunks per query is out of the question; running it on the fifty candidates that first-stage retrieval already shortlisted is entirely affordable. So the two stages divide the labor by their nature. The fast approximate stage handles the impossible scale, taking millions down to dozens. The slow precise stage handles the ordering, taking dozens down to the handful you will actually show the model, correctly ranked. You get most of the accuracy of comparing the query against everything, at a tiny fraction of the cost, which is the entire art of two-stage retrieval. It is worth holding a latency and cost budget for the whole path, since the reranker and the generation call, not the vector search, usually dominate both.

Prompt assembly and the context budget

Now you have a small set of high-quality, well-ranked chunks. The next job is to build the actual prompt you send to the model, and this is less obvious than it looks. The model has a context window, a hard limit on how many tokens it can read at once, and that budget has to hold the system instructions, the retrieved chunks, the user's question, and enough headroom for the answer the model still has to write. You cannot just dump everything in.

Fit the best-ranked chunks into the context window alongside instructions and question, keep the strongest chunks, leave room for the answer, and stop well short of the limit, because more context is not better.

The instinct, once context windows got large, was to stop worrying and stuff in as many chunks as fit. That instinct is wrong, and for reasons that go beyond cost and latency, though those are real too, since every token you send is money and time. The deeper problem is that more context often makes the answer worse. Models attend unevenly across a long context, and irrelevant passages act as distractors that pull the answer off course. Ten sharp, relevant chunks produce a better answer than fifty chunks where forty are noise. So prompt assembly is an editing job, not a dumping job. You take the reranked list and include the best chunks until you hit a sensible budget, well short of the physical maximum, and you stop.

Two details in assembly pay off later. First, tag each chunk with its source, a document title, a URL, an ID, so the model can cite where each claim came from and so you can trace any answer back to the passage that produced it. This is not decoration; it is how you debug the system and how a user learns to trust it. Second, order matters. Because models weight the beginning and end of a context more heavily than the middle, a phenomenon we will come back to under failure modes, it is worth placing the strongest chunks where the model attends best rather than burying them in the center. The prompt you assemble is the model's entire world for this one question. Everything it will say comes from what you chose to put in front of it, which is another way of saying the quality ceiling was set upstream, in retrieval, and prompt assembly only decides how much of that ceiling you actually reach.

Generation, grounded and honest

Finally the model gets its turn, and after all the retrieval machinery, its job is deliberately narrow. It is not being asked what it knows. It is being asked to read the passages you retrieved and answer the question from them, and only from them.

The model answers strictly from the retrieved passages, is instructed to say it does not know when the context lacks the answer, and cites the sources it used.

The instructions you give it are what make this trustworthy, and they run against the model's natural tendencies. You tell it to answer using only the provided context. You tell it, explicitly, that if the context does not contain the answer, it should say so plainly rather than reaching into its training knowledge or inventing something. This matters because a model's default, as we saw at the start, is to produce a fluent answer whether or not it has grounds for one, and "I do not know based on the provided documents" is a far more useful response than a confident fabrication. Getting the model to abstain when the context is insufficient is one of the highest-value behaviors you can instruct, and one of the hardest to enforce completely. You also ask it to cite, to point each claim back to the source chunk it came from, which turns the answer from an assertion into something a user can verify.

Grounding is the word for tying the answer to the retrieved evidence, and it is the whole reason RAG reduces hallucination. But notice the limit precisely. Grounding constrains the model to the context; it does not make the context correct. If retrieval delivered the wrong passages, the model will ground its answer firmly in the wrong passages and cite them neatly. The model has no way to know that the retrieval upstream failed. This is why, no matter how good your prompt and your instructions, you keep coming back to the same sentence: retrieval quality caps everything. The model is the last stage and the least of your worries. What you feed it is the whole game.

Where RAG breaks

A demo RAG system works on the first ten questions you try. A production one has to work on the ten thousandth question asked by someone who phrases things strangely, about a document that changed last week. The gap between those two is a set of specific, recurring failure modes, and knowing them by name is most of what separates a system that degrades gracefully from one that fails silently.

Four common failures: retrieval misses the relevant chunk, the right chunk is retrieved but buried, the index is stale, and the model ignores the context and hallucinates anyway.

The first and most common failure is a retrieval miss. The chunk that answers the question exists in your corpus, but retrieval did not return it, so it never reached the model. This is a recall failure, and it is invisible unless you measure for it, because the model will happily produce an answer from whatever it did get. The causes trace straight back to earlier decisions: chunks split so the answer is fragmented across a boundary, a query whose wording is semantically distant from the passage, a rare term that dense search glossed over and that no keyword search was there to catch. Most retrieval misses are chunking and hybrid-retrieval problems in disguise. One further fix is to transform the query before retrieval, rewriting a terse or conversational question into a fuller standalone query or expanding it with likely synonyms, so the embedded query lands nearer the passage that answers it. Questions that need evidence found only after a first lookup push toward iterative retrieve-reason-retrieve loops, at the cost of added latency and more places to fail.

The second failure is subtler. The right chunk was retrieved, but it was ranked low and buried in the middle of a long context, and the model, attending less to the middle, effectively overlooked it. This is the "lost in the middle" effect, and it is why reranking and disciplined, shorter contexts matter so much: it is not enough to retrieve the answer, you have to place it where the model will actually read it. The third failure is staleness. The corpus moved, a policy changed, a document was deleted, but the index still holds the old vectors, so the system confidently serves outdated answers, which we address in the next section. The fourth is the one people fear most and that grounding is meant to prevent: the model ignores the provided context and answers from its training memory anyway, hallucinating in spite of correct retrieval. It happens less with good instructions and good models, but it happens, which is why you verify groundedness rather than assume it. These four fail in different places, and no single fix touches all of them, which is exactly why you have to measure the two halves separately.

Evaluating a two-part system

You cannot improve what you do not measure, and RAG is deceptively hard to measure because it has two halves that fail differently, and a single end-to-end "was the answer good" score cannot tell you which half failed. An answer can be wrong because retrieval never found the evidence, or because the model fumbled evidence it was handed. Those need opposite fixes. So you evaluate retrieval and generation separately.

Retrieval metrics like recall@k and precision measure whether the right chunks were found; generation metrics like faithfulness and answer relevance measure what the model did with them.

On the retrieval side, the metrics are the familiar ones from information retrieval, and they need a set of questions with known relevant chunks to score against. Recall@k asks whether the relevant chunk was somewhere in the top k you retrieved, and it is the single most important number in the whole system, because a chunk that was never retrieved can never be used, no matter how good everything downstream is. If recall@k is low, nothing else you tune will save you, and you go back to chunking, embeddings, and hybrid retrieval. Precision asks what fraction of what you retrieved was actually relevant, which matters because irrelevant chunks are the distractors that hurt generation. Ranking-aware metrics tell you not just whether the right chunk was retrieved but whether it was ranked near the top, which is what reranking is meant to move.

On the generation side, the questions are different. Faithfulness, also called groundedness, asks whether every claim in the answer is actually supported by the retrieved context, or whether the model added things that were not there. This is your direct hallucination measurement, and it is the one to watch most closely, because a faithful answer that admits ignorance is safer than an unfaithful one that sounds complete. Answer relevance asks whether the answer actually addressed the question the user asked, rather than something adjacent. These generation metrics are increasingly scored by using a strong model as a judge against the context, which is imperfect and needs its own spot checks against human judgment, but is far better than flying blind. The discipline that matters is the separation itself. When quality drops, the first question is always which half broke, and only separate metrics can answer it.

Keeping the index fresh

The last production concern is time. Your corpus is not a fixed thing you index once. Documents get edited, added, and deleted continuously, and a RAG system whose index reflects last month's data will confidently serve last month's answers. Ingestion is not a one-time job you run at launch. It is a pipeline that has to run for as long as the system lives.

Documents change, so ingestion runs continuously: incremental upserts for new and edited content, and deletes for removed content, so the index never drifts from the corpus.

The mechanics are a set of operations you have to build for from the start. When a document is added or edited, you re-chunk and re-embed just that document and upsert its vectors into the index, updating in place rather than rebuilding the world. When a document is deleted, you must delete its vectors too, and this is the step teams forget, which is how a system keeps citing a document that no longer exists. Detecting what changed is its own small problem, usually solved with content hashes or modification timestamps so you re-process only what actually moved rather than re-embedding the entire corpus on every run, which would be ruinously expensive at scale. Repeated work is also cached, embeddings for unchanged content and answers or retrieval results for common queries, with care to invalidate when documents change.

Two larger events force bigger work. The first is changing your embedding model. Because vectors from different models are not comparable, upgrading to a better embedding model means re-embedding your entire corpus, not just new documents, and rebuilding the index. This is expensive and is why the choice of embedding model has real switching cost, so it is worth getting reasonably right early. The second is versioning. In a serious system you want to know which version of a document produced a given answer, both to debug and to handle the case where a user is asking about the state of things at a particular time. Treating ingestion as a living pipeline, with incremental updates, honest deletes, and a plan for re-embedding, is what separates a RAG demo that impressed everyone once from a RAG product that stays correct in the months after launch.

The patterns worth keeping

Strip away the specifics and RAG teaches a handful of lessons that transfer to systems that have nothing to do with language models.

Four transferable lessons: retrieval quality caps everything downstream; retrieve cheap then rerank precise; measure retrieval and generation separately; ground and cite to control hallucination.

The first is that retrieval quality caps everything downstream. The model, the prompt, the instructions, none of them can rise above the quality of the passages retrieved, and no amount of polish on the visible final stage compensates for a weak stage upstream. This is a general truth about pipelines: the earliest stage that loses information sets a ceiling that later stages can only fail to reach, never exceed. Find that stage and fix it there.

The second is the two-stage pattern, cheap and approximate to cut the field, then expensive and precise on the shortlist. Retrieve-then-rerank is one instance, but the shape appears anywhere you cannot afford to run your best method over everything: filter broadly with something fast, then judge narrowly with something accurate. The third is to measure the halves of a compound system separately, because a single end-to-end score hides which part failed and sends you tuning the wrong thing. The fourth is that grounding and citation are how you make a generative system trustworthy: constrain the output to verifiable evidence, show the evidence, and let the answer be checked. RAG is a retrieval problem wearing a language model's clothes, and once you see it that way, the model stops being the mystery and the retrieval becomes the craft.

Where to go from here

If this was useful, the thing worth keeping is the reframe. When a RAG system disappoints, the instinct is to reach for a bigger model or a cleverer prompt, and the answer is almost always upstream, in how you chunked, what you embedded, whether you retrieved the right passages at all, and whether you ever measured it. Fix retrieval and the model looks brilliant. Ignore retrieval and no model will save you.

I teach RAG and the rest of system design this way, from first principles with real diagrams and the trade-offs that only surface in production, as an interactive course at systemdesign.academy. The foundation lessons are free and need no signup.

Read the free lessons: https://systemdesign.academy

How Google Search Serves Results: The Inverted Index, Scatter-Gather, and Ranking

RONI DAS — Sat, 11 Jul 2026 11:39:20 +0000

You type four words into a box, press enter, and roughly two hundred milliseconds later you get ten results drawn from a corpus of several hundred billion documents. That sentence contains the entire engineering problem. No clever data structure makes searching hundreds of billions of documents cheap. There is only a stack of decisions, each trading some generality away to make the common case fast.

Most explanations stop at "it uses an inverted index," which is true and roughly as useful as saying a database uses a B-tree. The inverted index is the opening move. The interesting part is everything that has to be true around it: how the index is compressed so it fits in memory, how it is split across thousands of machines, how a query is broadcast to all of them and the answers merged, why that merge is dominated by the slowest machine rather than the average one, and why ranking is a cascade of progressively more expensive models rather than one scoring function.

We start from the constraints and the arithmetic, because the numbers rule out most architectures before you draw anything. One note on figures: Google does not publish current serving internals, so the numbers here come from public statements, the published research the system is built on, and back of envelope arithmetic consistent with both. They are the right order of magnitude, meant to drive reasoning rather than to be quoted as production metrics.

What the system actually has to do

The functional surface is small. Accept a text query, return a ranked list of documents with titles and snippets, and do it for anyone on the internet without authentication.

The design is set by a latency budget spent across four stages, not by the feature list, which fits in a sentence.

The constraints are where the design lives, and the tightest one is a budget rather than a feature. Take two hundred milliseconds as the total user perceived target. Network round trip to the nearest edge and back consumes eighty to a hundred of that on a typical connection. Query understanding, meaning parsing, correction, and expansion, has to fit in a handful of milliseconds. Snippet generation and page assembly take a slice at the end. That leaves something like thirty to fifty milliseconds for the part everyone thinks of as search: reaching into several hundred billion documents, finding candidates, and ranking them.

Thirty to fifty milliseconds is the number that kills most designs. It rules out disk on the critical path, any algorithm linear in corpus size, and a single machine, since no single machine holds the index. And because the answer must be assembled from many machines, it quietly rules out waiting for all of them, for reasons we will get to.

The other constraints are softer. Availability has to be very high, because search is the entry point to everything else. Freshness is tiered: a breaking news page needs to be findable in minutes, a decade old reference page can wait a week. Consistency you get to relax almost completely. Nobody can tell whether their query hit a replica nine minutes behind, and nobody would care if they could.

Three subsystems, and two of them are offline

It helps to separate web search into three systems that share a name and share nothing else.

Crawling and indexing are throughput problems measured in hours. Serving is a latency problem measured in milliseconds. They meet only at the index artifact.

Crawling discovers and fetches documents. Its hard problems are URL frontier scheduling, duplicate detection, per host politeness, and deciding what is worth recrawling and how often. Indexing turns fetched documents into the structures the serving path reads: it parses HTML, extracts text and anchor text, detects language, computes document level signals, assigns internal document identifiers, and builds the inverted index. Both are measured in documents per second and hours of wall clock.

Serving is the only one that runs while a user waits, and it is the subject of this piece. The two offline systems exist to move work out of the online one. Every signal computed at index time, every posting list presorted, is latency the serving path does not pay. When you see work in a search system, the first question is whether it can move to index time. That question drives more of the design than any data structure choice.

The numbers that set the architecture

The corpus is on the order of four hundred billion documents. Query volume is around eight and a half billion searches per day, roughly one hundred thousand per second on average, with peaks a few times that.

Four hundred billion documents times a thousand terms each is four hundred trillion postings. That number, and the memory it implies, sets the shard count.

A typical indexed document contributes on the order of one thousand distinct terms after tokenization. Four hundred billion documents times a thousand terms is four hundred trillion postings, where a posting is one entry saying "this term appears in this document." Uncompressed, a document identifier alone needs five bytes at that corpus size, so the raw index is petabytes before you store anything about the occurrence. Compressed to roughly one and a half bytes per posting, which is achievable and we will see how, the identifier and frequency index lands around six hundred terabytes. Add positions, needed for phrase queries and proximity scoring, and it roughly triples.

That has to live in memory, because the budget forbids disk. Take the identifier and frequency index, the part every query scans, as the thing that sets the machine count. If a serving machine dedicates two hundred and fifty six gigabytes of RAM to it, one complete copy needs on the order of two thousand machines. Positional data does not change that number, because it is held in a separate co-located tier and consulted only for the small set of candidates where phrase or proximity scoring actually needs it, rather than being decoded for every posting scanned. Two thousand is therefore the natural fanout of a query, and each machine holds about two hundred million documents. Then multiply by replication, because one copy cannot absorb a hundred thousand queries per second, let alone survive a machine failure. The fleet is many complete replicas of that two thousand machine set, spread across data centers, each able to answer any query independently.

The arithmetic has already decided several things. The index must be sharded. It must be in memory. Compression is what makes the machine count affordable. And a query has to touch about two thousand machines, which is the fact that will come back to hurt us.

The inverted index, and why the obvious index is wrong

The natural way to store documents is a forward index: document identifier maps to the terms in that document. It is the shape documents arrive in, and it is exactly wrong for search.

A forward index answers "what is in this document." A query asks the opposite question, so the index is inverted to answer "which documents contain this term."

A query does not ask what is in document 8,412,993. It asks which documents contain "distributed" and "consensus." Answering that from a forward index means scanning every document, the linear cost the budget forbids. So you invert it. The inverted index maps each term to a posting list, the sorted list of documents containing it. A single term query becomes a dictionary lookup and one list read. A two term query becomes two lookups and an intersection of sorted lists.

The structure has two parts. The lexicon maps a term string to metadata about its posting list: how many documents contain it, and where the list lives. It is small relative to the postings, tens of millions of distinct terms rather than hundreds of billions of entries, and it stays fully in memory in a compact form such as a finite state transducer or a front coded sorted array. The postings are the bulk. Both are built offline, so the serving path never sorts, never merges, and never builds anything. It reads.

The forward index does not disappear. Snippet generation needs document text, so a separate document store, sharded the same way, holds processed content. It is read only for the handful of documents that survive ranking, never for candidates.

What a posting list actually contains

A posting list is not just document identifiers, and the extra content is what makes ranking possible without reading documents.

Each posting carries the document identifier, the term frequency, and the position and field context of each occurrence, which is everything early scoring needs.

Each posting carries the document identifier, the number of occurrences, and for each occurrence a position plus context about where it happened. Position is what makes phrase search possible: "machine learning" as a quoted phrase requires documents where the position of "learning" is exactly one greater than that of "machine." The context bits encode whether the occurrence was in the title, a heading, the URL, anchor text, or the body, along with a coarse emphasis marker. The original Google paper described exactly this, packing a hit into two bytes with bits for capitalization, font size, and position, and splitting hits into fancy hits for titles, anchors, and URLs versus plain hits for body text.

The packing matters beyond space. Because occurrence context lives in the posting, a leaf server can score a candidate without touching the document, distinguishing a title match from a body match using data already in the cache line it just read. That is the same move again: do it at index time so the serving path does not have to.

Anchor text deserves a specific mention because it is one of the ideas that made web search work. When page A links to page B with the text "distributed consensus," that text is indexed as if it belonged to page B. You get descriptions written by other people, often better than what a document says about itself, and you can index documents whose content you cannot parse at all.

Compression is the reason this fits in memory

At four hundred trillion postings, the difference between four bytes and one and a half bytes per posting is the difference between five thousand machines and two thousand for a single index copy. Compression is a first class architectural concern, not a storage detail.

Sorting document identifiers turns large absolute numbers into small gaps, and small gaps compress into one or two bytes each.

Posting lists are sorted by document identifier, so instead of storing identifiers, store the gaps between consecutive ones. A list containing 1,004,215 then 1,004,318 then 1,004,902 becomes 1,004,215 then 103 then 584. The absolute values need twenty bits each. The gaps need seven and ten. This is delta encoding, and it works better the denser the list is, which is convenient because the longest lists are the ones worth compressing.

Then you need a variable length code for small integers. Variable byte encoding uses seven bits of each byte for data and the eighth as a continuation flag, so a gap under 128 costs one byte. Modern implementations usually prefer block based schemes such as PForDelta or SIMD friendly binary packing, which encode a block of 128 gaps at the bit width most of the block needs and store rare large outliers separately. These decode several times faster because they avoid an unpredictable branch on every value, and decode speed is what matters when you decode tens of millions of postings per query per machine. Elias-Fano is another option, close to the information theoretic bound while still supporting random access into the list, which matters for the skipping we are about to need.

One more trick has a large payoff. Document identifiers are assigned by the indexing pipeline, and you choose the assignment. Assign nearby identifiers to similar documents, for instance by sorting by URL so pages from one site get adjacent identifiers, and posting lists become clustered, gaps shrink, and the same encoder produces a meaningfully smaller index. That is a free reduction in machine count bought entirely at index time.

Splitting the index: by term or by document

Two thousand machines have to divide the index, and there are exactly two natural cuts. The choice is not close, but the losing option is the one that looks better on paper.

Term partitioning sends each query to a few machines but distributes load terribly. Document partitioning sends every query everywhere and is still the right answer.

Term partitioning assigns each term's entire posting list to one machine. A two term query touches exactly two machines, which sounds wonderful. It fails three ways. Load follows term popularity, which is Zipfian, so the machine holding a common term is hammered while the one holding rare technical vocabulary sits idle. Posting list lengths follow the same distribution, so shards are wildly unequal and a very common term's list may not fit on one machine. Worst, intersecting two terms means one machine ships its posting list to the other, which for common terms is tens of millions of postings crossing the network inside a thirty millisecond budget.

Document partitioning assigns each machine a subset of documents and builds a complete, self contained inverted index over just those documents. Every query goes to every shard, which does the full retrieval and ranking job over its own two hundred million documents and returns its own top results. Load is even by construction because documents are assigned by hash, index sizes are even for the same reason, and no posting list crosses the network because every intersection is local. The cost is a fanout of two thousand instead of two. Every production web scale search system makes this trade and then spends serious engineering on the consequence.

Scatter and gather

With document partitioning, the query flow is a tree, and the aggregation logic is where correctness gets subtle.

The root broadcasts one query to every leaf, each leaf returns only its local top results, and the root merges a few tens of thousands of candidates rather than billions.

A front end runs query understanding and hands a structured query to a root server. The root broadcasts to every leaf shard, in practice through one or two intermediate levels so no single machine manages two thousand outbound RPCs. Each leaf runs retrieval and scoring against its own index and returns its local top k, where k is small, on the order of twenty to a hundred document identifiers with scores. Intermediates merge their children and pass up their own top k. The root merges what it receives, applies whatever global reranking it can afford, fetches titles and snippets for the survivors from the document servers, and returns the page.

The volume reduction is the point. Two thousand leaves returning fifty results each gives the root one hundred thousand candidates, a trivial merge. If leaves returned every boolean match instead, the root would receive tens of millions and the network would be the bottleneck. This works because top k merging is decomposable: if a document's score depends only on that document and the query, the global top ten is guaranteed to lie inside the union of the per shard top tens. There is a condition hiding in that sentence. Scores from different shards have to be on the same scale, and the usual relevance weightings depend on corpus wide term statistics such as document frequency, which a shard can only see locally. So those statistics are computed globally at index time and distributed to every shard, rather than each shard computing its own. A shard scoring against its own local statistics would produce numbers that are not comparable, and the merge would quietly return the wrong ten documents. That guarantee, once it holds, licenses the architecture and also constrains ranking. Any signal that depends on other documents in the result set, such as diversity or cross shard deduplication, cannot be applied at the leaf and has to wait for the root, where the candidates finally meet.

Query understanding happens before retrieval

Before anything touches the index, the raw string becomes a structured query. This stage has an outsized effect on quality relative to how little budget it gets.

The raw string is normalized, corrected, segmented, and expanded into a structured query, and each step can change the result set completely.

Tokenization and normalization split the string, fold case, handle punctuation by context, and deal with scripts that have no whitespace word boundaries. Spelling correction runs against a model built from query logs, since the best evidence that "recieve" means "receive" is that millions of people typed the first and immediately typed the second. Segmentation decides "new york times" is one entity, not three words. Expansion adds synonyms and morphological variants, so "running shoes" can also match "run" or "sneakers," with expansion terms weighted below the originals so they do not overwhelm exact matches.

Intent classification decides what kind of answer the query wants. Navigational queries need a different result shape than informational ones, local intent triggers a geography dependent path, and some queries route to separate verticals so the final page is assembled from several backends.

Learned models live here too. Google's public disclosures describe RankBrain, introduced in 2015 for handling never before seen queries, and BERT based language understanding rolled out in 2019, which they said affected roughly one in ten searches in English in the United States by better modeling prepositions and word order. The architectural detail that matters is that these models are expensive, so they sit here at a fanout of one, before the broadcast, rather than at the leaves where their cost would be multiplied by two thousand.

You cannot afford to read the whole posting list

Inside a leaf, the naive algorithm intersects the posting lists for all query terms and scores every survivor. For rare terms that is fine. For a common word, the posting list on a single two hundred million document shard can hold tens of millions of entries, and scoring all of them does not fit.

Skip pointers let the intersection jump over regions that cannot match, and score bounds let the scorer skip documents that cannot reach the current top k threshold.

Two families of technique make this tractable. The first is skipping. Posting lists carry skip pointers, a sparse auxiliary structure that lets a reader jump forward to approximately a target identifier without decoding everything in between. Intersecting a rare term with a common one, you drive from the rare list and use skip pointers to advance the common one, so cost becomes proportional to the shorter list rather than the longer one. Always driving the intersection from the shortest list is the single most important implementation detail in a boolean retriever.

The second family is dynamic pruning, and WAND is the canonical algorithm. If you already have k candidates and the lowest scored 4.7, then any document whose maximum achievable score is below 4.7 cannot enter the top k and never needs scoring. To exploit that, the index stores an upper bound on each term's contribution. During evaluation you keep a running threshold and use those bounds to decide cheaply whether a candidate could beat it. Block-max WAND stores bounds per block of the posting list rather than per list, making them much tighter and letting the algorithm skip whole blocks. In practice these cut fully scored documents by one or two orders of magnitude with no change to the returned top k, which makes them exact rather than approximate. That distinction matters. You can turn them on without a quality review.

Ranking is a cascade, not a function

There is a tempting model where ranking is one scoring function applied to every match. At this scale that is not just wrong, it is impossible. Ranking is a funnel, and each stage earns the right to spend more compute per document by having fewer documents.

Each stage cuts the candidate count by orders of magnitude and spends orders of magnitude more compute per document, so no single stage dominates the budget.

The first stage is retrieval, the boolean and pruned scan above. It runs against the whole shard at nanoseconds per document and produces candidates. The second is a cheap ranker, a linear or tree based model over features already present in the posting and document metadata: term frequencies, field matches, proximity, quality priors such as PageRank, language, freshness. It costs a few hundred nanoseconds per document and reduces tens of thousands of candidates to a few hundred with high recall of the genuinely good ones. Twenty thousand candidates on one shard at two hundred nanoseconds each is about four milliseconds, which is the sort of number that has to be true for the stage to exist at all.

The third stage holds the expensive machinery, running on hundreds of documents rather than millions. Learned ranking models operate over rich features, and transformer based relevance models can afford a cross attention pass over the query and the specific passage that matched. Per document that pass is enormously more expensive than anything earlier, but it runs once per query rather than once per shard, and the few hundred candidates go through as a single batch on accelerator hardware, which is what keeps the stage inside a few milliseconds instead of a few hundred. Google's public description of passage ranking, scoring individual passages within a long document rather than the document as a whole, belongs here. It may sit at the leaf, an intermediate, or the root depending on which features it needs.

The final stage is at the root and concerns the result set rather than individual documents: deduplicating near identical pages, enforcing diversity so you do not get ten results from one site, blending verticals, applying policy filters, and personalizing lightly on location and language. The cascade is a compute allocation strategy. No stage dominates the budget, because candidate count falls about as fast as per document cost rises, and quality lands close to running the expensive model on everything at a fraction of the price.

Three caches, at three different layers

Query traffic is Zipfian the way term frequency is, so caching helps a great deal. The right answer caches at more than one layer, because the layers have different hit patterns.

A result cache serves repeated queries outright, a posting list cache serves repeated terms across different queries, and a document cache serves snippets.

The result cache sits at the front and maps a normalized query plus its context, meaning language, country, and personalization bucket, to a rendered result set. A hit skips the entire backend. Because a small set of queries repeats constantly, this absorbs a substantial share of traffic, commonly reported in the thirty to fifty percent range for web search workloads. Its weakness is that the tail of unique queries is enormous and a large fraction of daily queries have never been seen before. Entries also need short time to live values, because a cached result for a news query goes stale in minutes.

The posting list cache sits at the leaf and holds decompressed posting list blocks for hot terms. Its hit pattern is complementary: two completely different queries sharing one common term both benefit. Since term frequency is far more skewed than query frequency, this cache gets high hit rates even on queries the result cache has never seen. The document cache holds processed content for snippet generation, a surprisingly expensive step that has to find and highlight the passage justifying the result.

The tail is the system

Here is the failure mode that defines high fanout architectures, and it is not intuitive until you see the arithmetic.

With a fanout of two thousand, a leaf p99 of ten milliseconds means essentially every query waits on a straggler. The root cannot afford to wait for everyone.

Suppose a leaf responds within ten milliseconds ninety nine percent of the time. That is a good leaf. Now broadcast to two thousand of them and wait for all responses. The probability that every leaf comes back within ten milliseconds is 0.99 raised to the two thousandth power, about two in a billion.

At a fanout of two thousand, a one percent tail is not an outlier, it is every single query.

If the root waits for everyone, the p50 of the whole system is roughly the p99.97 of a single leaf. Your median user experiences your worst case. Jeff Dean and Luiz Barroso made this point precisely in "The Tail at Scale": with a hundred servers and a one percent chance of a one second response, sixty three percent of user requests exceed one second. Fanout does not average the tail out, it amplifies it.

The leaf tail comes from ordinary, unavoidable causes: a garbage collection pause, a background compaction, a noisy neighbor on the host, a kernel scheduling hiccup, a queue that briefly built up behind an expensive query. You cannot eliminate them, so the architecture has to tolerate them.

The mitigations are well established. The root sets a deadline and returns once it has heard from, say, ninety nine percent of leaves, accepting that a few shards contributed nothing. Because shards are random subsets of documents, missing one out of two thousand degrades recall by a fraction of a percent and is almost never visible, a remarkable trade for cutting p99 by an order of magnitude. Hedged requests send a duplicate to a second replica if the first has not answered within, for example, the p95 latency, and take whichever returns first, costing a few percent extra load. Tied requests refine that: send to two replicas immediately and have each cancel the other as soon as it starts executing. Below all of it sits hygiene, including micro-partitioning shards so a slow machine's load redistributes quickly, and putting a persistently slow replica into probation until it recovers.

A second, sharper failure mode bites during deploys. A new index version is a new set of files with a completely cold posting list cache. Shift traffic to it all at once and every leaf takes cache misses on every term simultaneously, latency spikes fleet wide, the root's deadline starts dropping large numbers of shards, and result quality visibly degrades. The fix is to warm the new index with shadow traffic before it serves anything real, then shift gradually so no large fraction of the fleet is ever cold at once. This shape, where a correlated cache invalidation converts a healthy system into an overloaded one, is not specific to search.

Keeping the index fresh

A fully rebuilt index is a clean artifact but a slow one. If a rebuild takes hours, a new document can be unfindable for hours, which is unacceptable for news, sports, and anything else people search for while it is happening.

A large, slowly rebuilt base index is queried alongside a small, constantly updated real-time index, and the results are merged at query time.

The standard answer is a two tier index. A large base index is built by the batch pipeline on a slow cadence. Alongside it sits a small real-time index, held in memory, receiving new and updated documents continuously. Every query goes to both and the leaf merges the two result sets. The real-time tier is small enough to be less compressed and more mutable, and periodically its contents fold into the base index during the next build. Deletions are handled by a separate deleted document bitmap consulted at query time, because physically removing a posting from a compressed list means rewriting it, which you would rather do during the merge.

Google's public work describes exactly this evolution. The original architecture rebuilt the index in large batch passes. Percolator, published in 2010, replaced the batch pipeline with incremental processing built on transactions and observers over Bigtable, and the paper reports that it halved the average age of a document in Google search results. The Caffeine indexing system that shipped that year was the production result, updating the index incrementally as documents were crawled rather than rebuilding layers wholesale.

Crawl scheduling completes the picture, because freshness is decided before indexing runs. Recrawl frequency is estimated per URL from observed change rates and importance, so a news homepage may be fetched every few minutes and a static reference page every few weeks. Spending crawl capacity where change actually happens is what makes minute level freshness affordable.

The trade-offs

Every choice bought something and cost something, and naming both sides is the difference between understanding the design and reciting it.

Document partitioning bought even load and local intersections, and cost a fanout of two thousand plus all the tail latency work that follows. Serving from memory bought the latency budget and cost a machine count driven entirely by index size, which is why compression became architectural. Aggressive compression bought that machine count and cost decode CPU on every query, which is why codecs that decode fast beat codecs that compress best. Dynamic pruning bought an order of magnitude fewer scored documents at no quality cost, but constrains scoring to functions where per term upper bounds are computable. The ranking cascade bought expensive model quality at cheap model cost, and cost a recall ceiling, since a document the cheap stage discards can never be recovered later. Deadline based aggregation bought a controlled p99 and cost a small amount of recall. Two tier indexing bought minute level freshness and cost a merge on every query plus two index formats to operate.

The patterns that transfer

Strip away the search vocabulary and these decisions show up constantly elsewhere.

Four moves carry over to most high fanout systems: invert the structure, shard by the unit you scan, cascade your compute, and never wait for the slowest replica.

Invert the data structure to match the question, not the data. The forward index is how documents arrive and the inverted index is how they are asked about, and building the second one offline is what makes the query cheap. Any time a read path is scanning to answer a question, ask what structure would answer it with a lookup, and whether you can build that structure at write time.

Shard by the unit you scan, not the unit you look up. Term partitioning optimizes the lookup and destroys the scan by skewing load and forcing data across the network. Document partitioning accepts broad fanout to keep every scan local and balanced. When the two conflict, the option that keeps work local usually wins, and you pay for it in fanout.

Structure expensive computation as a cascade. When you cannot afford your best model on every item, build a funnel of increasingly expensive stages where each cuts the candidate set by an order of magnitude. The cost is roughly that of the cheapest stage, the quality is close to the most expensive one, and the thing to watch is the recall ceiling each early stage imposes.

Never wait for the slowest replica. At any meaningful fanout the tail dominates, and the mitigations are always some combination of deadlines with partial results, hedged or tied duplicate requests, and taking slow replicas out of rotation. If your system fans out to more than about ten backends and you have not thought about this, your p99 is worse than you think.

Learn to see which of these levers your own bottleneck sits on, and the design stops being a memorized answer and starts being a method.

I teach system design this way, from first principles with real diagrams and the trade-offs that only surface in production, as an interactive course at systemdesign.academy. The foundation lessons are free and need no signup.

Read the free lessons: https://systemdesign.academy

Designing Uber Eats: The Three-Sided Marketplace, Dispatch, and ETA

RONI DAS — Sat, 11 Jul 2026 11:39:15 +0000

Food delivery looks like ride-hailing with a burger instead of a passenger. It is meaningfully harder, and understanding why is the key to the whole design. The difference comes down to timing: with a ride, the passenger is ready when the car arrives. With food, the meal is not ready, and the courier arriving too early wastes their time while arriving too late means cold food.

A visual from the full interactive lesson on systemdesign.academy.

Three sides, not two

A ride-hailing marketplace has two sides, riders and drivers. Food delivery has three: the eater who orders, the restaurant that cooks, and the courier who delivers. All three have to be coordinated in time. The eater wants hot food fast, the restaurant wants orders paced so its kitchen is not slammed, and the courier wants to be moving, not waiting. The system's job is to keep all three roughly happy at once, and their interests conflict.

This three-sided structure shapes the data model. You track eaters and their orders, restaurants and their menus and current kitchen load, and couriers with live location and availability. The order itself moves through a state machine: placed, accepted by restaurant, being prepared, ready, picked up, delivered. Much of the engineering is making those state transitions reliable across three parties who can each fail or go offline.

The dispatch problem

Dispatch is the matching engine: which courier gets this order. In ride-hailing you assign the nearest available driver. In food delivery, nearest-now is the wrong answer, because the food will not be ready for fifteen minutes. If you assign the closest courier the moment the order is placed, they arrive in five minutes and stand around for ten. The right question is which courier will be near the restaurant when the food is ready.

That means dispatch has to predict preparation time and assign against the future, not the present. A courier finishing a nearby delivery in twelve minutes might be a better match than one who is free now. Good systems also batch: if two orders come from the same restaurant heading the same direction, one courier can carry both, which is more efficient for the platform and often barely slower for the eater. Batching is a bin-packing problem layered on top of matching, and it is where a lot of the margin lives.

The matching itself is a geospatial query. You need to find couriers near a point quickly, which rules out scanning every courier. The standard tool is a spatial index, often geohashing (encoding a lat-long into a string prefix so nearby points share a prefix) or an H3 hex grid, so "couriers within two kilometers" becomes a cheap lookup over a few grid cells rather than a distance computation across the whole fleet.

ETA: the number everyone sees

The delivery estimate is the most visible output and one of the hardest to get right, because it is a sum of uncertain parts: time to assign a courier, courier travel time to the restaurant, food preparation time, and travel time to the eater. Each has its own variability. Travel time depends on live traffic; prep time depends on the dish and how busy the kitchen is; assignment time depends on courier supply nearby.

The honest way to build this is a model per component, trained on historical data. Prep time is learned per restaurant and per dish and adjusted for current order volume. Travel time comes from a routing engine using live traffic. You sum the components and, importantly, communicate a realistic estimate rather than an optimistic one, because a late delivery against a promised time hurts more than a longer honest estimate.

The core trade-off

The tension is efficiency versus experience. Batching orders and assigning couriers against future readiness maximizes courier utilization and keeps costs down, but push it too far and food sits, couriers get overloaded, and eaters wait. Assign eagerly and generously and eaters are delighted while your unit economics collapse. The dispatch system is constantly tuning that dial, per city, per time of day.

How the real systems do it

Uber and DoorDash both run dispatch as an optimization that assigns against predicted readiness, not current position, and both batch orders when routes align. Geospatial indexing (Uber open-sourced H3 for exactly this) makes the "who is nearby" query fast. ETA is a composition of learned sub-models over prep and travel time. The recurring theme is that the hard part is time, not distance.

I wrote the full breakdown, with diagrams and the data model, here: https://www.systemdesign.academy/interview/design-uber-eats

Designing a Real-Time Leaderboard with Redis Sorted Sets

RONI DAS — Sat, 11 Jul 2026 11:33:53 +0000

A leaderboard looks like a database sort with a LIMIT. It is not, and the reason is one query: a player wants to know their own rank. Answering that fast, for millions of players, updated live, is where the design gets interesting.

A visual from the full interactive lesson on systemdesign.academy.

Why the database sort fails

The obvious approach is a scores table with an index on score. Showing the top 100 is easy, just sort descending and take 100. But now a player ranked 4,000,012th opens the app and asks where they stand. To compute that rank, the database has to count how many players have a higher score, which means scanning or counting a huge slice of the table. Do that on every profile view for millions of users and your database melts. Worse, scores change constantly, so any precomputed rank is stale the moment you write it.

The Redis sorted set

The right tool is a sorted set, the data structure Redis calls a ZSET. It stores members (player IDs) each with a score, and it keeps them ordered by score automatically. What makes it perfect for leaderboards is the set of operations it supports in logarithmic time:

Update a score: ZADD board 5000 player42. It reinserts the player at the correct position in one call.
Get the top N: ZREVRANGE board 0 99 WITHSCORES returns the highest 100.
Get a player's rank: ZREVRANK board player42 returns their position directly.

That last one is the whole game. The rank query that would scan a database is a single fast operation on a sorted set, because the structure maintains ordering as an invariant. Getting your own rank, the page around your rank, and the top of the board are all cheap.

Under the hood a ZSET uses a skip list plus a hash map, which is why both "where does this member rank" and "who is at rank K" are fast. You get the ordering of a balanced tree with the point-lookup speed of a hash.

Handling scale with sharding

One Redis instance holds a lot, but a global board with tens of millions of active players and constant writes can outgrow a single node's memory and throughput. The usual move is to shard, and the clean way to shard a leaderboard is by segment rather than by hashing player IDs.

Most leaderboards are naturally partitioned already: per region, per game mode, per weekly season. Each of those is a separate sorted set, often on a separate instance, and each is independently small and fast. A player's rank within their own board is a local query. The hard case is a single global ranking across shards, because a player's global rank depends on every shard. You handle that either by accepting an approximate global rank or by periodically merging shard tops into a global board. Which brings us to the real scaling insight.

The approximate-rank trick

Exact rank for a player buried in the middle of tens of millions is expensive to keep globally consistent, and nobody actually needs it. The difference between rank 4,000,012 and rank 4,000,050 is meaningless to the user. So a common optimization is to bucket scores into ranges and report an approximate rank ("top 15 percent") for players outside the top tier, while keeping exact ranks only for the top few thousand where precision matters and competition is real. This turns a hard distributed-counting problem into a cheap histogram lookup.

Durability and the trade-off

Redis holds the board in memory, so you back it with a durable store. The source of truth for scores lives in your primary database; Redis is the fast serving layer you rebuild from that source if a node is lost. The trade-off across the whole design is memory for speed: you are keeping the ordered index resident in RAM because the alternative, computing ranks from a disk-based store on demand, cannot meet the latency a live leaderboard needs.

How the real systems do it

Game platforms and apps from mobile titles to fitness trackers lean on Redis sorted sets for exactly this reason, and managed offerings build leaderboard APIs directly on the ZSET primitives. The pattern is consistent: sorted set for the live index, segment sharding for scale, exact ranks at the top, approximate ranks in the long tail, and a durable database as the system of record.

I wrote the full breakdown, with diagrams and the data model, here: https://www.systemdesign.academy/interview/design-leaderboard

How Zoom Delivers Low-Latency Video: SFU vs MCU, WebRTC, and Simulcast

RONI DAS — Sat, 11 Jul 2026 11:33:49 +0000

A one-to-one video call is easy: two peers send packets directly to each other. A forty-person meeting is where the interesting engineering starts, because the naive approach falls apart fast, and the fix explains how every serious video platform is built.

A visual from the full interactive lesson on systemdesign.academy.

Why not just connect everyone to everyone

In a mesh, each participant sends their video to every other participant. With N people that is N minus 1 uploads from each device. At four people it is fine. At twenty, every laptop is trying to encode and upload nineteen copies of its own video, which no consumer uplink can handle. Mesh does not scale past a handful of people. You need a server in the middle.

SFU vs MCU

There are two kinds of server in the middle, and the difference is the central design decision.

An MCU (multipoint control unit) receives every participant's stream, decodes them, composites them into a single combined video, re-encodes it, and sends one stream back to each person. The client's job is trivial: it uploads one stream and downloads one stream. The cost is brutal on the server. Decoding and re-encoding video for every meeting is enormously CPU-heavy, and the re-encode adds latency and degrades quality.

An SFU (selective forwarding unit) does far less. It receives each participant's stream and simply forwards the packets to the others, without decoding or mixing. Each client uploads one stream but downloads one stream per other participant, then arranges them locally. The server is now cheap, just routing packets, which is why SFU is what almost everyone uses for large calls. The trade-off is more download bandwidth and more decoding work on the client, but modern devices handle that better than servers handle mass transcoding.

WebRTC and why UDP

The transport underneath is WebRTC, and the key choice it makes is UDP over TCP. TCP guarantees every byte arrives in order, retransmitting anything lost. For real-time video that guarantee is a liability. A packet that arrives 300 milliseconds late because it was retransmitted is useless, the moment it described has already passed. Better to drop it and move on. UDP does exactly that: it does not wait, does not retransmit by default, and lets the video codec conceal the occasional missing packet. Late video is worse than lost video.

Simulcast: the trick that makes SFU work

Here is the problem an SFU creates. It is just forwarding one stream from each sender, but the receivers are wildly different. One is on a laptop on fiber that wants full resolution; another is on a phone on spotty cellular that can barely handle a thumbnail. If the sender uploads one high-quality stream, the phone drowns. If it uploads one low-quality stream, the laptop gets a blurry picture.

Simulcast solves it by having each sender upload several versions of its video at once, say high, medium, and low resolution, as separate streams. The SFU then forwards the appropriate version to each receiver based on that receiver's measured bandwidth and the size it is displaying the video at. The person shown in a small tile gets the low stream; the active speaker shown large gets the high stream. The sender does a bit more encoding work, but the SFU can match every receiver's capacity without any transcoding. This is how one call serves a phone and a workstation gracefully.

The trade-offs

SFU plus simulcast is the modern default because it keeps the server cheap and adapts to heterogeneous clients. The costs are real: clients decode many streams, and uploading three resolutions raises the sender's outbound bandwidth. For very large webinars with thousands of passive viewers, platforms shift those viewers to a one-way streaming path (HLS or similar) since they only watch and never send, and pure real-time latency matters less for an audience than for participants.

How the real systems do it

Zoom, Google Meet, and Microsoft Teams all run SFU-based architectures with simulcast (or its cousin, scalable video coding) so one meeting adapts across devices and networks. WebRTC over UDP is the common transport, with the codec doing packet-loss concealment rather than the transport retransmitting. The through-line is the same everywhere: forward, do not mix; send multiple qualities; and never wait for a late packet.

I wrote the full breakdown, with diagrams and the data model, here: https://www.systemdesign.academy/interview/design-zoom

Designing a Stock Exchange Matching Engine: The Order Book and Price-Time Priority

RONI DAS — Sat, 11 Jul 2026 11:28:33 +0000

At the center of every stock exchange is one component that is smaller and stranger than people expect: the matching engine. Its whole job is to hold a book of buy and sell orders and pair them off according to a strict rule. Get that rule and its data structure right and everything else, the gateways, the market data feeds, the risk checks, is plumbing around it.

The core problem

Buyers post bids (the price they will pay) and sellers post asks (the price they will accept). A trade happens when a bid meets an ask at a compatible price. The engine has to decide, when a new order arrives, whether it crosses with an existing one, and if several resting orders qualify, which one gets filled first. That tiebreak rule has to be fair, deterministic, and fast, because on a busy day the engine sees enormous message volume and every participant expects the same rules.

Price-time priority

The standard rule is price-time priority. Better prices go first: the highest bid and the lowest ask sit at the front of the queue because they are the most competitive. Among orders at the same price, the one that arrived earlier goes first. This is the fairness guarantee that makes an exchange trustworthy. You cannot jump the queue by being bigger or louder, only by offering a better price or getting there sooner.

The order book data structure

The order book is two sorted structures, one for bids and one for asks. Each distinct price level holds a FIFO queue of orders at that price. When a market buy order arrives, you look at the best ask (the lowest sell price), fill against the orders in its queue in arrival order, and if the incoming order is larger than the resting quantity, move to the next price level and keep going until the order is filled or the book runs dry.

The operations you need are: find the best price on each side instantly, add an order at a given price level, remove or reduce an order when it fills or cancels. A common implementation is a map from price to a linked list of orders, with the best prices tracked so the top of book is O(1). Cancellations are frequent, often the majority of messages, so you also keep a direct index from order ID to its node in the book to cancel in constant time without scanning.

Why single-threaded is faster

Here is the counterintuitive part. You might reach for locks and multiple threads to go fast. The best matching engines do the opposite: they run the core matching loop on a single thread. Concurrency on a shared order book means locking, and lock contention plus cache-line bouncing between cores costs more than it saves. A single thread processing a stream of orders from an in-memory queue has no locks, perfect cache locality, and, just as important, deterministic behavior. The same sequence of orders always produces the same fills, which is essential for auditing and for rebuilding state after a crash.

The trade-off is that one thread caps your throughput at what a single core can do. Exchanges accept this because determinism and latency matter more than raw parallelism, and they scale out by running separate engines per symbol or per group of symbols. Each engine is independent, so different stocks match in parallel across cores while any single stock stays serial and fair.

Durability without losing speed

The engine holds the book in memory for speed, but a crash cannot lose trades. The standard answer is an append-only log. Every incoming order is written to a durable sequenced log before it is matched. If the engine dies, you replay the log to rebuild the exact book state. Because matching is deterministic, replay reproduces the identical set of fills. This event-sourced design gives you both microsecond matching and full recoverability.

How the real systems do it

The LMAX exchange popularized this approach and open-sourced the Disruptor, a lock-free ring buffer that feeds a single-threaded business logic core millions of messages per second. Nasdaq and other major venues run per-symbol engines with append-only journaling for recovery. The pattern is remarkably consistent: an in-memory order book, price-time priority, one thread per matching unit, and a durable log underneath it all.

I wrote the full breakdown, with diagrams and the data model, here: https://www.systemdesign.academy/interview/design-stock-exchange

How a Recommendation System Actually Works: Candidate Generation and Ranking

RONI DAS — Sat, 11 Jul 2026 11:28:29 +0000

Most explanations of recommendation systems start with a model and stop there. Collaborative filtering, matrix factorization, maybe a neural network, and a hand wave at the end about how it all gets served. That framing is why so many engineers can describe embeddings fluently and still have no idea what happens between a user opening an app and twenty items appearing on their screen a tenth of a second later.

The interesting engineering is not the model. It is the fact that you have ten million candidate items, a hard latency budget of roughly a hundred and twenty milliseconds, and a model good enough to rank well that is far too expensive to run more than a few hundred times per request. Everything about the architecture, from the two-tower split to the vector index to the feature store, exists to resolve that single contradiction. Once you see it that way, the design stops being a pile of machine learning components and becomes a fairly ordinary distributed systems problem with an unusual constraint.

A recommendation system is not a model with some infrastructure around it, it is a funnel where each stage buys the next stage the right to be expensive.

This piece works the problem the way a design review should. We start with the numbers, because the latency budget and the scoring rate settle the architecture before any model choice does. Then we build the funnel: candidate generation from several independent retrievers, approximate nearest neighbour search over the embedding space, feature retrieval, ranking with a multi-task model, and slate-level reranking. Along the way we deal with the three subproblems that actually separate a working system from a demo: training-serving skew, biased labels, and the feedback loop a system creates by learning from its own output. Every number below is an industry-typical figure chosen to drive the sizing, not a measured metric from any specific company.

Start with the budget, not with the model

A recommendation request arrives when a user opens a feed. The whole response has to be back inside a budget that, for a feed or a homepage, is usually somewhere between one hundred and two hundred milliseconds of server time. Take a hundred and twenty as the working number and split it into the stages that will need it.

Ranking gets the largest slice of a hundred and twenty milliseconds, and the sizing math on the right explains why the vector index fits in memory while the ranker cannot see the whole catalog.

Now put the scale next to it. A catalog of ten million items, two hundred million monthly active users, four hundred million feed requests per day. Four hundred million requests a day averages about forty six hundred per second, and peak traffic runs roughly double that, so call it nine thousand requests per second. If the ranking model scores three hundred items per request, the ranker alone is doing about two point seven million model evaluations per second at peak. That is the number that kills the naive design. If you tried to rank the full catalog, you would need ninety billion evaluations per second, which is not a tuning problem, it is a different universe.

The storage math points the other way and is unexpectedly friendly. Ten million item embeddings at a hundred and twenty eight float dimensions is ten million times a hundred and twenty eight times four bytes, about five point one gigabytes. The entire searchable representation of your catalog fits in the memory of one ordinary machine, with room for the index structure on top. That asymmetry, an enormous scoring cost against a tiny representation cost, is exactly what the architecture exploits. Cheap comparisons against a compact representation cut the field down, and only then does the expensive model get involved.

A funnel that trades breadth for precision

The resolution is a cascade. Each stage takes the output of the previous one, applies a more expensive method to fewer items, and hands a shorter list forward. Cost per item goes up at every step and the count comes down faster, so the total work stays flat while the quality of the decision keeps improving.

Ten million candidates become twenty delivered items across six stages, each one allowed to be more expensive than the last because it sees fewer items.

Concretely, candidate generation reduces ten million to roughly sixteen hundred using methods cheap enough to behave like a lookup. Filtering removes items the user has already seen, items blocked in their region, items that fail eligibility or policy rules, and near-duplicates, taking the list to around nine hundred. That step is bookkeeping, not modelling, and it is where a surprising number of production bugs live. A pre-ranking stage, a small model over cheap features, narrows to about three hundred when candidate volume is high enough to justify it. At three hundred items you can usually skip pre-ranking entirely, and many systems do. The full ranking model then scores those three hundred with hundreds of features each and produces an ordering. Finally a reranking stage applies constraints that only make sense over the list as a whole and delivers twenty items.

The arithmetic that makes this work is worth stating plainly. Divide each stage's time budget by the number of items it sees and you get the cost it is allowed to spend per item. Ranking gets about forty five milliseconds for three hundred items, so it can spend roughly a hundred and fifty microseconds each, which is enough for a real forward pass over hundreds of features. A pre-ranking stage carved out of that same budget gets maybe five milliseconds for nine hundred items, so about five microseconds each, enough for a small model on a handful of cheap features and nothing more.

Run the same division on retrieval and it breaks. Twenty milliseconds spread over ten million items is two nanoseconds each, and no method scores anything in two nanoseconds. That is the real reason retrieval cannot be a scoring stage at all. It has to be a search that never looks at most of the catalog, which is exactly what an approximate index does: it examines tens of thousands of vectors and ignores the rest. Each stage exists precisely because the next one cannot afford its input, and retrieval exists because nothing can afford the full catalog.

Three clocks, one system

The other structural decision is temporal. The pieces of a recommendation system run on wildly different clocks, and the cleanest way to organize the whole thing is by clock speed rather than by feature area.

The online band has a latency budget, the nearline band keeps things fresh in seconds, and the offline band produces the artifacts both of the others consume.

The online band is everything inside the request. Routing, candidate generation, feature fetch, ranking, reranking. It has a deadline, so it contains no training, no aggregation, and no scans. Every component here either reads a precomputed value or runs a bounded forward pass.

The nearline band runs in seconds to minutes. It consumes the event stream of impressions, clicks, dwell times, and hides, and it turns them into things the online band can read cheaply: rolling counters written into the online feature store, refreshed embeddings for new and updated items, and periodic rebuilds of the vector index. Nothing in this band blocks a request, but everything in it determines how stale the online band's inputs are.

The offline band runs in hours to days. It joins events into training data, trains the retrieval and ranking models, evaluates them on held-out periods, and publishes versioned artifacts to a registry. Every arrow between these bands points from slow to fast. The offline band produces artifacts, the nearline band keeps them fresh, and the online band only ever reads. That direction is what guarantees no request ever waits on a batch job, and it is a discipline worth enforcing explicitly rather than assuming.

Candidate generation is many small recall systems

Candidate generation is where most people expect a single clever model and instead find a committee. The reason is that relevance has several independent causes, and no single retriever captures all of them.

Six independent retrievers run in parallel, each optimizing a different notion of relevance, and a blender unions them under per-source quotas.

A learned embedding retriever finds items semantically close to the user's current interest vector. An item-to-item collaborative signal finds things that co-occur with what the user just engaged with, which catches short-term intent that a slow-moving user embedding misses entirely. A social or graph source returns items from accounts the user follows, which is a relevance signal no content model can infer. A freshness source returns recent items with popularity decay, which is the only way anything published in the last hour ever gets seen. A geographic source returns items popular in the user's locale. And an exploration source deliberately samples items with low exposure.

Each of these is allowed to have poor precision on its own. That is the whole point of putting a ranker downstream. What matters at this stage is recall, meaning the probability that an item the user would have loved appears somewhere in the union. An item that no retriever surfaces cannot be recovered later, no matter how good the ranking model is. This is the single most common silent failure in recommendation systems, and it is invisible in every downstream metric because you can only measure performance on candidates you actually produced.

The sources run in parallel, so the latency cost is the slowest source rather than the sum. A blender unions the lists, drops duplicates, and enforces a per-source quota so that one prolific generator cannot crowd out the others. Those quotas are a product lever, not a model output, and they deserve to be tuned deliberately and reviewed like any other policy.

The matrix is almost entirely holes

Underneath the learned retrievers sits a structural fact about the data. Represent the world as a matrix with users as rows and items as columns, and a cell filled in whenever a user interacted with an item. In any real system that matrix is more than ninety nine point nine percent empty.

Users interact with a vanishing fraction of the catalog, so the modelling problem is filling in the unknown cells rather than memorizing the observed ones.

That sparsity is the defining property of the problem, and it rules out entire families of approach. You cannot memorize the observed cells, because they cover almost none of the space you need to score. Whatever method you choose has to generalize across both users and items.

The classical answer is factorization. Approximate the huge sparse matrix as the product of two much smaller dense matrices, one holding a vector per user and one holding a vector per item, chosen so their dot products reproduce the observed interactions. Scoring a user against an item becomes a dot product between two dense vectors of, say, a hundred and twenty eight dimensions. That reduction is what makes everything downstream possible, because a dot product is cheap enough to run at retrieval scale and, more importantly, is a geometric operation that specialized indexes can accelerate.

Pure factorization has a hole in it, and the hole has a name. A brand new item has no interactions, so it has no learned vector, so it cannot be scored at all. The same is true of a brand new user. Cold start is not an edge case in a system where the catalog turns over constantly, it is the daily condition of a large fraction of your inventory, and the fix is to learn vectors from content features rather than from identity alone. Which leads directly to the model that dominates modern retrieval.

Two towers that meet only at a dot product

The two-tower model takes the factorization idea and makes it learnable from features. One network, the user tower, maps user features and request context to a vector. A second network, the item tower, maps item features to a vector in the same space. The two are trained jointly, but they never share weights and they never see each other's inputs. The only place they touch is the dot product at the end.

The user and item networks are deliberately kept apart so that every item vector can be computed offline and only the user vector has to be computed per request.

That separation looks like an arbitrary restriction and is in fact the entire design. Because the item tower depends only on item features, you can run it offline over the whole catalog and store the results. Because the user tower depends only on user features, you run it once per request. Retrieval then reduces to finding the item vectors with the largest dot product against one query vector, which is a geometric search problem with well understood solutions.

Compare that to a model that concatenates user and item features at the input and passes them through a joint network. That model is strictly more expressive, because it can learn arbitrary interactions between the two sides. It is also completely unusable for retrieval, because scoring requires one forward pass per candidate, and you have ten million candidates. The expressive model is exactly right for ranking, where you only have three hundred items. The constrained model is exactly right for retrieval. The same trade appears in almost every large system: you give up expressiveness in exchange for the ability to precompute one side of the comparison.

Training deserves a note. The standard approach uses sampled softmax with in-batch negatives, meaning the other items in the same training batch act as negative examples. This is cheap and effective, but it introduces a bias, because popular items appear in batches more often and therefore get pushed down more often as negatives. The correction is to subtract the log of each item's sampling probability from its logit, which is a small change with a large effect on how much the model over-favours the head of the catalog.

Approximate search is a recall dial

With ten million item vectors stored, retrieval means finding the top few hundred by dot product against the query vector. Exact search means ten million dot products, which at a hundred and twenty eight dimensions is more than a billion multiply-add operations per request. That is too slow, so every production system uses approximate nearest neighbour search, and it is worth understanding what "approximate" costs you.

Both index families expose the same underlying knob, and turning it toward speed quietly removes candidates you will never know you lost.

There are two dominant families. HNSW builds a layered proximity graph where the top layers are sparse and the bottom layer contains every vector. A search enters at the top, greedily hops toward the query, and descends layer by layer, refining as the graph gets denser. It delivers high recall at very low latency, typically above ninety five percent recall in single-digit milliseconds, but the graph links cost memory on top of the vectors, typically adding somewhere between a third and a half again on top of the raw vector data depending on how many neighbours each node keeps. Its search-time parameter, usually called ef_search, controls how many candidates the greedy search keeps alive.

The IVF family partitions the vector space into cells around learned centroids and stores each vector in its nearest cell. A query finds the nearest few centroids and scans only those cells. The parameter, nprobe, is the number of cells to scan. Combined with product quantization, which compresses each vector into a handful of bytes by quantizing subvectors against learned codebooks, IVF makes billion-scale indexes affordable at the cost of some precision in the distance computation.

The important point is not which family you pick. It is that both expose the same underlying trade and that trade is invisible from downstream metrics. An index running at seventy percent recall against exact search is silently discarding almost a third of the candidates it should have returned, and no click-through metric will ever explain that to you, because the missing items were never shown. Measure index recall directly, against exact search, on a held-out set of real query vectors, and treat it as a monitored quantity rather than a launch-day check.

One feature definition, two very different pipelines

Once candidates exist, ranking needs features, and feature infrastructure is where recommendation systems most often fail in ways that are difficult to diagnose. The problem is that the same feature has to be computed twice, in two completely different environments, and the two computations have to agree exactly.

The offline path reconstructs features as of the label's timestamp while the online path serves them in milliseconds, and any disagreement between the two poisons training.

In training, features are computed in bulk over historical events. In serving, the same features are read one row at a time under a few milliseconds of budget. A feature store exists to make those two paths produce identical values from a single definition, generating both the batch job and the streaming job from the same source.

The failure this prevents is called training-serving skew, and the most damaging version of it is temporal. Suppose a feature is "number of clicks this item received in the last twenty four hours." At serving time, that obviously means the twenty four hours before now. If the backfill job computes it by grouping the day's events, it will include clicks that happened after the impression it is attached to. The model then trains on a feature that partially encodes the answer. Offline metrics look excellent, because the leakage is genuinely predictive in the training set. Online performance is flat or worse, because at serving time that information does not exist yet.

The fix is a point-in-time join. Every event carries a timestamp, every feature value carries the time it became valid, and the training join takes the value that was current at the moment the label was produced, never a later one. This is not an optimization, it is a correctness requirement, and it is the reason offline feature stores keep full history with event timestamps rather than just current values. When someone reports an offline improvement that does not reproduce online, skew is the first thing to check and usually the thing you find.

Ranking is one model with several opinions

The ranking model is where the expensive computation goes. It sees a few hundred candidates and a wide feature vector for each, and it produces the scores that determine the ordering.

Hundreds of features feed a shared trunk with several prediction heads, and the product decides afterwards what each predicted behaviour is worth.

The features cluster into groups with different characteristics. User features describe the person: account age, historical engagement rates, topic affinities built from long-run behaviour. Item features describe the content: topic, creator, age in hours, media type, length. Context features describe the moment: device, hour of day, position in the session, network quality. Counter features are rolling aggregates over one hour, twenty four hour, and seven day windows, and they are the ones with the tightest freshness requirements. Embedding features carry the user and item vectors plus their dot product, which lets the ranker reuse retrieval's learned representation.

Then there are cross features, which combine an attribute of the user with an attribute of the item: this user's historical engagement with this creator, or with this topic, or with this media format. These consistently deliver the largest gains, because they hand the model the interaction it would otherwise have to discover from scratch across a sparse space.

Modern rankers are multi-task. Rather than predicting one number, a shared trunk feeds several heads, each predicting a different behaviour: probability of click, probability of a long dwell, probability of an explicit positive like a save or a like, probability of a negative like a hide or a report. Sharing the trunk lets rare labels benefit from the representation learned on abundant ones. It also separates two things that should be separate. The model predicts what will happen. The product decides what those outcomes are worth, by blending the head outputs with weights and subtracting a penalty for predicted negatives.

That separation has a practical requirement: the head outputs must be calibrated, meaning a predicted probability of zero point three actually corresponds to a thirty percent rate in reality. Neural network outputs are frequently not calibrated, particularly after training on downsampled negatives, so a calibration step such as isotonic regression sits between the model and the blend. Without it, the weighted combination is arithmetic on incomparable quantities, and tuning the weights becomes guesswork.

Your labels are the model's real architecture

The hardest problem in a recommendation system is not learning from the data. It is that the data is generated by the system itself, and therefore encodes the system's past decisions as much as it encodes user preference.

Implicit feedback is noisy, position distorts what a click means, and easy negatives teach the model nothing it did not already know.

Start with implicit feedback. You do not have ratings, you have clicks, watch times, saves, shares, follows, and hides. A click is weak evidence of interest and gets weaker the more clickbait exists in your catalog. A non-click is barely evidence at all, because the overwhelming majority of non-clicks are items the user never saw, never scrolled to, or scrolled past without registering. Treating every non-click as a negative teaches the model that most of the catalog is bad, which is not what the data says. The practical response is to weight labels by strength, treating a long dwell or a completion as much stronger evidence than a click, and to be careful about which non-events you are willing to call negatives.

Then there is position bias. The top slot of a feed collects clicks nearly regardless of what occupies it. If you train on raw clicks, the model learns the ranking policy that produced the logs rather than the underlying preference. The standard correction is inverse propensity weighting: estimate the probability that a position would be examined at all, and divide each observed click by that probability, so a click at position twelve, where the user rarely looks, counts for far more than a click at position one. Estimating those propensities is itself work, and the cleanest source is a small slice of randomized traffic where the ordering is shuffled, which gives an unbiased view of examination probability at the cost of some engagement on that slice.

Negatives round out the set of problems. A random item drawn from ten million is trivially irrelevant, so the model learns to separate it from a positive with no effort and no useful gradient. The informative negatives are the hard ones: items the current model ranked highly that the user nonetheless skipped. Mining those is what pushes a retrieval model past mediocre. And as noted earlier, in-batch negatives systematically over-represent popular items, so a sampling correction is not optional if you want the model to learn relevance rather than popularity.

Offline metrics filter ideas, they do not decide them

Given all of the above, evaluation is harder than it looks, and the gap between offline and online results is a permanent feature of this domain rather than a sign that something is broken.

Offline lift and online lift agree often enough to be useful and disagree often enough that only the A/B test decides.

Offline evaluation splits into two questions. For retrieval, the metric is recall at k: of the items the user actually engaged with, what fraction did candidate generation surface at all. This is measurable and meaningful, and it is the right way to compare retrieval strategies. For ranking, the metric is usually NDCG or a similar position-weighted measure computed over the logged candidate set. That metric has a built-in ceiling, because it is scored against a candidate list the previous system chose. It can tell you whether you order those candidates better. It cannot tell you anything about items the old system never showed.

This is why offline and online results diverge in specific, predictable ways. A change that improves ordering within the old system's candidate set usually improves both. A change that retrieves a genuinely different set of items often looks flat or negative offline while winning online, because the offline metric penalizes disagreeing with the logs. And a change that improves offline while hurting online is usually overfitting to the logged distribution, which is what deeper retrieval into a poorly calibrated ranker tends to do.

The operational conclusion is to use offline evaluation as a filter and never as a verdict. It is cheap, it runs in minutes, and it correctly kills most bad ideas. The A/B test is the measurement that ships or blocks a model, and it needs to run long enough to get past novelty effects, which typically means at least two weeks and a guardrail set that includes more than the primary metric.

Six stores, each chosen by its access pattern

There is no single database in a recommendation system, and trying to build one is a reliable way to end up with something bad at everything.

Every component here has a query shape narrow enough that the storage engine picks itself.

The vector index is the only component queried by similarity rather than by key, which is why it is a specialized engine rather than a table. It holds the item embeddings, lives in memory, and is sharded by item so that each shard searches its own partition and the results are merged.

The online feature store sits directly on the critical path. It is read by primary key, one row per user and one row per item, with a tail latency requirement of a few milliseconds. That is a memory-resident key-value workload with no scans and no joins, which points at an in-memory store rather than anything general purpose.

The offline feature store answers a completely different question: what was this feature's value at this moment in history. That requires full retention with event timestamps and efficient scans over date ranges, which is a columnar lake partitioned by day.

The event log holds raw impressions, clicks, dwell events, and hides. It is append-only and needs replay, so it is a partitioned durable log rather than a database. The model registry holds versioned model and index artifacts read at deploy time, which is object storage plus a metadata table. And the seen-item filter, which stores the last couple of thousand item ids per user, is deliberately approximate: a Bloom filter in the key-value store, where a false positive simply hides an item you might have shown anyway and the memory saving is worth it many times over.

Reranking scores the list, not the item

The ranker evaluates each item independently. That is what makes it cheap enough to run, and it is also what makes it blind to a whole class of problems. It cannot notice that the top five items are all from the same creator, or that three of them are the same news story from different outlets, or that the user has seen this topic in every session this week.

Constraints that span positions, from creator caps to a reserved exploration slot, can only be applied by a stage that sees the whole list.

Reranking is the stage that operates on the slate. The order of its steps matters. Calibration and blending come first, so that everything downstream is working with one comparable value per item. Near-duplicate collapsing comes next, because if duplicates survive into the diversity step they each consume a slot and the diversity constraint appears satisfied while the user sees the same thing three times. Then come the diversity constraints, expressed over windows of positions: at most two items per creator in any ten, at most three per topic. Then policy and eligibility rules that demote borderline content or enforce advertiser and legal requirements. Then, finally, exploration.

That exploration slot is worth defending explicitly, because it will be the first thing someone proposes cutting. Reserving a position for an item with low exposure costs measurable short-term engagement, and it is the only reliable source of unbiased training data you have. Without it, everything the model learns tomorrow is conditioned on what the model chose to show today, which brings us to the failure mode.

The failure mode is a loop, not a crash

Recommendation systems rarely fail loudly. They fail by slowly optimizing themselves into a smaller and smaller world while every dashboard stays green.

Nothing in this loop is a bug. Each step is correct behaviour, and the composition is a system that recommends a shrinking catalog.

The mechanism is straightforward. The model has more observed data about popular items, so it predicts their engagement with more confidence and ranks them higher. Higher rank means more impressions, which means more engagement data, which means those items are even better represented in tomorrow's training set. Meanwhile items that were never shown generate no data at all and never get the chance to prove themselves. Each retraining amplifies the tilt. Nothing errors, nothing alerts, and engagement metrics typically stay flat or improve throughout, because the model is genuinely getting better at the objective you gave it.

The damage shows up in quantities nobody is watching. Catalog coverage, the fraction of items that received any impressions in a period, falls quarter over quarter. The exposure distribution across creators concentrates. New items take longer to find their audience, so creators publish less, so the catalog itself degrades. By the time this surfaces as a business problem it has been running for a year.

The defenses are all about measurement and injection. Measure catalog coverage and the Gini coefficient of the exposure distribution as first-class metrics, reviewed alongside engagement. Guarantee exploration traffic so that some fraction of impressions is not chosen by the model. Correct for popularity in the training objective rather than only in the serving policy. And separate a new item's cold-start period, where it is scored on content features and given deliberate exposure, from its steady state, where it is scored on its own observed performance. None of these are free. Each one costs engagement in the short run and buys a healthier system in the long run, which is exactly the kind of trade that needs an explicit owner or it never gets made.

The trade-offs, and what transfers

Every choice above bought something and cost something. The multi-stage funnel bought a tractable latency budget and cost you the ability to recover a candidate that early stages dropped. The two-tower split bought precomputable item vectors and cost you the expressiveness of early feature interaction. Approximate search bought single-digit millisecond retrieval and cost you a recall percentage you have to monitor deliberately. The multi-task ranker bought shared representation across sparse labels and cost you a calibration step and a set of blend weights that someone has to own. The feature store bought consistency between training and serving and cost you a second pipeline for every feature.

Four decisions from this design, stage by cost, precompute one side, distrust your logs, and watch what your objective ignores, transfer to every ranked retrieval system.

Four lessons transfer beyond recommendations, and you will meet all of them again in search, in advertising, and in any system that ranks a large corpus under a deadline.

First, stage by cost rather than by concern. When a large candidate set meets a fixed budget, split the work into stages where each one is cheap enough for the size of its input and precise enough for the size of its output. The stages are not arbitrary layers, they are the direct consequence of the arithmetic.

Second, constrain the model so that one side of every comparison can be precomputed. The two-tower dot product is a specific instance of a general move, and it is the move that makes real-time retrieval over millions of items possible at all. Whenever you find yourself needing to compare one thing against many, look for the factorization that lets you prepare the many in advance.

Third, assume your data is biased, because it is. Logs record what the previous system chose to show, at what position, to which users. Position bias, exposure bias, and selection bias are properties of the data rather than defects you can train away, and correcting them is usually worth more than the modelling improvement you were going to ship instead.

Fourth, watch what your objective ignores. Any system trained on data it produced itself needs guardrail metrics that look at coverage, diversity, and freshness, because the primary metric is measured on a distribution the system controls. A metric that only ever goes up is not evidence that nothing is wrong.

Get those four right and the specific model you use matters far less than it seems to from the outside. Get them wrong and no architecture will save you.

Read the free lessons: https://systemdesign.academy

Designing an API Gateway: Routing, Auth, and the Filter Chain

RONI DAS — Sat, 11 Jul 2026 11:23:13 +0000

Once you split a monolith into services, a new problem appears: the client now needs to know about twenty endpoints, each with its own auth, its own retry logic, and its own rate limits. An API gateway is the single front door that hides all of that.

A visual from the full interactive lesson on systemdesign.academy.

The problem it solves

Every service needs the same cross-cutting concerns. Authentication, TLS termination, rate limiting, request logging, and CORS all have to happen somewhere. If each service implements them, you get twenty slightly different, slightly buggy copies. The gateway pulls these concerns out of the services and does them once, in one place, before the request is routed anywhere.

It also gives clients a stable contract. The mobile app talks to api.example.com and does not care that the orders service moved, split in two, or got rewritten in a different language.

Routing

The core job is mapping an incoming request to a backend. A request to /orders/123 needs to reach the orders service; /users/me reaches the user service. Routes are usually matched by path prefix, sometimes by host header or HTTP method. The gateway holds a route table, and modern gateways refresh this table dynamically from service discovery so you are not redeploying the gateway every time a backend changes address.

A useful pattern here is the backend for frontend. A web client and a mobile client often want differently shaped responses. Instead of one bloated API, you run a thin gateway per client type, each aggregating the same services but tailoring the payload.

The filter chain

The cleanest way to structure a gateway is as an ordered chain of filters that every request passes through. Each filter does one thing and either passes the request along or rejects it. A typical inbound chain looks like this:

TLS termination and request parsing
Authentication: validate the token, reject if invalid
Authorization: does this identity have access to this route
Rate limiting: has this client exceeded its quota
Request transformation: add headers, strip internal fields
Routing: pick the backend and forward

On the way back, a smaller outbound chain runs: response transformation, adding CORS headers, and logging the status and latency. The value of this design is that each concern is isolated and ordered. Auth must run before rate limiting so you are counting against the right identity, and both must run before you spend resources forwarding to a backend.

Auth: where to draw the line

The common approach is token-based. The client sends a JWT or an opaque token, and the gateway validates it. A JWT can be verified locally using a public key, which is fast because there is no network call, but revoking one before it expires is hard. An opaque token requires a lookup against an auth service or cache on every request, which is slower but gives you instant revocation. Many teams use short-lived JWTs to get the speed while capping the blast radius of a leaked token.

Crucially, the gateway authenticates but usually does not do fine-grained authorization. It confirms who you are and passes a trusted identity header to the backend, which decides whether you can touch this specific resource. Coarse checks at the edge, fine checks at the service.

The trade-off you cannot ignore

A gateway is a single point of failure and a potential bottleneck. Every request in your system flows through it, so it must be horizontally scaled and stateless, with any shared state (like rate limit counters) pushed to Redis. Keep the filters cheap. Heavy work such as response aggregation across many services adds latency to every call, so measure it. The gateway should add single-digit milliseconds, not tens.

How the real systems do it

Netflix built Zuul around exactly this filter-chain model, with pre, route, and post filters. Kong runs as a set of Lua plugins on top of Nginx, where each plugin is a filter you enable per route. Envoy, the data plane behind many service meshes, expresses the same idea as a chain of HTTP filters configured dynamically. The shape is always the same: an ordered pipeline, cross-cutting concerns pulled forward, and a stable front door for clients.

I wrote the full breakdown, with diagrams and the data model, here: https://www.systemdesign.academy/interview/design-api-gateway