How to discuss latency in a system design interview

Originally published on four-leaf.ai.

Most system design candidates say "we'll add a cache to keep latency low" and move on. Google's Site Reliability Engineering book gives the example of a hypothetical web service averaging 100 ms at 1,000 requests per second, where 1% of requests might easily take 5 seconds. In our view, that gap between the average and the slow tail is the thing a strong interviewer is listening for when latency comes up.

This guide covers the handful of numbers worth carrying into the room, why you quote a percentile instead of a mean, how fan-out turns rare slowness into common slowness, and how to build a latency budget out loud so the panel can follow your reasoning.

Why do interviewers care how you talk about latency?

Latency is where a system design answer stops being a diagram and becomes an engineering argument. Any candidate can draw a load balancer, three services, and a database. Far fewer can say how long a request spends in each box, which box dominates, and what happens to users when that box has a bad minute.

A good interviewer can use latency to check three things at once. First, whether you know the physical cost of the operations you are proposing, because a design that crosses an ocean twice per request has already lost. Second, whether you think in distributions, since a single average hides the users who are having the worst experience. Third, whether your stated targets and your architecture agree with each other. A candidate who promises a fast read path and then adds two synchronous network hops to it has contradicted themselves.

The Four-Leaf breakdown of system design interview questions for senior engineers treats autocomplete as a latency-budget round in its own right. The skill in this guide applies to every design question, from a URL shortener to a chat service, because every one of them has a user waiting on the other end.

Which latency numbers should you know before a system design interview?

Orders of magnitude are enough. The point is to know which operations are effectively free, which cost a millisecond, and which cost a noticeable fraction of a second.

The table most engineers learn from is Peter Norvig's list of approximate timings for a typical PC, on a page his site dates 2001 to 2014, which Jeff Dean later popularized. Norvig lists a fetch from main memory at 100 nanoseconds and reading 1MB sequentially from disk at 20,000,000 nanoseconds, which is 20 milliseconds (a spinning disk, judging by the table's 8,000,000-nanosecond seek). A packet sent from the US to Europe and back costs about 150 milliseconds. A widely shared GitHub version of the table, credited to Dean and originally to Norvig, puts a round trip within the same datacenter at 500,000 nanoseconds, half a millisecond.

Treat these as relative sizes rather than benchmarks. Hardware has moved since they were written, and knowing that a cross-ocean round trip is hundreds of times slower than a datacenter hop is more useful in the room than any specific figure. In practice, four tiers carry most interviews.

1. Memory and cache. Effectively free at the scale of one request.
2. A network hop inside one datacenter. Fractions of a millisecond each, which add up when a request makes dozens of them.
3. Disk and database work. Tens of milliseconds on spinning disks and less on SSDs, and usually the tier where most of your budget goes.
4. Cross-region and cross-ocean trips. Tens to hundreds of milliseconds, which is why read paths stay inside one region.

Why should you quote p99 instead of the average?

Quote a percentile because averages hide exactly the requests your users complain about. Google's Site Reliability Engineering book shows a service where a typical request is served in about 50 ms while 5% of requests are 20 times slower, and notes that alerting on the average alone would show no change while the tail moved significantly.

The same book explains what each percentile is for. A high-order percentile, such as the 99th or 99.9th, shows you a plausible worst-case value, while the 50th percentile emphasizes the typical case. It also reports that people typically prefer a slightly slower system to one with high variance in response time, which is the user-facing reason to care about the tail at all.

So when you state a target, state it the way the SRE book writes its examples. "99% of Get RPC calls will complete in less than 100 ms" is a sentence an interviewer can hold you to and test. Say which percentile, which operation, and which threshold, then design to it. If the interviewer pushes on a number, you can move it. The structure of the claim is the part worth getting right.

How does fan-out turn a rare slow call into a common one?

Fan-out is the reason tail latency matters more as a system grows. When one user request calls many backends in parallel and waits for all of them, the user's response is only as fast as the slowest backend.

The arithmetic is short enough to do on a whiteboard. Suppose each backend is slow on one call in a hundred, independently of the others, and a request fans out to 100 of them. The chance that all 100 come back fast is 0.99 multiplied by itself 100 times, which is about 0.37. Roughly two of every three user requests hit at least one slow call, even though every individual backend looks healthy. Google's Site Reliability Engineering book puts it in one line, warning that the 99th percentile of one backend can easily become the median response of your frontend.

Jeff Dean and Luiz André Barroso built a paper on this problem, "The Tail at Scale" in Communications of the ACM. Its abstract argues that large online services need to create a predictably responsive whole out of less predictable parts. In an interview, show that you see the multiplication, then offer a mitigation that fits the design, such as reducing how many backends a request must wait for, setting a timeout and returning partial results, or sending a backup request to a second replica when the first is slow.

How do you build a latency budget out loud?

Start from the user and work inward. The abstract of "The Tail at Scale", citing a 1991 study by Card and colleagues, notes that systems responding within 100 milliseconds feel more fluid and natural to users than those that take longer. That threshold is end to end, as the user perceives it, so the server's share has to be smaller. Propose a server budget inside it, ask the interviewer whether it fits the product, and then spend it.

A worked example for a feed read might sound like this. "The user should see the feed within about 100 ms end to end. Their network will eat part of that, so I'll target the 99th percentile at 50 ms inside our datacenter for loading the feed. Inside the datacenter, each hop is well under a millisecond, so a handful of hops is cheap. Dozens would start to add up. The real cost is storage. A precomputed feed in a cache is a single fast read. Building the feed on request means querying every followed account, which is a fan-out, and its tail will blow the budget. So I'll precompute on write and accept the write-side cost."

That paragraph does three things worth showing. It names a percentile and a number. It assigns the budget to components using rough tiers rather than invented precision. And it lets the budget make an architectural decision, which is the moment the latency discussion pays for itself. Keep read and write budgets separate while you do this, because they usually have different users and different tolerances.

How should you measure the latency you promised?

A target with no measurement plan is half an answer, and a panel can reasonably ask how you would know the system is meeting it. Google's Site Reliability Engineering book names the four golden signals of monitoring as latency, traffic, errors, and saturation, and says that if you can only measure four metrics of a user-facing system, those are the four.

Two details from the same chapter are worth saying out loud. Measure the latency of failed requests separately from successful ones, because a fast error can make your numbers look better than the user experience is, and a slow error is even worse than a fast error. And collect request counts bucketed by latency rather than raw averages, so you can render a histogram and see the tail. The SRE book suggests distributing histogram boundaries approximately exponentially, by factors of roughly 3.

Mentioning both takes a few sentences. It tells the panel your latency target is something you would alert on.

What is overrated

Memorizing nanosecond figures is overrated. In our experience, an interviewer is unlikely to fail you for saying a memory read is "about a hundred nanoseconds" rather than a precise value, and reciting a whole table wastes minutes you need for the design. Know the tiers and the ratios between them.

Optimizing the average is overrated too. Candidates often propose caching to "make it faster" without saying which percentile improves. A cache can lower the median while doing nothing for cold reads in the tail, and it is worth saying so unprompted.

Finally, a latency number nobody asked you to defend is overrated. Stating "p99 under 50 ms" and then never connecting it to a design decision is decoration. The number earns credit only when it changes what you build.

A five-step playbook for the latency part of the round

1. Propose a target early. Name the operation, the percentile, and the threshold, then ask the interviewer to confirm or adjust it.
2. Split read and write paths. Give each its own budget, because they rarely share the same tolerance.
3. Walk the request through the tiers. Count network hops, disk or database reads, and any cross-region trips, using orders of magnitude.
4. Find the fan-out. Wherever one request waits on many calls, do the tail arithmetic out loud and offer one mitigation.
5. Say how you would measure it. Latency for successes and errors separately, bucketed into a histogram, with an alert on the percentile you promised.

Practicing this sequence silently is not the same as saying it under time pressure. Four-Leaf's AI mock interviews include a system design round for software roles that you can answer by voice, and the feedback checks whether your design handles the scale you stated and whether you called out trade-offs explicitly. For the wider preparation plan, the technical interview preparation guide covers the coding, take-home and behavioral rounds around it.

Where this is heading

System design rounds reward candidates who can connect a number to a decision, and latency is the most direct way to show that. The numbers themselves are easy to look up. Reasoning in percentiles, spotting fan-out, and spending a budget in front of someone who is taking notes are harder, and they only get easier with practice out loud.

The candidates who do well in this part of the round treat every box on the whiteboard as a cost the user pays. Say what it costs, and the design argues for itself.

About Four-Leaf

Four-Leaf is the all-in-one AI job search assistant covering every stage from application to signed offer: voice mock interviews across 25 roles, resume tailoring, cover letters, salary negotiation, email drafting, AI job search, and LinkedIn optimization. 3-day free trial, no credit card. Pricing: $20/mo or $5 / 5 Day Pass.

• Website: https://four-leaf.ai
• Pricing: https://four-leaf.ai/pricing
• Contact: [e-mail]