Design a web crawler: the system design interview walkthrough

Design a web crawler: the system design interview walkthrough

The request to design a web crawler sounds like a weekend script — download a page, pull out its links, follow them, repeat. For a hundred pages that is exactly what it is. But the interview asks you to do it for billions of pages without hammering any single site, without re-crawling the same URL forever, and without getting trapped in an endlessly generated maze of links — and that is a scale and politeness problem, not a downloader. This is one of the most common system design interview prompts because the one-line brief hides a graph traversal whose frontier can explode, and a crawler architecture that lives or dies on how it schedules work.

The round turns on two decisions. First, the shape of the queue of URLs waiting to be visited — the URL frontier — which must do two jobs at once: visit important pages first, and never overload one host. Second, how you avoid redoing work: a page you have already fetched should not re-enter the queue, a page whose content you have already seen should not be stored twice, and a site that generates infinite links should not swallow your crawler whole. This walkthrough follows the path a strong candidate draws on the whiteboard — requirements, estimate, API and data model, high-level design, then the deep dives — and at each step it names the follow-up the interviewer uses to tell a memorised answer from an engineer who understands the tradeoff.

How the web crawler round is actually run

A web crawler round runs about 45 minutes on a shared canvas — you draw the boxes and arrows while you talk, and the interviewer interrupts. It is a favourite because the one-line problem opens onto deep water fast: the moment you write "follow the links," the question becomes in what order, how fast, and how do you not do it twice — and every answer after that (prioritisation, politeness, deduplication, traps) compounds. Get the frontier and the seen-set clean and you have shown the whole loop in one problem.

The interviewer rarely lets "download pages and follow their links" stand. "Your crawler is pointed at one site and it's sending that site a thousand requests a second — what's wrong, and how do you fix it?" That question — politeness under a design that is otherwise just a fast loop — is the one a copied answer can never rehearse, and the one this round is really testing.

The rounds here are modelled on how the loop runs, not scraped from any candidate's recall. What matters for prep is the shape: a blank canvas, a clock, and a follow-up after every decision you make.

The five-step framework for any HLD round

Every high-level design answer fits the same spine. Say it out loud at the start so the interviewer knows you have a plan, then walk it:

1. Clarify requirements — functional and non-functional, and what you are explicitly not building.
2. Estimate — pages, storage and fetch rate, to justify every later choice.
3. High-level design — the boxes and arrows: seed URLs, frontier, fetchers, parser, filter, store.
4. Deep dive — the two or three decisions that actually decide this system.
5. Follow-ups — bottlenecks, failure modes, and what you would change at ten times the scale.

What the debrief measures

The score is not "right or wrong design." A good HLD debrief reads on four dimensions, and every move below maps to one: problem solving (did you structure the frontier so priority and politeness both come out of the queue), communication (did you say why one queue per host in a sentence), technical execution (persist the seen-set as a bloom filter, respect robots.txt and crawl-delay, cache DNS), and time management (did you spend your 45 minutes on the frontier and dedup, not on the HTML parser). Reading a walkthrough teaches the design; it does not teach you to deliver it on the clock while someone pushes back.

Step 1 — Requirements: what a web crawler must do

Spend the first few minutes here. Jumping to boxes before you have scoped the problem is the fastest way to design the wrong system.

Type	Requirement
Functional	Start from a set of seed URLs; download each page; extract the links on it; add the new, unseen URLs back into the crawl; store the page content; re-crawl periodically to pick up changes.
Functional (optional)	Respect freshness and priority (crawl important or fast-changing pages more often); parse multiple content types (HTML, PDF, images) — call these out, then scope most of them for later.
Non-functional	Massive scale (billions of pages); politeness (never overload a single host); avoid duplicates and crawler traps; be extensible (new content types, new parsers) and fault-tolerant (a dead worker loses nothing).

The sentence that earns points: "The downloader is the easy part — the system is really a scheduler that keeps billions of URLs in order, stays polite to every host, and never does the same work twice." Say that and you have framed the entire design before drawing a box.

Step 2 — Capacity estimation (back of the envelope)

Estimation here is about proving one thing: the hard axis is scheduling the frontier and not redoing work, not raw bandwidth.

Quantity	Assumption	Result
Pages to crawl	1B pages/month	~400 pages/sec sustained
Average page size	~500KB	~500TB of raw content/month → petabytes over time
Seen-URL set	~10B URLs tracked	too big for a plain set in memory → bloom filter
Fetch rate per host	politeness-capped	one host trickles; throughput comes from crawling many hosts at once

Two conclusions fall out, and you should say both out loud. First, the frontier is the system — throughput does not come from hitting one site hard (politeness forbids it) but from fanning out across millions of hosts in parallel, which is a scheduling and queueing problem. Second, the content store is large but simple — write-heavy blob storage (object store for the raw page, a metadata DB for status and timestamps) — so you spend your minutes on the frontier and the seen-set, not on the store. You have now pointed the interview at the real deep dive before anyone asked.

Step 3 — API and data model

A crawler has no public API — it is a pipeline — so the "interface" is the internal contract between stages, plus the tables that hold crawl state.

Stage call	Operation
frontier.add(url, priority)	enqueue a discovered URL, if it passes the seen-set filter
frontier.next() → url	hand the next due, polite URL to a free fetcher
fetch(url) → content	download the page (after a robots.txt + crawl-delay check)
parse(content) → links[]	extract outbound links and the stored document

The data model is thin — most state is the frontier and the seen-set:

Entity	Notes
url_record	url (normalised), host, priority, status, last_crawled_at, next_due_at.
seen_urls	a hash set / bloom filter of URL fingerprints, to answer "crawled this already?" in O(1).
content_store	content_hash (for duplicate detection), raw document in object storage, metadata row.
domain	host → robots_rules, crawl_delay, last_fetched_at, for per-host politeness.

Step 4 — High-level design: the crawl loop

Now the boxes. The whole system is one loop that drains the frontier and refills it with newly discovered links. Around that sit the pieces that make it polite, deduplicated and fault-tolerant:

• Seed URLs → URL frontier: the frontier is the queue of URLs waiting to be crawled; it decides what next by priority and how fast per host by politeness.
• Fetchers (DNS resolver → HTML downloader): a pool of workers that resolve the host (via a cached DNS resolver) and download the page content.
• Content parser: reads the downloaded document, extracts the text/content to store and the outbound links.
• Link extractor → URL filter / seen-set: normalises each discovered link, drops disallowed and already-seen URLs through the seen-set, and feeds the survivors back into the frontier.
• Content store + metadata: raw pages in object storage, a content hash for duplicate detection, and a metadata DB for status and timestamps.

Say explicitly that this is a cycle, not a pipeline — the output of the parser (new links) loops back through the filter into the frontier, which is exactly why the seen-set and trap detection matter: without them the loop never terminates. Naming the loop unprompted is a signal the interviewer is listening for.

Deep dive: the URL frontier

This is the first decision that actually decides the round. The frontier is nominally a BFS queue — you visit the seed pages, then everything they link to, then the next ring out — but a single FIFO queue fails on two counts at once: it has no notion of which pages matter more, and it will happily dequeue a hundred URLs from the same host in a row and hammer it. The senior answer is a frontier with two layers.

Layer	What it does	Why it exists
Front queues (priority)	several queues, one per priority level; a prioritiser assigns each URL a level (by PageRank, update frequency, host importance).	So important and fast-changing pages are crawled first — not strict FIFO.
Back queues (politeness)	many queues, one per host; a URL only ever sits in its host's queue, and each back queue is released on a timer.	So a single host is fetched at a controlled rate no matter how many of its URLs you have discovered.

The flow between them: a URL is scored and dropped into a front queue by priority; a router moves it into the back queue for its host; each back queue has a per-host release time, and a worker is only handed a URL when its host's politeness timer says it is due. Priority and politeness are therefore both encoded in the frontier itself — you do not bolt a rate limiter on afterwards, you make the queue structure do the work.

"You've got a million URLs from the same host and ten from another — what order do you fetch them, and what stops you from flooding the first host?" Priority decides the front queue; one back queue per host plus a release timer decides the rate. Say both halves — most candidates name priority and forget politeness, or vice versa.

System Design
HLD · Search & discovery · 45 min
Design a web crawler out loud, with follow-ups

Practise on Zynter →

Deep dive: politeness, robots.txt and deduplication

This is the decision most candidates under-think. The online path is easy; the follow-ups are always "how do you not overload the site?" and "how do you not crawl the same thing forever?"

Politeness and robots.txt. Before a worker fetches any page, it checks the host's robots.txt — the file a site uses to tell crawlers which paths are off-limits (Disallow) and how long to wait between requests (Crawl-delay). You cache each host's rules in the domain table so you fetch robots.txt once, not per page. The per-host back queue from the frontier is what enforces the crawl delay: the queue is simply not released until the host's timer says it is due. For a distributed crawler, track the last-fetched time per host in a centralised store (e.g. Redis) so two workers cannot both decide a host is free at the same instant.

Deduplication, two levels. URL-level: every discovered link is normalised (lowercase host, strip fragments, sort query params) and checked against a seen-URL set. At ten billion URLs a plain hash set will not fit in memory, so you use a bloom filter — O(1), tiny, and its only error is a rare false positive (you skip a page you have not seen, which is acceptable). Content-level: hash the downloaded body (a content fingerprint / SimHash for near-duplicates) and skip pages whose content you have already stored, so mirror sites and reprinted articles do not bloat the store.

Crawler traps. Some sites generate infinite URLs — a calendar with a "next month" link forever, or session IDs baked into every path. Defend with a max crawl depth, a per-host page cap, URL normalisation to collapse equivalent links, and simple pattern detection for suspiciously deep or repetitive paths.

"The same URL keeps coming back into the queue — how do you stop re-crawling it?" The seen-URL set, backed by a bloom filter, rejects it at the filter before it ever re-enters the frontier; content fingerprinting catches the case where a different URL serves identical content. Name both the URL-level and content-level check.

System Design
HLD · Politeness & dedup · 45 min
Defend a polite, duplicate-free crawl out loud

Practise on Zynter →

Deep dive: distributed crawling, storage and the loop

Groups of workers change the shape of the problem. One machine fetching billions of pages is impossible; the crawl must spread across many machines, and the moment it does, three things need care.

The flow, end to end: a worker pulls a due URL from the frontier, resolves its host through a DNS resolver — which is often the real bottleneck, because a DNS lookup can take tens of milliseconds and you do millions of them, so you cache resolved hosts aggressively — then an HTML downloader fetches the page. The content parser extracts the document and its links; the document goes to the content store, the links go through the URL filter / seen-set, and the survivors go back to the frontier. That is the loop closing.

Concern	Naive	Scaled
Work distribution	one queue, one worker	shard the frontier by host across machines so each host's politeness state lives in one place and workers scale horizontally.
DNS	resolve on every fetch	a cached DNS resolver in front of the downloaders — the biggest single latency win.
Fault tolerance	a dead worker drops its URLs	a URL is only removed from the frontier once its page is stored; a crashed worker's in-flight URLs become due again and another worker picks them up.

Repeat the real scaling axis: throughput comes from crawling many hosts in parallel, bounded per host by politeness — so the frontier's sharding and the DNS cache, not the downloader's raw speed, are where this system lives or dies.

"A page links to itself through a million generated URLs — how do you not get stuck?" A max crawl depth and per-host page cap bound the trap, URL normalisation collapses the generated variants into one, and trap detection flags suspiciously repetitive paths — all before the links re-enter the frontier.

The follow-ups that decide the web crawler round

Scan the probes from each deep dive and they rhyme. Master these five and you are ready for a version of the question that is not drawn exactly like this one.

• "What order do you crawl, and what stops you flooding one host?" → Front queues by priority; one back queue per host with a release timer. (problem solving)
• "The same URL keeps coming back — how do you stop re-crawling it?" → A seen-URL set backed by a bloom filter rejects it at the filter; content hashing catches duplicate bodies behind different URLs. (technical execution)
• "A thousand requests a second to one site — what's wrong?" → Politeness: honour robots.txt and the crawl-delay, enforced by the per-host back queue and a centralised last-fetched time. (communication)
• "A page generates a million links to itself — how do you not get stuck?" → Depth and page caps, URL normalisation, trap detection before links re-enter the frontier. (problem solving)
• "What's the real bottleneck at scale?" → Not bandwidth — DNS resolution (cache it) and frontier scheduling across sharded hosts. (technical execution)

Each maps to a debrief dimension. Reading this list teaches you the answers; it does not teach you to deliver them while someone interrupts — that only comes from saying them out loud.

How to prepare: a one-week plan

You do not need fifty systems. You need this one cold and the framework automatic, so the next "design X" is just new nouns in the same five steps.

• Days 1–2: Learn the five-step framework until it is reflex. Do the crawler end to end on paper — requirements, estimate, data model, the seed → frontier → fetch → parse → filter → frontier loop.
• Days 3–4: Drill the frontier deep dive: BFS, front queues for priority, back queues for per-host politeness, and why both live in the queue structure. Say each tradeoff in one sentence.
• Days 5–6: Own politeness and dedup — robots.txt and crawl-delay, the bloom-filter seen-set, content fingerprinting, and trap defences. Answer each of the five follow-ups out loud.
• Day 7: A full timed 45-minute round, narrating the whole way, with someone (or something) interrupting. This is the rep that counts.

This post is the deep walkthrough behind one row of the broader Top 25 system design interview questions — once the crawler is automatic, the same spine carries the rest of the list.

Frequently asked questions

How do I design a web crawler step by step?

Follow the five-step HLD framework. Clarify requirements (seed URLs, download, extract links, store content, re-crawl, at billions of pages, politely); estimate to show the hard axis is scheduling the frontier and not redoing work, not bandwidth; define the internal stage contract plus the url, seen-set, content and domain tables; draw the crawl loop (seed URLs → frontier → fetchers → parser → link extractor → URL filter → back to the frontier, with a content store); then deep-dive the decisions that matter — the URL frontier, politeness and deduplication, and distributed crawling. Finish on the follow-ups about traps and the real bottleneck.

What is the URL frontier in a web crawler?

The URL frontier is the queue of URLs waiting to be crawled, and it does two jobs at once. Front queues hold URLs by priority level so important or fast-changing pages are visited first rather than in strict FIFO order. Back queues hold one queue per host with a per-host release timer, so a single site is fetched at a controlled rate no matter how many of its links you have discovered. Prioritisation and politeness are both encoded in the frontier structure itself, which is why it is the heart of the design.

How does a web crawler stay polite and respect robots.txt?

Before fetching any page, the crawler checks the host's robots.txt file, which lists disallowed paths and a crawl-delay, and it caches those rules per host so robots.txt is fetched once rather than per page. The per-host back queue in the frontier enforces the delay by not releasing the next URL until the host's timer is due. In a distributed crawler you track the last-fetched time per host in a centralised store such as Redis, so two workers cannot both decide the same host is free at the same instant.

How does a web crawler avoid crawling duplicate pages?

It deduplicates at two levels. URL-level: every discovered link is normalised and checked against a seen-URL set; at ten billion URLs that set is a bloom filter, which answers "seen this?" in O(1) with tiny memory and only a rare, acceptable false positive. Content-level: the downloaded body is hashed (a content fingerprint, or SimHash for near-duplicates) and pages whose content has already been stored are skipped, so mirror sites and reprinted articles do not bloat the store.

What is a crawler trap and how do you handle it?

A crawler trap is a site that generates effectively infinite URLs — a calendar with a perpetual "next month" link, or session IDs baked into every path — which can swallow a crawler that blindly follows links. You defend with a maximum crawl depth and a per-host page cap, URL normalisation that collapses equivalent links into one, and simple pattern detection that flags suspiciously deep or repetitive paths. All of these run at the URL filter, before the generated links are ever allowed back into the frontier.

How do I practise a system design interview out loud?

Reading a walkthrough is not the same as defending a design while someone interrupts. On Zynter, the HLD track runs live voice rounds modelled on how real system design loops run — you talk through a shared design on the canvas, and a voice interviewer pushes with the follow-ups above one at a time, then gives you a debrief on problem solving, communication, technical execution, and time management. The score is a practice signal, not a hiring decision.

Practise it on Zynter

You can read every hop. The round is won on the frontier and the politeness case.

Pick the Web Crawler round, draw the crawl loop on the canvas, and defend it out loud to a voice interviewer that keeps asking "what stops you flooding one host?" and "how do you not crawl the same URL twice?" — then read the debrief. Zynter.ai is on-demand mock interviews modelled on real company loops, with 150+ rounds across 20 companies including the HLD system design track (Sep 2026).

Practise the Web Crawler round →
Browse all interview tracks