DEV Community

Cover image for Design a RAG system: the system design interview walkthrough
Gowtham Potureddi
Gowtham Potureddi

Posted on

Design a RAG system: the system design interview walkthrough

Design a RAG system: the system design interview walkthrough

The request to design a RAG system sounds like a glue exercise — embed your documents, search them, paste the top matches into a prompt, let the model answer. For a demo over a dozen PDFs that is exactly what it is. But the interview asks you to answer questions grounded in a corpus of millions of documents, cite the source for every claim, stay correct as those documents change under you, and never let the model confidently invent a fact the corpus never contained — and that is a retrieval-and-grounding problem, not a prompt. This is a fast-rising retrieval augmented generation interview prompt because the one-line brief hides a pipeline whose answer quality is decided long before the LLM is ever called.

The round turns on two decisions. First, how you turn a living document corpus into something searchable — how you split documents into chunks, embed them, and keep that index fresh as the source changes — because retrieval can only return what indexing captured. Second, how you make the model answer from the retrieved text rather than from its own weights: construct the prompt so the context grounds the answer, attach citations, and decide what happens when nothing relevant comes back. This walkthrough follows the path a strong candidate draws on the whiteboard — requirements, estimate, API and data model, high-level design, then the deep dives — and at each step it names the follow-up the interviewer uses to tell a memorised answer from an engineer who understands the tradeoff.

Title card for a system design interview walkthrough on designing a RAG system, showing a document corpus flowing through chunking and an embedding model into a vector database, and a user query being embedded, retrieved and passed to an LLM for a grounded answer, on a dark citrus-and-lavender background.

How the RAG system round is actually run

A RAG round runs about 45 minutes on a shared canvas — you draw the boxes and arrows while you talk, and the interviewer interrupts. It is a favourite in the llm system design interview because the one-line problem opens onto deep water fast: the moment you write "retrieve the relevant docs and let the model answer," the question becomes relevant how, retrieved how fast, and grounded how — and every answer after that (chunking, embeddings, index choice, reranking, freshness, hallucination control) compounds. Get retrieval quality and grounding clean and you have shown the whole pipeline in one problem.

The interviewer rarely lets "embed the docs and prompt the model" stand. "Your RAG bot answered a policy question confidently — but the policy document never said that, the model made it up. Where in the system did that go wrong?" That question — grounding and hallucination under a design that is otherwise just embed-search-prompt — is the one a copied answer can never rehearse, and the one this round is really testing.

The rounds here are modelled on how the loop runs, not scraped from any candidate's recall. What matters for prep is the shape: a blank canvas, a clock, and a follow-up after every decision you make.

The five-step framework for any HLD round

Every high-level design answer fits the same spine. Say it out loud at the start so the interviewer knows you have a plan, then walk it:

  1. Clarify requirements — functional and non-functional, and what you are explicitly not building.
  2. Estimate — corpus size, chunks, embedding dimensions and query rate, to justify every later choice.
  3. High-level design — the boxes and arrows: the ingestion path that builds the index and the query path that answers from it.
  4. Deep dive — the two or three decisions that actually decide this system.
  5. Follow-ups — bottlenecks, failure modes, and what you would change at ten times the scale.

What the debrief measures

The score is not "right or wrong design." A good HLD debrief reads on four dimensions, and every move below maps to one: problem solving (did you separate the ingestion path from the query path and put retrieval quality before the LLM), communication (did you say why you chose that chunk size, or hybrid search over pure vector, in a sentence), technical execution (an ANN index with a stated recall/latency tradeoff, reranking, citations, incremental re-embedding on document change), and time management (did you spend your 45 minutes on chunking, retrieval and grounding, not on prompt wording). Reading a walkthrough teaches the design; it does not teach you to deliver it on the clock while someone pushes back.

Step 1 — Requirements: what a RAG system must do

Spend the first few minutes here. Jumping to boxes before you have scoped the problem is the fastest way to design the wrong system.

Type Requirement
Functional Ingest a corpus of documents; answer a user's natural-language question using the content of that corpus; return the answer with citations to the source passages; keep answers fresh as documents are added, edited or removed; say "I don't know" when the corpus has no good match.
Functional (optional) Per-user or per-tenant access control on which documents can be retrieved; multi-turn conversation with retrieval each turn; multiple document types (PDF, HTML, tables) — call these out, then scope most of them for later.
Non-functional Large scale (millions of documents, many concurrent queries); low retrieval latency; high answer relevance and groundedness; freshness (an edited document is reflected quickly); and cost control on embeddings and LLM calls.

The sentence that earns points: "The LLM is the easy part — the system is really a retrieval engine that finds the right passages and a grounding step that forces the model to answer from them, not from memory." Say that and you have framed the entire design before drawing a box.

Step 2 — Capacity estimation (back of the envelope)

Estimation here is about proving one thing: the hard axis is retrieval quality and index freshness, and the recurring cost is embeddings plus LLM calls, not raw storage.

Quantity Assumption Result
Corpus 10M documents, ~10 chunks each ~100M chunks to embed and index
Embeddings 100M chunks × ~1.5KB per vector (≈768 dims) ~150GB vector index → needs an ANN index, not brute force
Query rate ~500 queries/sec peak each query = 1 embed + 1 ANN search + optional rerank + 1 LLM call
Cost driver re-embed changed docs + one LLM call per answer embedding and generation cost, not disk, dominates the bill

Two conclusions fall out, and you should say both out loud. First, retrieval is the system — a 150GB vector space cannot be scanned per query, so an approximate nearest-neighbour (ANN) index and the recall/latency tradeoff it forces are where the design lives. Second, the cost is in the model calls, not the bytes — embeddings on ingest (and on every document change) plus one LLM generation per answer dominate the bill, so batching embeddings and caching retrieval are real levers. You have now pointed the interview at the real deep dive before anyone asked.

Step 3 — API and data model

RAG has a thin public surface — one query endpoint — and an internal ingestion pipeline. The "interface" is both: the external API plus the internal contract between ingestion stages.

Call Operation
POST /query {question} → {answer, citations[]} embed the question, retrieve top-k chunks, build the prompt, call the LLM, return a grounded answer with sources
ingest(doc) chunk the document, embed each chunk, upsert vectors + metadata into the index
retrieve(query_vec, k, filters) → chunks[] ANN search over the vector index, optionally filtered by metadata
rerank(query, chunks) → chunks[] reorder candidates by a cross-encoder for precision before the prompt

The data model is thin — most state is the chunk store and the vector index:

Entity Notes
document doc_id, source_uri, version, updated_at, access/tenant tags — the unit of freshness.
chunk chunk_id, doc_id, text, position, metadata (title, section, tags) for filtering and citation.
vector_index chunk_id → embedding vector, stored in an ANN index (HNSW / IVF) for top-k similarity search.
cache optional query → retrieved-chunks / answer cache, to skip re-embedding and re-retrieving hot questions.

Step 4 — High-level design: the two paths

Now the boxes. A RAG system is two pipelines that meet at the vector index: an ingestion path that builds the searchable index offline, and a query path that answers from it online. Draw them as two flows, not one:

  • Ingestion path — documents → chunking → embedding model → vector DB + metadata: each source document is split into chunks, each chunk is embedded into a vector, and the vectors plus their metadata are upserted into the vector index. This runs on ingest and again whenever a document changes.
  • Query path — user query → embed → vector search (ANN) → optional rerank → prompt builder → LLM → grounded answer with citations: the question is embedded with the same model, the top-k nearest chunks are retrieved, optionally reranked, stitched into a prompt as context, and sent to the LLM, which answers and cites the chunks it used.

Say explicitly that the two paths are decoupled — ingestion is write-heavy and can run offline or in batch; the query path is latency-sensitive and read-only — and that they share exactly one thing, the embedding model, which must be identical on both sides or the query vector and the chunk vectors live in different spaces and retrieval returns noise. Naming that shared constraint unprompted is a signal the interviewer is listening for.

A RAG system architecture diagram showing an ingestion path where documents are chunked, passed through an embedding model and upserted into a vector database with metadata, and a query path where a user query is embedded, searched against the vector index with ANN, reranked, built into a prompt with retrieved context, and answered by an LLM with citations.

Deep dive: chunking and embeddings

This is the first decision that actually decides the round, and it happens entirely on the ingestion side — before any query is ever asked. Retrieval can only surface what indexing captured, so the quality of your chunks and embeddings sets the ceiling on every answer.

Choice What it trades The senior answer
Chunk size big chunks carry more context but dilute the embedding and waste prompt budget; small chunks are precise but can lose the surrounding meaning. a few hundred tokens per chunk, tuned to the content, not a round guess.
Chunk overlap no overlap splits a sentence across a boundary and loses it; too much overlap bloats the index and retrieves near-duplicates. a small overlap so a fact that straddles a boundary survives in at least one chunk.
Embedding model a bigger model is more accurate but slower and costlier to run on every chunk and every query. one model, used on both ingest and query, chosen for the domain and the recall you need.
Metadata raw chunks can't be filtered or attributed. store title, section, source and tags per chunk for metadata-filtered retrieval and citations.

The flow: split each document into overlapping chunks, embed each chunk with the embedding model, and upsert {vector, chunk_id, metadata} into the index. The non-obvious point to say out loud is re-embedding on change — when a document is edited, its old chunks are stale and must be re-chunked and re-embedded, and the superseded vectors removed, or the index will happily retrieve a passage the source no longer contains. Freshness is an ingestion concern, and it starts here.

"The answer cited the wrong document — where in the pipeline did that go wrong?" Walk it backward: the citation points at whatever chunk retrieval returned, so the fault is a chunk that embedded close to the query but came from the wrong source — a chunk too large and mixing topics, a stale vector never removed after an edit, or retrieval with no metadata filter. Name the chunking and freshness causes, not just "the model was wrong."

System Design
HLD · System design · practise out loud
Design a RAG system out loud, with follow-ups

Practise on Zynter →

Deep dive: vector search and retrieval quality

This is the decision most candidates under-think. The happy path is "search the vector DB," but the follow-ups are always "search it how at this scale?" and "what if the top result is close in vector space but wrong?"

Approximate nearest neighbour (ANN). A 150GB vector space cannot be scanned exactly per query, so you use an ANN index — HNSW (a navigable small-world graph, fast and high-recall, memory-hungry) or IVF (inverted file / clustering, smaller memory, recall tuned by how many clusters you probe). The tradeoff to state plainly is recall versus latency: searching more of the graph or more clusters finds better matches but costs milliseconds, and you tune that knob to the latency budget.

Hybrid search. Pure vector search matches meaning but misses exact terms — an error code, a product SKU, a person's name — because those are rare tokens the embedding smooths over. Combine vector similarity with keyword/BM25 search and fuse the two rankings, so both semantic and exact-match queries land.

Reranking and top-k. Retrieve a generous candidate set (say top-50) cheaply with ANN, then rerank with a cross-encoder that scores each candidate against the full query for precision, and pass only the top few into the prompt. Tuning top-k matters both ways: too few and the answer's supporting passage never reaches the model; too many and you blow the context window and bury the relevant chunk in noise.

"Retrieval returns nothing relevant — what should the system do?" Set a similarity threshold: if the best match is below it, the honest behaviour is to not stuff weak context into the prompt and instead return "I don't know" or ask to narrow the question — because forcing the model to answer from irrelevant context is exactly how hallucinations happen. Say the threshold and the graceful fallback, not just "return the top-k anyway."

A RAG retrieval-quality diagram showing a query vector searched against an ANN index with HNSW and IVF options and a recall-versus-latency tradeoff, a hybrid path fusing vector similarity with keyword BM25 search, a cross-encoder reranker reordering the candidate set, and a similarity threshold gate that returns "I don't know" when no match is good enough.

System Design
HLD · System design · practise out loud
Defend retrieval quality and grounding out loud

Practise on Zynter →

Deep dive: grounding, freshness and hallucination control

This is where a RAG round is won or lost, because it is the part a naive design skips entirely. Retrieval found the passages; now the system has to make the model answer from them and stay correct over time.

Grounding and citations. Build the prompt so the retrieved chunks are the context and the instruction is explicit — answer using only the provided passages, and cite the chunk each claim came from. Because every chunk carries its doc_id and source metadata, the citation is just the metadata of the chunks the model used, which also gives the user a way to verify the answer and gives you a way to catch a wrong one.

Freshness. Keep the index in step with the source with incremental updates, not a nightly full rebuild: on a document create/edit/delete, re-chunk and re-embed just that document and upsert or remove its vectors, so an edit is reflected in minutes. State the staleness window honestly — the gap between "document changed" and "its new vectors are live" is your freshness SLA, and it is bounded by how fast ingestion runs, not by the query path.

Handling "no good match" and evaluating quality. The system must have a path for no answer: below the similarity threshold, return "I don't know" rather than hallucinate. And because answer quality is not a unit test, you evaluate it — measure retrieval (did the right chunk come back) and generation (is the answer grounded in and faithful to the retrieved context, not invented) on a labelled set, so a regression in chunking or model choice is caught before users see it.

"A document is updated — how fast does the system reflect the change?" As fast as incremental ingestion runs: on the update event, re-chunk and re-embed that one document, upsert the new vectors and delete the superseded ones, and the next query retrieves the new passages. Give the number as a freshness window bounded by ingestion lag — and note that a full nightly re-embed would make that window a whole day, which is why incremental updates matter.

A RAG grounding and freshness diagram showing retrieved chunks assembled into a prompt with an answer-only-from-context instruction, an LLM producing an answer with citations drawn from chunk metadata, a similarity threshold that routes a weak match to an "I don't know" response, and an incremental ingestion loop that re-embeds a changed document and upserts or removes its vectors.

The follow-ups that decide the RAG system round

Scan the probes from each deep dive and they rhyme. Master these five and you are ready for a version of the question that is not drawn exactly like this one.

  • "The model made up a fact — where did grounding fail?" → Weak retrieval below threshold stuffed into the prompt, or a prompt that didn't force answer-from-context; fix with a threshold gate and an explicit grounding instruction. (problem solving)
  • "The answer cited the wrong document — where in the pipeline?" → A bad chunk retrieved close in vector space: too-large mixed-topic chunks, a stale vector left after an edit, or retrieval without a metadata filter. (technical execution)
  • "Retrieval returns nothing relevant — what then?" → A similarity threshold and a graceful "I don't know" or clarify, never forcing an answer from irrelevant context. (communication)
  • "A document is updated — how fast is it reflected?" → Incremental re-chunk and re-embed of just that document; the freshness window is bounded by ingestion lag, not the query path. (problem solving)
  • "What's the real bottleneck and cost at scale?" → ANN recall-vs-latency and the embedding/LLM bill — not disk; batch embeddings, cache hot queries, and tune top-k. (technical execution)

Each maps to a debrief dimension. Reading this list teaches you the answers; it does not teach you to deliver them while someone interrupts — that only comes from saying them out loud.

How to prepare: a one-week plan

You do not need fifty systems. You need this one cold and the framework automatic, so the next "design X" is just new nouns in the same five steps.

  • Days 1–2: Learn the five-step framework until it is reflex. Do RAG end to end on paper — requirements, estimate, data model, and the two paths: ingestion (chunk → embed → index) and query (embed → retrieve → rerank → prompt → LLM).
  • Days 3–4: Drill the retrieval deep dive: chunk size and overlap, the shared embedding model, ANN (HNSW vs IVF) and recall-vs-latency, hybrid search, reranking, and top-k. Say each tradeoff in one sentence.
  • Days 5–6: Own grounding and freshness — prompt construction with citations, the "no good match" threshold, incremental re-embedding on document change, and how you'd evaluate answer quality. Answer each of the five follow-ups out loud.
  • Day 7: A full timed 45-minute round, narrating the whole way, with someone (or something) interrupting. This is the rep that counts.

This post is the deep walkthrough behind one row of the broader LLM and system design interview set — once RAG is automatic, the same spine carries the rest of the list.

Frequently asked questions

How do I design a RAG system step by step?

Follow the five-step HLD framework. Clarify requirements (answer questions grounded in a document corpus, cite sources, stay fresh as documents change, say "I don't know" on no match, at millions of documents); estimate to show the hard axis is retrieval quality and the cost is embeddings plus LLM calls, not disk; define the query API and the internal ingestion contract plus the document, chunk and vector-index model; draw the two paths (ingestion: chunk → embed → vector DB; query: embed → ANN retrieve → rerank → prompt → LLM → cited answer); then deep-dive the decisions that matter — chunking and embeddings, vector search and retrieval quality, and grounding and freshness. Finish on the follow-ups about hallucination and the real bottleneck.

What is the difference between the ingestion path and the query path in RAG?

The ingestion path builds the searchable index offline: it splits each document into chunks, embeds each chunk, and upserts the vectors and their metadata into the vector database, re-running whenever a document changes. The query path answers online: it embeds the user's question with the same embedding model, retrieves the top-k nearest chunks with ANN search, optionally reranks them, stitches them into a prompt as context, and calls the LLM for a grounded answer with citations. The two are decoupled — ingestion is write-heavy and batchable, the query path is latency-sensitive and read-only — and they share exactly one thing, the embedding model, which must be identical on both sides.

How does a RAG system avoid hallucination?

It grounds the answer in retrieved context and refuses when there isn't any. The prompt is built so the retrieved chunks are the only source and the model is instructed to answer from them and cite each claim, which makes a wrong answer visible against its citation. Crucially, retrieval applies a similarity threshold: if no chunk is a good enough match, the system returns "I don't know" or asks to narrow the question rather than forcing the model to answer from irrelevant context, because that forcing is exactly how confident fabrication happens.

How do you keep a RAG system's answers fresh as documents change?

Use incremental index updates instead of full rebuilds. On a document create, edit or delete, re-chunk and re-embed just that one document, upsert its new vectors and remove the superseded ones, so the change is reflected in minutes rather than on a nightly cycle. The freshness window — the gap between a document changing and its new vectors going live — is your SLA, and it is bounded by how fast ingestion runs, not by the query path, which is why incremental ingestion is the thing to name.

What is chunking in RAG and how do you choose a chunk size?

Chunking splits each document into smaller passages so they can be embedded and retrieved independently. The size is a tradeoff: large chunks carry more surrounding context but dilute the embedding and burn prompt budget, while small chunks are precise but can lose the meaning around a fact. A common answer is a few hundred tokens per chunk with a small overlap so a fact that straddles a boundary still survives intact in at least one chunk — and the size is tuned to the content and the recall you need, not guessed. Each chunk also carries metadata (title, section, source) for filtering and citations.

How do I practise a system design interview out loud?

Reading a walkthrough is not the same as defending a design while someone interrupts. On Zynter, the HLD track runs live voice rounds modelled on how real system design loops run — you talk through a shared design on the canvas, and a voice interviewer pushes with the follow-ups above one at a time, then gives you a debrief on problem solving, communication, technical execution, and time management. The score is a practice signal, not a hiring decision.

Practise it on Zynter

You can read every hop. The round is won on retrieval quality and the grounding case.

Take a system design round on the HLD track, draw the ingestion and query paths on the canvas, and defend them out loud to a voice interviewer that keeps asking "what if retrieval returns nothing relevant?" and "where did the model's made-up fact come from?" — then read the debrief. Zynter.ai is on-demand mock interviews modelled on real company loops, with 150+ rounds across 20 companies including the HLD system design track (Sep 2026).

Practise a system design round →
Browse all interview tracks

Top comments (0)