DEV Community

Cover image for Design ChatGPT: the LLM system design interview walkthrough
Gowtham Potureddi
Gowtham Potureddi

Posted on

Design ChatGPT: the LLM system design interview walkthrough

Design ChatGPT: the LLM system design interview walkthrough

The request to design ChatGPT sounds like wiring a text box to a model: take the user's message, call the model, show the reply. For one user on a laptop that is exactly what it is. But the interview asks you to run a multi-turn chat product for millions of concurrent users, streaming the answer word by word, remembering the conversation, staying safe and within budget — all on top of GPUs that cost orders of magnitude more than the web servers in front of them. That turns a simple API call into a serving and cost problem, and that is why design chatgpt system design has become one of the sharpest prompts in the modern system design interview: the one-line brief hides an autoregressive model that emits tokens one at a time, and an LLM serving architecture that lives or dies on how efficiently it keeps those GPUs busy.

The round turns on two decisions that have nothing to do with the model's intelligence. First, how you get the answer to the user — you cannot wait for a long reply to finish and return it in one response, so the whole stack has to stream tokens as they are generated. Second, how you afford it — a GPU serving one chat at a time is mostly idle, so throughput comes from batching many users into a single pass and reusing a KV cache across the turns of a conversation. This is a llm system design interview at heart, distinct from the retrieval-heavy design a RAG system round: here the hard axis is serving a conversational model at scale, not fetching documents. This walkthrough follows the path a strong candidate draws on the whiteboard — requirements, estimate, API and data model, high-level design, then the deep dives — and at each step it names the follow-up the interviewer uses to tell a memorised answer from an engineer who understands the tradeoff.

Title card for a system design interview walkthrough on designing ChatGPT, showing a user prompt flowing into an inference orchestrator and a GPU model-serving cluster that streams tokens back, on a dark citrus-and-lavender background.

How the LLM serving round is actually run

An LLM serving round runs about 45 minutes on a shared canvas — you draw the boxes and arrows while you talk, and the interviewer interrupts. There is no single "ChatGPT round" to memorise; this is the general high-level design (HLD) track pointed at a conversational AI product. It is a favourite because the one-line problem opens onto deep water fast: the moment you write "call the model and return the answer," the question becomes how do you return it, how do you pay for the GPU, and how do you remember the conversation — and every answer after that (streaming, batching, context management) compounds. Get the serving path and the GPU economics clean and you have shown the whole system in one problem.

The interviewer rarely lets "send the message to the model and return the reply" stand. "Your demo streams fine for one user. Now a million people are chatting at once on a fixed GPU fleet — what breaks first, and what do you do about it?" That question — efficiency under a design that is otherwise just an API call — is the one a copied answer can never rehearse, and the one this round is really testing.

The rounds here are modelled on how the loop runs, not scraped from any candidate's recall. What matters for prep is the shape: a blank canvas, a clock, and a follow-up after every decision you make.

The five-step framework for any HLD round

Every high-level design answer fits the same spine. Say it out loud at the start so the interviewer knows you have a plan, then walk it:

  1. Clarify requirements — functional and non-functional, and what you are explicitly not building.
  2. Estimate — concurrent sessions, tokens per second and GPU memory, to justify every later choice.
  3. High-level design — the boxes and arrows: client, gateway, orchestrator, GPU serving cluster, conversation store.
  4. Deep dive — the two or three decisions that actually decide this system.
  5. Follow-ups — bottlenecks, failure modes, and what you would change at ten times the scale.

What the debrief measures

The score is not "right or wrong design." A good HLD debrief reads on four dimensions, and every move below maps to one: problem solving (did you make streaming and batching fall out of the architecture rather than bolting them on), communication (did you say why you stream tokens in one sentence), technical execution (continuous batching, a reused KV cache, context truncation, pre/post moderation, rate limits), and time management (did you spend your 45 minutes on the serving path and GPU economics, not on the chat UI). Reading a walkthrough teaches the design; it does not teach you to deliver it on the clock while someone pushes back.

Step 1 — Requirements: what a ChatGPT-style product must do

Spend the first few minutes here. Jumping to boxes before you have scoped the problem is the fastest way to design the wrong system.

Type Requirement
Functional Accept a user message in a multi-turn chat; generate a model reply; stream the reply back token by token; maintain conversation context across turns; apply moderation to inputs and outputs; enforce per-user rate limits and quotas.
Functional (optional) Multiple models / sizes, regenerate and edit, system prompts, file or image inputs — name them, then scope most of them for later.
Non-functional Scale to millions of concurrent chats; low time-to-first-token; high GPU utilisation (the dominant cost); cost control and safety; be fault-tolerant (a dead GPU worker loses no conversation).

The sentence that earns points: "The model call is the easy part — the system is really an efficient serving layer that streams tokens, packs expensive GPUs full of concurrent users, and remembers each conversation within a fixed context window." Say that and you have framed the entire design before drawing a box.

Step 2 — Capacity estimation (back of the envelope)

Estimation here is about proving one thing: the hard axis is GPU throughput and memory, not the number of HTTP requests.

Quantity Assumption Result
Output rate per user ~20–40 tokens/sec (near reading speed) time-to-first-token is the latency users feel, not total request time
Model weights a ~70B model in fp16 ≈ 140GB needs several GPUs per model replica; this is not a CPU workload
KV cache grows with context length × concurrent sessions the real memory pressure — it caps how many chats share one GPU
Cost GPUs, not CPUs, dominate the bill throughput per GPU (via batching) is the lever, not request count

Two conclusions fall out, and you should say both out loud. First, the GPU is the system — a GPU generating tokens for one user at a time is mostly idle waiting on memory, so throughput comes not from more replicas but from batching many users into each forward pass and reusing the KV cache across a conversation's turns. Second, the stateful chat data is large but simple — append-only messages in a conversation store, with only a recent window ever loaded into a prompt — so you spend your minutes on the serving loop, not on the message table. You have now pointed the interview at the real deep dive before anyone asked.

Step 3 — API and data model

The public surface is a streaming chat API; the internal contract is how a request flows from the gateway to a GPU worker and back, plus the tables that hold conversation state.

Stage call Operation
POST /v1/chat/completions (stream) open an SSE stream, send messages[], receive token deltas as they are generated
buildContext(session_id) → prompt assemble the system prompt plus recent turns that fit the context window
moderate(text) → allow / block pre-generation (user input) and post-generation (model output) safety filter
enqueue(prompt) → stream hand the prompt to the inference orchestrator's request queue for a GPU worker

The data model is thin — most of it is the conversation log plus per-session serving state:

Entity Notes
conversation session_id, user_id, model, system_prompt, created_at.
message conversation_id, role (user / assistant / system), content, token_count, created_at — append-only.
kv_cache GPU-resident key/value tensors for a session's active context; reused across its turns, evicted when the session idles.
usage / quota tokens in/out per user, rate-limit counters, and cost guardrails.

Step 4 — High-level design: the serving loop

Now the boxes. The whole system is a loop that turns a user message into a stream of tokens and keeps the GPUs full while it does. Around that sit the pieces that make it safe, metered and fault-tolerant:

  • Client → API gateway: terminates the connection, authenticates the user, and applies rate limits and quotas before any GPU time is spent.
  • Moderation (pre): the user message is screened before it ever reaches a model.
  • Context builder → inference orchestrator: assembles the prompt (system prompt + recent turns from the conversation store) and places it on a request queue for the serving cluster.
  • Model-serving cluster (GPU workers): a pool of GPU workers runs the model with continuous batching, holding a KV cache per session, and emits tokens one at a time.
  • Streaming back (SSE / WebSocket): each generated token is pushed to the client as it is produced; moderation (post) screens the output on the way out.
  • Conversation store: every user and assistant turn is appended, so the next turn can be rebuilt within the context window.

Say explicitly that this is a streaming loop, not a request/response call — the GPU produces tokens incrementally and the transport carries them out as they appear, which is exactly why SSE, the KV cache and batching all show up here rather than in a normal CRUD service. Naming the streaming shape unprompted is a signal the interviewer is listening for.

A ChatGPT serving architecture diagram showing a client and API gateway with rate limiting, a pre-moderation filter, a context builder reading the conversation store, an inference orchestrator with a request queue feeding a GPU model-serving cluster with continuous batching and a per-session KV cache, and tokens streamed back over SSE through a post-moderation filter.

Deep dive: token streaming and the serving path

This is the first decision that actually decides the round. A language model is autoregressive — it generates the answer one token at a time, and each new token is conditioned on every token before it, including the ones it just produced. A long reply therefore takes seconds to finish. If you wait for the whole thing and return it as one JSON response, the user stares at a spinner for the entire generation, and you have thrown away the one property that makes chat feel alive.

Approach What the user gets Why it loses / wins
One-shot JSON nothing until the full reply is done, then it all appears time-to-first-token = total generation time; feels frozen; a dropped connection loses everything.
Streaming (SSE) the first word in a fraction of a second, then a steady flow time-to-first-token is tiny and decoupled from length; the server flushes each token delta as it is generated.

The two latencies to name separately are time-to-first-token (TTFT) — how long until the first word appears, which is what users feel — and inter-token latency / throughput — how fast the rest streams, which reading speed hides. Server-Sent Events (SSE) is the usual transport: a long-lived HTTP response where the server pushes data: chunks as tokens appear; WebSockets do the same job when you also need client-to-server messages mid-stream. The orchestrator holds the open connection and forwards each token from the GPU worker the instant it is produced.

"Why can't you just generate the whole answer and return it in one response?" Because generation is autoregressive and sequential — a long reply takes seconds, so one-shot response means a seconds-long spinner and a total loss on any dropped connection. Streaming tokens over SSE puts the first word on screen in a fraction of a second and makes perceived latency independent of answer length. Name TTFT as the metric that improves.

System Design
HLD · System design · practise out loud
Design ChatGPT out loud, with follow-ups

Practise on Zynter →

Deep dive: GPU efficiency — batching and the KV cache

This is the decision most candidates under-think, and the one that separates "I've read a blog" from "I understand serving." GPUs are the entire cost of this system, and a GPU running a single conversation is mostly idle: loading the model's weights from memory for one user's next token wastes the hardware's parallelism. Two mechanisms fix that.

Continuous batching. Instead of generating tokens for one request at a time, the serving engine merges many users' next-token steps into a single GPU forward pass — the weights are read once and amortised across the whole batch. Naive static batching waits for a fixed group of requests and then runs them to completion together, so a short reply is held hostage by the longest one. Continuous (dynamic) batching fixes this: as soon as any sequence in the batch finishes, a waiting request takes its slot mid-flight, keeping the batch — and the GPU — full. Batching is the throughput lever: it is why one GPU can serve dozens of concurrent chats, not one.

The KV cache. Because generation is autoregressive, computing each new token needs the attention keys and values of all prior tokens. Recomputing them every step is quadratic and wasteful, so the worker caches the key/value tensors for the context and reuses them — each new token becomes a cheap incremental step, and the cache persists across the turns of a conversation so you do not re-encode the history every message. The catch is memory: the KV cache grows with context length × batch size, and it, not the model weights, is what caps how many sessions a GPU can hold at once. Managing that memory (paging, eviction of idle sessions) is the real constraint on concurrency.

Concern Naive Scaled
Batching one request per forward pass continuous batching — new requests join mid-flight as others finish, keeping the GPU saturated.
Attention state recompute keys/values every token a reused KV cache per session; incremental, and shared across a conversation's turns.
Memory limit ignore it until OOM KV cache sized by context × batch bounds concurrency; page it and evict idle sessions.

"Two users hit the same GPU at the same time — how do you serve both efficiently?" Continuous batching: merge their next-token steps into one forward pass so the model weights are loaded once and amortised across both, and let a third user slot in the instant either finishes. Each keeps its own KV cache for its context. The GPU is the cost, so throughput per GPU — via batching — is the whole game.

A GPU efficiency diagram showing several user requests merged into one continuous-batch forward pass on a single GPU, each request keeping its own KV cache, with a note that the KV cache grows with context length times batch size and bounds how many chats a GPU can hold.

System Design
HLD · System design · practise out loud
Defend GPU batching and the KV cache out loud

Practise on Zynter →

Deep dive: context, memory and safety at scale

Groups of users and long conversations change the shape of the problem. A single chat is easy; millions of multi-turn chats, each with a growing history and a fixed model window, is where three things need care.

Context-window limits. A model has a fixed maximum number of tokens it can attend to (its context window). A 100-message conversation blows past that, so the context builder cannot just concatenate everything — it must decide what to keep. The two levers: truncation (drop the oldest turns, keep the system prompt and the most recent messages) and summarisation (compress earlier turns into a running summary that rides along with the recent window). You always keep the system prompt and the latest user turn; what you sacrifice is the stale middle.

Conversation storage. Every turn is appended to the conversation store, but only the windowed context is ever loaded into a prompt — the full history is for display and for rebuilding context, not for stuffing into every forward pass. This keeps the expensive GPU input small while the cheap database holds the long tail.

Safety and metering. A moderation filter runs twice — on the user input before generation and on the model output before it streams out — to block or redact unsafe content. Rate limiting and quotas at the gateway cap tokens per user and tier, applying backpressure before GPU time is spent. And cost guardrails — a max-tokens cap per response, routing simple queries to a cheaper/smaller model, caching common completions — keep the GPU bill bounded. These are not afterthoughts; at this scale they are load-bearing.

"A conversation is 100 messages long and now exceeds the context window — what do you drop?" Keep the system prompt and the most recent turns; drop or summarise the stale middle. Truncation is the simple answer; summarisation — compressing older turns into a running summary that travels with the recent window — preserves continuity without blowing the token budget. Say you never drop the system prompt or the latest user message.

A ChatGPT context and safety diagram showing a long conversation in the store, a context builder that keeps the system prompt and recent turns while summarising older ones to fit the context window, pre and post moderation filters around generation, and a gateway enforcing rate limits, quotas and a max-token cost guardrail.

The follow-ups that decide the ChatGPT round

Scan the probes from each deep dive and they rhyme. Master these five and you are ready for a version of the question that is not drawn exactly like this one.

  • "Why not return the whole answer in one response?" → Generation is autoregressive and slow; stream token deltas over SSE so time-to-first-token is tiny and independent of length. (problem solving)
  • "Two users, one GPU — serve both efficiently." → Continuous batching merges their steps into one forward pass; each keeps its own KV cache. (technical execution)
  • "What caps how many chats a GPU can hold?" → Not the weights — the KV cache, which grows with context length × batch size; page and evict idle sessions. (technical execution)
  • "A 100-message chat exceeds the window — what goes?" → Keep the system prompt and recent turns; truncate or summarise the stale middle. (communication)
  • "What's the real bottleneck and cost at scale?" → GPUs — so throughput per GPU via batching, plus rate limits, cheaper-model routing and max-token caps to bound the bill. (problem solving)

Each maps to a debrief dimension. Reading this list teaches you the answers; it does not teach you to deliver them while someone interrupts — that only comes from saying them out loud.

How to prepare: a one-week plan

You do not need fifty systems. You need this one cold and the framework automatic, so the next "design X" is just new nouns in the same five steps.

  • Days 1–2: Learn the five-step framework until it is reflex. Do ChatGPT end to end on paper — requirements, estimate, data model, the client → gateway → orchestrator → GPU cluster → stream-back loop.
  • Days 3–4: Drill the streaming and serving deep dive: autoregressive generation, SSE, time-to-first-token, and why one-shot JSON loses. Say each tradeoff in one sentence.
  • Days 5–6: Own GPU efficiency and context — continuous batching, the KV cache and its memory limit, context truncation/summarisation, moderation and rate limits. Answer each of the five follow-ups out loud.
  • Day 7: A full timed 45-minute round, narrating the whole way, with someone (or something) interrupting. This is the rep that counts.

This post is the deep walkthrough behind one row of the broader Top system design interview questions — once ChatGPT serving is automatic, the same spine carries the rest of the list, including the retrieval-focused design a RAG system round.

Frequently asked questions

How do I design ChatGPT step by step?

Follow the five-step HLD framework. Clarify requirements (multi-turn chat, token streaming, conversation context, moderation, rate limits, at millions of concurrent sessions); estimate to show the hard axis is GPU throughput and memory, not request count; define the streaming chat API plus the conversation, message, KV-cache and quota tables; draw the serving loop (client → gateway → pre-moderation → context builder → orchestrator queue → GPU serving cluster → stream tokens back → post-moderation, with the conversation store); then deep-dive the decisions that matter — token streaming, GPU batching and the KV cache, and context management with safety. Finish on the follow-ups about the window and the real cost.

Why does ChatGPT stream tokens instead of returning the whole answer?

Because a language model is autoregressive: it generates one token at a time, each conditioned on all the tokens before it, so a long reply takes several seconds to finish. If the server waited and returned it as one response, the user would see a spinner for the entire generation and lose everything on a dropped connection. Streaming each token over Server-Sent Events puts the first word on screen in a fraction of a second and makes perceived latency independent of the answer's length — time-to-first-token, not total time, is the metric that matters.

What is the KV cache and why does it matter for LLM serving?

The KV cache stores the attention key and value tensors for the tokens a model has already processed, so each new token is a cheap incremental step instead of recomputing the whole context every time. It is reused across the turns of a conversation, so the history is not re-encoded on every message. It matters because it is the main memory pressure on the GPU: the cache grows with context length times batch size, and that — not the model weights — is what caps how many concurrent chats a single GPU can hold.

How do you serve many users on the same GPU efficiently?

With continuous batching. A GPU serving one request at a time is mostly idle, so the serving engine merges many users' next-token steps into a single forward pass and amortises the cost of loading the model weights across the whole batch. Continuous (dynamic) batching improves on static batching by letting a new request take a finished sequence's slot mid-flight, keeping the batch and the GPU full. Because GPUs are the dominant cost, throughput per GPU through batching is the central efficiency lever of the whole system.

What happens when a conversation exceeds the context window?

The model can only attend to a fixed number of tokens, so the context builder cannot send the whole history once a chat grows long. It keeps the system prompt and the most recent turns, and handles the overflow by either truncating the oldest messages or summarising earlier turns into a running summary that travels alongside the recent window. The full conversation still lives in the store for display and context rebuilding; only the windowed slice is loaded into each prompt, which also keeps the expensive GPU input small.

How do I practise a system design interview out loud?

Reading a walkthrough is not the same as defending a design while someone interrupts. On Zynter, the HLD track runs live voice rounds modelled on how real system design loops run — you talk through a shared design on the canvas, and a voice interviewer pushes with the follow-ups above one at a time, then gives you a debrief on problem solving, communication, technical execution, and time management. The score is a practice signal, not a hiring decision.

Practise it on Zynter

You can read every hop. The round is won on streaming and the GPU economics.

Take a system design round, draw the ChatGPT serving loop on the canvas, and defend it out loud to a voice interviewer that keeps asking "why not return the whole answer at once?" and "two users, one GPU — how do you serve both?" — then read the debrief. Zynter.ai is on-demand mock interviews modelled on real company loops, with 150+ rounds across 20 companies including the HLD system design track (Sep 2026).

Practise a system design round →
Browse all interview tracks

Top comments (0)