Design a distributed message queue: the system design interview walkthrough

Design a distributed message queue: the system design interview walkthrough

The request to design a distributed message queue sounds like one line — producers send messages, consumers read them — and that one line hides the whole interview. Do it for one process and you reach for an in-memory list; do it for millions of messages per second, surviving a broker crash without losing a message, preserving order, and letting many consumers read independently, and it becomes a durability-plus-throughput problem with a dozen sharp edges. This is one of the most common message queue system design prompts because it forces every distributed-systems muscle at once: durable writes, partitioning, offsets, delivery guarantees, and replication.

The round turns on one structural insight. A classic in-memory queue pops a message off and it is gone; a distributed messaging system instead writes every message to a partitioned append-only log on disk, hands each one an offset, and lets consumers read at their own pace from any position. Get that one reframe — the log is the system, not a queue of transient items — and the rest of the system design interview is deciding how to partition it, how consumers track where they are, and how replication keeps a message alive when the machine holding it dies. This walkthrough follows the path a strong candidate draws on the whiteboard — requirements, estimate, API and data model, high-level design, then the deep dives — and at each step it names the follow-up the interviewer uses to tell a memorised answer from an engineer who understands the tradeoff.

How the message queue round is actually run

A message queue round runs about 60 minutes on a shared canvas — you draw the boxes and arrows while you talk, and the interviewer interrupts. It is a favourite because the one-line problem opens onto deep water fast: the moment you draw "producer writes, consumer reads," the question becomes where does the message live, and every answer after that (ordering, parallelism, durability) compounds from the log. Get the append-only spine clean and you have shown the whole loop in one problem.

The interviewer rarely lets "producers write, consumers read" stand. "A broker holding a partition crashes right after it accepts a write but before anyone reads it — is that message lost? Walk me through exactly what has to be true for the answer to be no." That question — durability under failure — is the one a copied answer can never rehearse, and the one this round is really testing.

The rounds here are modelled on how the loop runs, not scraped from any candidate's recall. What matters for prep is the shape: a blank canvas, a clock, and a follow-up after every decision you make.

The five-step framework for any HLD round

Every high-level design answer fits the same spine. Say it out loud at the start so the interviewer knows you have a plan, then walk it:

1. Clarify requirements — functional and non-functional, and what you are explicitly not building.
2. Estimate — throughput, message size, retention and consumer count, to justify every later choice.
3. High-level design — the boxes and arrows: producers, brokers, partitioned logs, consumers.
4. Deep dive — the two or three decisions that actually decide this system.
5. Follow-ups — bottlenecks, failure modes, and what you would change at ten times the scale.

What the debrief measures

The score is not "right or wrong design." A good HLD debrief reads on four dimensions, and every move below maps to one: problem solving (did you reach a partitioned append-only log that is durable and independently consumable), communication (did you say why an append-only log beats an in-memory queue in a sentence), technical execution (sequential disk writes, per-partition offsets, ISR and acks), and time management (did you spend your 60 minutes on the log, consumer groups, and replication, not on the message schema). Reading a walkthrough teaches the design; it does not teach you to deliver it on the clock while someone pushes back.

Step 1 — Requirements: what a message queue must do

Spend the first few minutes here. Jumping to boxes before you have scoped the problem is the fastest way to design the wrong system.

Type	Requirement
Functional	Producers publish messages to a named topic; consumers subscribe and consume; messages are durable and retained for a window (hours to days); support many independent consumer groups reading the same topic; scale horizontally as load grows.
Functional (optional)	Exactly-once delivery, message priority, delayed/scheduled delivery — call these out, then scope most of them for later.
Non-functional	High throughput (millions of messages/sec); durability (no loss on a broker crash); ordering guarantee at least within a partition; high availability; horizontal scalability across brokers.

The sentence that earns points: "The hard part isn't the queue semantics, it's doing durable, ordered, high-throughput writes while many consumers read independently — so the core of the design is a partitioned append-only log on disk, not a classic in-memory queue." Say that and you have framed the entire design before drawing a box.

Step 2 — Capacity estimation (back of the envelope)

Estimation here is about proving one thing: the hard axis is durable, high-throughput append plus independent consumption, not clever queue logic.

Quantity	Assumption	Result
Write throughput	millions of messages/sec	sequential disk append, not random writes
Message size	~1 KB average	~GB/sec of write bandwidth at peak
Retention	keep messages for ~7 days	terabytes on disk per topic; storage is a first-class concern
Consumers	many producers, many consumer groups	each group reads the full log at its own offset, so reads multiply

Two conclusions fall out, and you should say both out loud. First, the append-only log is the system — because writes are sequential appends to disk, a commodity broker can sustain enormous throughput, and retention makes the log the durable source of truth rather than a transient buffer. Second, because every consumer group reads the whole log independently, you cannot delete a message when one reader consumes it; you delete by retention policy, not by acknowledgement. You have now pointed the interview at the real deep dive — the partitioned log — before anyone asked.

Step 3 — API and data model

The API is deliberately small: producers append, consumers poll by offset, and offsets are committed back.

Operation	Shape
publish(topic, key, message)	producer appends; the key decides the partition; returns the assigned (partition, offset).
poll(topic, group, max)	consumer pulls a batch from its assigned partitions, starting at its committed offset.
commit(topic, group, partition, offset)	consumer records how far it has processed, so a restart resumes from there.

The data model is thin. The log entry is the unit, and the broker stamps the offset so position never depends on a client:

Entity	Notes
message	offset (broker-assigned, monotonically increasing per partition), key, value, timestamp.
partition	an append-only log file (plus index) on a broker's disk; ordered by offset.
consumer_offset	(group, topic, partition) → last committed offset, the bookmark each group keeps.

Step 4 — High-level design: the partitioned-log architecture

Now the boxes. A producer publishes to a topic, which is split into partitions; each partition is an append-only log living on a broker's disk. Around that sit the pieces that make scaling and durability work:

• Producers: choose a partition (by key hash, or round-robin) and append; they get back the (partition, offset).
• Brokers: servers that each host a set of partition logs — some as leader, some as follower replicas — and serve reads and writes for the partitions they lead.
• Partitioned logs on disk: the topic's data, split across brokers so throughput scales with partition count; each append is a fast sequential write.
• Consumers in consumer groups: pull by offset; within a group each partition is read by exactly one member, so the group splits the work.
• Coordinator (metadata / leader election): tracks which broker leads each partition, assigns partitions to consumers, and drives failover — often backed by a consensus service.

Say explicitly that a topic is not one queue — it is a set of independent partition logs, and that single fact is what gives you ordering, parallelism, and durability all at once. Naming that unprompted is a signal the interviewer is listening for.

Deep dive: the partitioned append-only log

This is the first decision that actually decides the round. Where does a message live, and why does that shape everything else?

A topic is split into partitions, and each partition is an append-only log on disk: new messages are written only at the tail, and each gets a monotonically increasing offset. Three properties fall straight out of that choice:

Property	Why the append-only log gives it
Throughput	Appends are sequential disk writes — the OS page cache and sequential I/O make this far faster than random writes, so a single broker sustains huge write rates.
Ordering	Within a partition, offsets are strictly increasing, so order is guaranteed. Across partitions there is no global order — a deliberate trade for parallelism.
Replayability	Because messages are retained and never popped, a consumer can re-read from any offset; the log is the durable source of truth, not a transient buffer.

Partitioning is also how you scale. One partition is one ordered log that one consumer reads; split a topic into N partitions and N consumers can read in parallel. The cost is that ordering is only per-partition — so you choose a partition key (e.g. user_id) to guarantee that all messages that must stay ordered land on the same partition.

"You keep saying 'the log' — why not just an in-memory queue that pops messages off?" Because a popped message is gone, so you lose durability and replay, and a single queue is one ordered stream you can't parallelise. An append-only log on disk is durable, replayable, and splits into partitions for throughput — order is kept per partition, which is all most workloads need.

System Design
HLD · Messaging & realtime · 60 min
Design a distributed message queue out loud, with follow-ups

Practise on Zynter →

Deep dive: delivery semantics and consumer groups

This is the decision most candidates under-think. The write path is easy; the follow-up is always "how do many consumers read this without stepping on each other, and what happens when one fails mid-message?"

Consumers are pull-based. Rather than the broker pushing, each consumer polls for a batch starting at its own offset. Pull lets a slow consumer fall behind without being overwhelmed, lets it batch reads, and keeps the broker simple — it just serves bytes from a given offset.

Consumer groups split the work. A consumer group is a set of consumers sharing a subscription; the coordinator assigns each partition to exactly one member of the group, so the group collectively reads every message once and the load spreads across members. Add a member and partitions rebalance; lose one and its partitions move to survivors. Crucially, different groups are independent — each keeps its own offsets and each receives every message, which is how one topic feeds analytics, billing, and search at once.

Offsets are the bookmark, and when you commit them decides your delivery semantics:

Semantic	How	Cost
At-most-once	Commit the offset before processing.	If the consumer crashes mid-process, that message is skipped — possible loss.
At-least-once	Commit after processing.	On a crash before commit, the message is re-delivered — so the consumer must be idempotent. The common default.
Exactly-once	At-least-once plus deduplication (idempotent producer + transactional commit of output and offset together).	Correct but expensive — extra coordination and throughput cost; reach for it only when the workload truly can't tolerate duplicates.

The honest interview answer is: default to at-least-once and make consumers idempotent; treat exactly-once as a costly option you scope, not a free guarantee.

"Two consumers read the same partition — do they both get every message?" Not within one group — a partition is assigned to exactly one member, so the group sees each message once. But two different groups each get the full stream independently, because each group tracks its own offsets. That split is the whole point of groups.

System Design
HLD · Delivery & offsets · 60 min
Defend delivery semantics and consumer groups out loud

Practise on Zynter →

Deep dive: replication, durability and fault tolerance

Groups and offsets decide who reads; this decides whether the message survives at all. The follow-up is always the crash case.

Each partition is replicated across several brokers: one broker is the leader for that partition and the others hold follower replicas. All writes and reads for the partition go through the leader; followers continuously pull from the leader to stay current. The set of replicas that are fully caught up is the in-sync replica (ISR) set.

Producer acks decide the durability/latency trade:

acks setting	Broker confirms when	Trade
acks=0	never waits	fastest, but a crash loses unconfirmed writes — fire-and-forget.
acks=1	the leader has written	fast, but if the leader dies before followers copy the write, that message is lost.
acks=all	all in-sync replicas have the write	strongest durability — the message survives any single broker loss; higher latency.

Durability also means persisting to disk plus retention: the log is written to disk (not held only in memory), and messages live for the retention window regardless of whether they've been consumed, so a reader can always re-read and a crash can't erase committed data.

On leader failure, the coordinator promotes an in-sync follower to leader, producers and consumers redirect to the new leader, and because the promoted replica was in the ISR it already has every committed message — no loss. This is exactly the crash the interviewer opened with: with acks=all and replication, a broker dying mid-write does not lose the message, because it was only acknowledged once an in-sync replica also held it.

"A broker holding a partition crashes mid-write — is the message lost?" Not if it was produced with acks=all: the write is acknowledged only after every in-sync replica has it, and on failure the coordinator promotes an in-sync follower that already holds the message. With acks=1 you'd risk losing an unreplicated write — which is the trade you call out explicitly.

The follow-ups that decide the message queue round

Scan the probes from each deep dive and they rhyme. Master these five and you are ready for a version of the question that is not drawn exactly like this one.

• "Why an append-only log and not an in-memory queue?" → A popped message loses durability and replay; a log on disk is durable, replayable, and partitions for throughput, with order kept per partition. (communication)
• "A broker crashes mid-write — is the message lost?" → Not with acks=all and replication: acknowledged only after an in-sync replica holds it, and an ISR follower is promoted on failure. (problem solving)
• "Two consumers read the same partition — do both get it?" → Within one group, no — one member per partition; different groups each get the full stream via their own offsets. (technical execution)
• "How do you guarantee no duplicates — exactly-once?" → Default to at-least-once with idempotent consumers; exactly-once needs idempotent producers plus transactional offset+output commit, and costs throughput. (technical execution)
• "You need strict order but also more throughput — how?" → Order is per-partition, so partition by key; add partitions for parallelism, accepting that order holds only within a key. (problem solving)

Each maps to a debrief dimension. Reading this list teaches you the answers; it does not teach you to deliver them while someone interrupts — that only comes from saying them out loud.

How to prepare: a one-week plan

You do not need fifty systems. You need this one cold and the framework automatic, so the next "design X" is just new nouns in the same five steps.

• Days 1–2: Learn the five-step framework until it is reflex. Do the message queue end to end on paper — requirements, estimate, API, data model, the partitioned-log boxes.
• Days 3–4: Drill the log deep dive: append-only writes, offsets, per-partition ordering, partition keys, and why partitioning is how you scale. Say each tradeoff in one sentence.
• Days 5–6: Own delivery and durability — pull-based consumers, consumer groups and offsets, at-least-once vs exactly-once, replication with ISR and acks, leader failover. Answer each of the five follow-ups out loud.
• Day 7: A full timed 60-minute round, narrating the whole way, with someone (or something) interrupting. This is the rep that counts.

This post is the deep walkthrough behind one row of the broader Top 25 system design interview questions — once the message queue is automatic, the same spine carries the rest of the list.

Frequently asked questions

How do I design a distributed message queue step by step?

Follow the five-step HLD framework. Clarify requirements (durable publish/subscribe, retention, many independent consumer groups, horizontal scale); estimate to show the hard axis is durable high-throughput append plus independent reads, not clever queue logic; define the small API (publish by key, poll by offset, commit offset) with a thin message/partition/offset model; draw the partitioned-log design (producers to topic partitions across brokers, each an append-only log on disk with replicas, consumer groups pulling by offset, a coordinator for assignment and leader election); then deep-dive the decisions that matter — the partitioned log, delivery semantics and consumer groups, and replication. Finish on the follow-ups about ordering and exactly-once.

Why use an append-only log instead of an in-memory queue?

Because a classic in-memory queue pops a message off when it is read, so the message is gone — you lose durability and the ability to replay, and a single queue is one ordered stream you cannot parallelise. An append-only log writes every message to disk at the tail, assigns it an offset, and keeps it for a retention window, so consumers read at their own pace from any position and can re-read after a failure. Splitting the log into partitions lets many consumers read in parallel, with ordering guaranteed within each partition.

How do consumer groups and offsets work in a message queue?

A consumer group is a set of consumers sharing a subscription. The coordinator assigns each partition to exactly one member of the group, so the group collectively reads every message once and the work spreads across members; add or lose a member and partitions rebalance. Each group tracks its own committed offset per partition — the bookmark of how far it has processed — so different groups read the same topic completely independently, which is how one stream can feed analytics, billing, and search at the same time.

What is the difference between at-least-once and exactly-once delivery?

It comes down to when you commit the offset. Commit after processing and a crash before the commit causes re-delivery — that is at-least-once, so consumers must be idempotent, and it is the common default. At-most-once commits before processing, risking loss. Exactly-once layers deduplication on top — an idempotent producer plus committing the output and the offset in one transaction — which is correct but costs extra coordination and throughput, so you scope it only when duplicates are genuinely unacceptable.

How does replication keep messages durable when a broker crashes?

Each partition is replicated across brokers: one leader handles reads and writes, and follower replicas pull from it to stay current, forming the in-sync replica (ISR) set. With producer acks=all, a write is acknowledged only after every in-sync replica has persisted it to disk, so no single broker loss can drop a committed message. When the leader fails, the coordinator promotes an in-sync follower that already holds every committed message, and producers and consumers redirect to it — delivery continues with no loss.

How do I practise a system design interview out loud?

Reading a walkthrough is not the same as defending a design while someone interrupts. On Zynter, the HLD track runs live voice rounds modelled on how real system design loops run — you talk through a shared design on the canvas, and a voice interviewer pushes with the follow-ups above one at a time, then gives you a debrief on problem solving, communication, technical execution, and time management. The score is a practice signal, not a hiring decision.

Practise it on Zynter

You can read every offset. The round is won on durability and independent consumption.

Pick the Distributed Message Queue round, draw the partitioned-log design on the canvas, and defend it out loud to a voice interviewer that keeps asking "a broker crashes mid-write — is it lost?" and "two consumers, same partition — who gets it?" — then read the debrief. Zynter.ai is on-demand mock interviews modelled on real company loops, with 150+ rounds across 20 companies including the HLD system design track (Sep 2026).

Practise the Message Queue round →
Browse all interview tracks