Design a logging system: the system design interview walkthrough
The request to design a logging system sounds like a one-liner — write a line to a file, maybe timestamp it. For one service on one box that is exactly what it is. But the interview asks you to collect, store and search logs from thousands of services emitting terabytes a day, without dropping the signal when traffic spikes, without paying to keep every line instantly searchable forever, and while an on-call engineer still tails a live stream and queries last Tuesday in seconds — and that is a throughput and storage problem, not a file writer. This is one of the most common system design interview prompts because the one-line brief hides a write-heavy pipeline whose ingest can jump tenfold in a minute, and a distributed logging system design that lives or dies on how it buffers and tiers.
The round turns on two decisions. First, the shape of the write path — how log lines get off thousands of hosts and into a central place fast enough that a sudden burst does not take the pipeline down, which is really a question about buffering and backpressure. Second, where the data lives once it arrives: recent logs need to be searched in milliseconds, old logs are queried rarely but must be kept for ninety days, and keeping all of it in a fast index would cost a fortune — so log aggregation system design is as much about storage tiers as it is about ingestion. This walkthrough follows the path a strong candidate draws on the whiteboard — requirements, estimate, API and data model, high-level design, then the deep dives — and at each step it names the follow-up the interviewer uses to tell a memorised answer from an engineer who understands the tradeoff.
How the logging system round is actually run
A logging system round runs about 45 minutes on a shared canvas — you draw the boxes and arrows while you talk, and the interviewer interrupts. It is a favourite because the one-line problem opens onto deep water fast: the moment you write "send the logs to a central store," the question becomes how fast, how do you not fall over in a burst, and how do you not pay to keep all of it hot — and every answer after that (buffering, tiering, search, alerting) compounds. Get the ingest buffer and the storage tiers clean and you have shown the whole pipeline in one problem.
The interviewer rarely lets "ship the logs to a central database" stand. "Traffic 10x's for five minutes and the log volume with it — what absorbs the spike, and what are you willing to drop?" That question — surviving a burst on a design that is otherwise just a firehose into a store — is the one a copied answer can never rehearse, and the one this round is really testing.
The rounds here are modelled on how the loop runs, not scraped from any candidate's recall. What matters for prep is the shape: a blank canvas, a clock, and a follow-up after every decision you make.
The five-step framework for any HLD round
Every high-level design answer fits the same spine. Say it out loud at the start so the interviewer knows you have a plan, then walk it:
- Clarify requirements — functional and non-functional, and what you are explicitly not building.
- Estimate — log lines per second, bytes per day and read vs write mix, to justify every later choice.
- High-level design — the boxes and arrows: app, agent, queue, indexer, hot store, cold store, query.
- Deep dive — the two or three decisions that actually decide this system.
- Follow-ups — bottlenecks, failure modes, and what you would change at ten times the scale.
What the debrief measures
The score is not "right or wrong design." A good HLD debrief reads on four dimensions, and every move below maps to one: problem solving (did you put a buffer between producers and the index so a spike cannot topple it), communication (did you say why some log loss is acceptable in a sentence), technical execution (batch at the agent, Kafka as the shock absorber, hot index plus cold object storage, trace_id correlation), and time management (did you spend your 45 minutes on ingestion and storage tiers, not on the log parser's regex). Reading a walkthrough teaches the design; it does not teach you to deliver it on the clock while someone pushes back.
Step 1 — Requirements: what a logging system must do
Spend the first few minutes here. Jumping to boxes before you have scoped the problem is the fastest way to design the wrong system.
| Type | Requirement |
|---|---|
| Functional | Ingest logs from many services and hosts; centralise them; make them searchable by text and fields; support a real-time tail of a live stream and historical query over days; evaluate alert rules on the stream; retain and archive logs on a schedule. |
| Functional (optional) | Correlate a single request across services via a trace_id; sample high-volume debug logs; multi-tenant isolation per team — call these out, then scope most of them for later. |
| Non-functional | Very high, bursty write throughput; durable-enough (losing a small fraction of logs under extreme load is acceptable); fast search over recent data; cost-efficient storage for the long tail; horizontally scalable and fault-tolerant. |
The sentence that earns points: "Writes dwarf reads and arrive in bursts — this is a write-optimised pipeline with a buffer in front, and a storage layer that gets cheaper as logs get older, not one database doing everything." Say that and you have framed the entire design before drawing a box.
Step 2 — Capacity estimation (back of the envelope)
Estimation here is about proving one thing: the hard axis is absorbing a write-heavy, bursty stream and storing it cheaply over time, not serving queries.
| Quantity | Assumption | Result |
|---|---|---|
| Log volume | 10k services × ~100 lines/sec | ~1M log lines/sec at peak |
| Average line size | ~1KB | ~1GB/sec → ~80TB/day → petabytes at 90-day retention |
| Write vs read | millions of writes/sec, a few hundred searches/min | write-heavy by ~10,000:1 → optimise the write + storage path |
| Burst factor | traffic can 10x in minutes | peak ≫ average → a buffer must absorb the spike |
Two conclusions fall out, and you should say both out loud. First, the write path is the system — the pipeline has to swallow a million lines a second that can spike to ten million, so a buffer (a durable queue) sits between the producers and anything slower downstream, and you accept bounded loss rather than block the apps. Second, storage is huge but tiered — recent logs live in a fast search index, everything older rolls to cheap object storage — so you spend your minutes on ingestion and the hot/cold split, not on the query UI. You have now pointed the interview at the real deep dive before anyone asked.
Step 3 — API and data model
A logging system's "interface" is a thin ingestion contract plus a query API, and underneath it a flat, append-only event schema — most of the design is the pipeline, not the API.
| Stage call | Operation |
|---|---|
agent.emit(event) |
a per-host agent buffers and batches events, then ships a compressed batch upstream |
ingest.publish(batch) |
append the batch to the durable queue (Kafka topic), partitioned by service or trace_id |
search(query, time_range) → hits[]
|
query the hot index over a recent window, or fall back to cold storage for older ranges |
tail(filter) → stream
|
subscribe to the live stream for a real-time follow of matching events |
The data model is a single append-only log event — flat, schema-light, indexed on a few fields:
| Entity | Notes |
|---|---|
log_event |
ts, service, host, level, trace_id, message, plus a bag of structured fields (JSON). |
hot_index |
time-partitioned inverted index (e.g. Elasticsearch) over recent events, indexed on ts, service, level, trace_id. |
cold_store |
raw/compressed events in object storage (e.g. S3), partitioned by date/service, for cheap long-term retention. |
alert_rule |
query + threshold + window, evaluated continuously on the stream, not on the index. |
Step 4 — High-level design: the logging pipeline
Now the boxes. The whole system is one pipeline that moves an event from an app to a place you can search it, with a buffer in the middle so bursts never hit the slow parts directly:
- App → log agent: a lightweight agent on each host collects stdout/log files, buffers and batches events, compresses them, and ships them upstream — so the app never blocks on logging.
- Agent → ingestion queue (Kafka): a durable, partitioned message queue that absorbs bursts and decouples producers from consumers; it is the shock absorber of the whole design.
- Processing / indexing workers: consume from the queue, parse and enrich each event (structure the fields, attach metadata), then fan out to two stores.
- Hot store (search index) + cold store (object storage): recent events go into a time-partitioned inverted index for fast search; the same events are written to cheap object storage for the long retention tail.
- Query / UI + alerting: a query service searches the hot index (and cold storage for old ranges) and serves the live tail; alert rules are evaluated on the stream as events flow through.
Say explicitly that the queue makes this a buffered pipeline, not a direct write — the apps produce into Kafka and walk away, and the indexers consume at their own pace, which is exactly why a spike that outruns the indexers only grows the queue (and, at worst, drops the oldest) instead of taking logging down for everyone. Naming the buffer unprompted is a signal the interviewer is listening for.
Deep dive: high-throughput ingestion and buffering
This is the first decision that actually decides the round. Writing a million lines a second straight into a search index does not work — the index cannot keep up with bursts, and if it stalls the apps stall behind it. The senior answer puts two shock absorbers in the path and is explicit about what gives when the pressure is too high.
| Layer | What it does | Why it exists |
|---|---|---|
| Agent-side batching | the per-host agent buffers events in memory and ships them in compressed batches on a size/time trigger, not one request per line. | So one network round-trip carries thousands of events, and the app never blocks on a slow downstream. |
| Kafka as a buffer | a durable, partitioned queue sits between agents and indexers; producers append at memory speed, consumers drain at their own pace. | So a burst that outruns the indexers piles up in the queue instead of toppling the pipeline — the queue is the backpressure. |
The flow under pressure: when log volume 10x's, the agents keep batching, Kafka's partitions grow, and the indexers fall behind but never fall over. Because this is a logging system — not a bank ledger — you choose best-effort / at-least-once delivery over strict exactly-once: if the agent's buffer overflows or the queue hits its retention cap, you drop low-value logs (debug lines, sampled traffic) rather than block the app or lose the host. Stating that some loss is acceptable and deliberately chosen is the point most candidates miss — they over-engineer durability the problem does not need.
"Traffic 10x's for five minutes — what absorbs it, and what do you drop?" The agent batches and Kafka's partitions absorb the burst as backlog; the indexers drain it afterwards. If the buffer still overflows you shed the lowest-value logs (debug, sampled) first — best-effort, because losing a fraction of debug lines beats blocking production traffic. Name both the absorber and the drop policy.
System Design
HLD · Infrastructure · 45 min
Design a logging system out loud, with follow-ups
Deep dive: storage tiers and retention
This is the decision most candidates under-think. The follow-up is always about money: you keep ninety days of logs but almost every search hits the last two — so storing all ninety days in a fast search index is burning cash for data nobody queries.
Hot tier — the search index. Recent logs (say the last few days) live in a time-partitioned inverted index such as Elasticsearch: fast full-text and field search, but expensive per gigabyte because it holds the index structures in memory and on fast disk. You keep the hot tier small on purpose — only the window people actually search interactively.
Cold tier — object storage. Every event is also written, compressed, to cheap object storage (S3 and the like), partitioned by date and service. It is a fraction of the cost per terabyte, searches are slower (you scan files instead of hitting an index), but that is fine because old-log queries are rare and can tolerate seconds. An index lifecycle / TTL policy ages data out of the hot index on a schedule — roll it to cold after N days, delete from cold after the retention limit.
Controlling index size. Beyond tiering, you keep the hot index affordable by sampling high-volume logs (store 1 in N debug lines, keep all errors), capping the fields you index, and using time-based indices so dropping an old day is a cheap partition delete, not a mass row deletion.
"You keep 90 days but usually search the last 2 — how do you store that cheaply?" A hot inverted index holds only the recent searchable window; everything rolls to compressed object storage, partitioned by date, for the long tail. A lifecycle policy moves data hot → cold → deleted on a timer, and you sample high-volume logs so the hot index stays small. Name the two tiers and the lifecycle rule.
System Design
HLD · Infrastructure · 45 min
Defend ingestion and storage tiers out loud
Deep dive: search, tailing and alerting
The read side is where the three functional modes — historical search, live tail, and alerting — each take a different path through the same pipeline, and where the distributed-tracing link shows up.
The three paths, end to end: historical search hits the inverted index over a time range; because indices are time-partitioned, a query for "last Tuesday" touches only that day's partition instead of scanning everything, and older ranges fall back to scanning cold object storage. Real-time tail does not touch the index at all — it subscribes to the live stream coming off Kafka and filters events as they flow, so an on-call engineer watches production in real time. Alerting rules (error rate over a threshold in a window) are evaluated on the stream as events pass the processing stage, not by polling the index, so an alert fires in seconds, not after an indexing lag.
The tracing link closes the loop: when an incident spans five services, you follow one request by its trace_id, stamped on every log line as the request crosses service boundaries and carried through the pipeline as an indexed field. One search on that trace_id pulls every service's logs for that request into a single timeline — which is exactly why a logging system and a distributed tracing system are built on the same ingest-and-correlate backbone.
| Concern | Naive | Scaled |
|---|---|---|
| Historical search | scan one giant index | time-partitioned indices so a dated query touches only the relevant partition; cold-storage scan for old ranges. |
| Live tail | poll the index repeatedly | subscribe to the Kafka stream and filter in flight — no index load at all. |
| Alerting | cron a query every minute | evaluate rules on the stream as events flow, so alerts fire in seconds. |
Repeat the real correlation axis: the single field that makes logs useful across a distributed system is the trace_id — index it, and "search, tail, and follow one request across five services" are all the same primitive on the same pipeline.
"An incident spans five services — how do you follow one request across all of them?" Every log line carries a trace_id stamped when the request enters and propagated across service calls; it is indexed, so one search on that trace_id reconstructs the whole request's timeline across all five services. That shared correlation id is the bridge from logging to distributed tracing.
The follow-ups that decide the logging system round
Scan the probes from each deep dive and they rhyme. Master these five and you are ready for a version of the question that is not drawn exactly like this one.
- "Traffic 10x's for five minutes — what absorbs it and what do you drop?" → Agent batching plus a Kafka buffer absorb the burst as backlog; shed low-value logs (debug, sampled) first — best-effort delivery. (problem solving)
- "You keep 90 days but search the last 2 — how do you store that cheaply?" → A small hot inverted index for recent logs, everything rolled to cheap object storage, aged by a lifecycle/TTL policy. (technical execution)
- "An incident spans five services — how do you follow one request?" → A trace_id stamped on every line and propagated across services, indexed so one search rebuilds the whole request timeline. (communication)
- "Can you lose logs? How much durability do you actually need?" → Durable-enough: at-least-once from the agent and queue, but bounded loss under extreme load is acceptable for logs, so you never block production. (problem solving)
- "What's the real bottleneck at scale?" → Not query speed — ingestion throughput and keeping the hot index small; the buffer and the hot/cold split are where this system lives. (technical execution)
Each maps to a debrief dimension. Reading this list teaches you the answers; it does not teach you to deliver them while someone interrupts — that only comes from saying them out loud.
How to prepare: a one-week plan
You do not need fifty systems. You need this one cold and the framework automatic, so the next "design X" is just new nouns in the same five steps.
- Days 1–2: Learn the five-step framework until it is reflex. Do the logging system end to end on paper — requirements, estimate, data model, the app → agent → queue → indexer → hot/cold → query pipeline.
- Days 3–4: Drill the ingestion deep dive: agent batching, Kafka as the buffer, backpressure, at-least-once vs best-effort, and why some log loss is acceptable. Say each tradeoff in one sentence.
- Days 5–6: Own storage and search — the hot index vs cold object storage split, lifecycle/TTL and sampling, time-partitioned search, the live tail off the stream, and trace_id correlation. Answer each of the five follow-ups out loud.
- Day 7: A full timed 45-minute round, narrating the whole way, with someone (or something) interrupting. This is the rep that counts.
This post is the deep walkthrough behind one row of the broader Top 25 system design interview questions — once the logging pipeline is automatic, the same spine carries the rest of the list.
Frequently asked questions
How do I design a logging system step by step?
Follow the five-step HLD framework. Clarify requirements (ingest from many services, centralise, search, live tail plus historical query, alerting, retention — at a million lines a second, bursty, durable-enough); estimate to show the hard axis is a write-heavy, bursty stream stored cheaply over time, not query load; define the ingestion and query contract plus the flat log_event schema, hot index, cold store and alert_rule; draw the pipeline (app → agent → Kafka → indexing workers → hot search index + cold object storage → query/tail/alerting); then deep-dive the decisions that matter — high-throughput ingestion and buffering, storage tiers and retention, and search/tailing/alerting with trace_id correlation.
Why put a message queue like Kafka in a logging system?
Because writes are enormous and bursty while the search index is comparatively slow to write, so you never let producers write to the index directly. A durable, partitioned queue sits between the agents and the indexers: producers append at memory speed and move on, consumers drain at their own pace, and a spike that outruns the indexers piles up as queue backlog instead of toppling the pipeline. The queue is the backpressure and the shock absorber — it is what lets the system survive a 10x burst and lets you drop only the lowest-value logs when the buffer is truly full.
How does a logging system store logs cheaply at scale?
It tiers storage by age. Recent logs — the window people actually search interactively — live in a fast, time-partitioned inverted index that is expensive per gigabyte but quick to query. Every event is also written, compressed, to cheap object storage partitioned by date and service for the long retention tail, where searches are slower but rare. An index lifecycle or TTL policy moves data hot → cold → deleted on a schedule, and sampling high-volume debug logs while keeping all errors keeps the hot index small, so you pay index prices only for data that is actually queried.
Is it acceptable to lose some logs?
For a logging system, usually yes, and saying so out loud earns points. Logs are telemetry, not a financial ledger, so the design targets "durable-enough": the agent and queue give at-least-once delivery in the normal case, but under extreme load — an overflowing agent buffer or a queue at its retention cap — you deliberately drop low-value logs (debug lines, sampled traffic) rather than block production apps or crash a host. Over-engineering exactly-once durability the problem does not need is a common way to lose time in this round.
How does trace_id tie logs to distributed tracing?
A trace_id is a single correlation id stamped on a request when it enters the system and propagated across every service call it makes, and it is written onto every log line and indexed as a field. When an incident spans five services, one search on that trace_id pulls every service's logs for that one request into a single ordered timeline, instead of you grepping five systems separately. That shared, indexed correlation id is exactly the backbone a distributed tracing system is built on, which is why logging and tracing are designed on the same ingest-and-correlate pipeline.
How do I practise a system design interview out loud?
Reading a walkthrough is not the same as defending a design while someone interrupts. On Zynter, the HLD track runs live voice rounds modelled on how real system design loops run — you talk through a shared design on the canvas, and a voice interviewer pushes with the follow-ups above one at a time, then gives you a debrief on problem solving, communication, technical execution, and time management. The score is a practice signal, not a hiring decision.
Practise it on Zynter
You can read every hop. The round is won on the ingest buffer and the storage tiers.
Pick the Distributed Tracing round, draw the logging pipeline on the canvas, and defend it out loud to a voice interviewer that keeps asking "what absorbs a 10x spike?" and "how do you keep 90 days cheaply?" — then read the debrief. Zynter.ai is on-demand mock interviews modelled on real company loops, with 150+ rounds across 20 companies including the HLD system design track (Sep 2026).
Practise the Distributed Tracing round →
Browse all interview tracks




Top comments (0)