DEV Community: Gowtham Potureddi

AWS Lambda for ETL: Event-Driven Data Pipelines

Gowtham Potureddi — Wed, 09 Sep 2026 11:39:59 +0000

aws lambda etl is the pattern that turns your data pipeline from a clock-driven batch job that wakes up every hour into an event-driven system that reacts the instant a file lands, a message arrives, or a record is streamed — and it is the single serverless building block that most cleanly maps "something changed upstream" to "run this transform now." A raw CSV dropped into a landing bucket, a JSON payload pushed onto a queue, a clickstream record flowing through a shard: each of these is an event, and each event can invoke a small stateless function that reads the payload, transforms it, and writes it downstream to a warehouse, a curated bucket, or another queue — without a single server you have to patch, scale, or keep warm. The whole appeal is that you stop paying for idle compute and start paying per invocation, while the platform handles scaling from zero to thousands of parallel executions on its own.

But the appeal hides a set of hard edges that decide whether the design survives contact with production. Every invocation is stateless and time-boxed to fifteen minutes; delivery is at-least-once, so the same event can arrive twice; a sudden burst can spin up so many concurrent executions that it stampedes a fragile downstream database; the first invocation after a quiet period pays a cold start penalty; and a poison-pill record can wedge an entire stream partition until you route it to a dead-letter queue. This guide is the senior-data-engineering walkthrough of the trade-offs that actually come up in design reviews and interviews — when serverless functions fit an ETL job and when the fifteen-minute wall sends you to a different tool, how the event source mapping polls SQS and Kinesis while S3 triggers push, how concurrency, memory and cold starts interact, how retries, DLQ, and idempotency keep an at-least-once world correct, and how fan-out and a Step Functions handoff let you scale past the limits. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse the trigger and streaming muscle on the event-processing practice library →, and sharpen the transform axis with the data-processing practice library →.

On this page

When Lambda fits ETL (and when it doesn't)
Event sources & triggers — S3 / SQS / Kinesis
Concurrency, memory & cold starts
Error handling — DLQ, retries, idempotency
Patterns & limits — fan-out, Step Functions handoff
Cheat sheet — AWS Lambda ETL recipes
Frequently asked questions
Practice on PipeCode

1. When Lambda fits ETL (and when it doesn't)

Event-driven, stateless, fifteen-minute compute — the fit is glue and transform, not heavy batch

The one-sentence invariant: AWS Lambda is event-driven, stateless compute that runs for at most fifteen minutes per invocation and scales horizontally by running one sandbox per concurrent event — which makes it the right engine for lightweight, per-event ETL (transform one file, enrich one message, fan work out) and the wrong engine for anything that needs long-running state, a large in-memory shuffle, or a single job that must chew through terabytes end to end. The fit question is not "can Lambda do ETL" — it obviously can — but "does this job's duration, per-event data volume, and concurrency shape stay inside Lambda's envelope, and does the cost curve beat Glue or EMR at your event rate." Get that judgment right and you get a pipeline that scales from zero to thousands of parallel transforms with no cluster to babysit; get it wrong and you either hit the fifteen-minute wall mid-file or pay more than a provisioned cluster would have cost.

The axes that matter.

Job duration. A single Lambda invocation has a hard fifteen-minute ceiling. If one unit of work (one file, one batch, one shard read) cannot reliably finish in well under fifteen minutes with headroom, Lambda is the wrong home for that unit — you either shrink the unit (smaller files, smaller batches) or hand the job to Step Functions / Glue.
Data volume per event. Lambda gives you up to 10 GB of memory and up to 10 GB of ephemeral /tmp. A per-event transform that streams a modest file is a great fit; a job that must load a 50 GB file fully into memory is not. The trick is to process per object or per record batch, never "the whole dataset."
Concurrency shape. Lambda scales by concurrency — one sandbox per in-flight event. Spiky, bursty, event-triggered workloads are the sweet spot. A steady, always-on, high-throughput stream can still work but starts to look like a cluster you should have provisioned.
Statelessness. Every invocation starts fresh. There is no durable local state between invocations you can rely on (the sandbox may be reused, but you must not depend on it). Aggregations that need cross-event state must externalise it to DynamoDB, Redis, or a warehouse.
Cost curve. Lambda bills per millisecond × memory. At low-to-medium event rates it is dramatically cheaper than a warm cluster because you pay nothing when idle. Past a crossover rate, an always-busy Lambda fleet costs more than a right-sized Glue/EMR job. Every senior design names the crossover.

The 2026 reality — Lambda for the edges, big engines for the middle.

Lambda wins the glue. The canonical fit is the trigger-and-transform step: an object lands in S3, Lambda validates/converts/enriches it and writes a curated copy; a message hits SQS, Lambda upserts it into a warehouse; a Kinesis record streams in, Lambda routes it. These are short, per-event, embarrassingly parallel jobs.
Lambda wins fan-out and orchestration glue. Splitting a manifest into thousands of parallel tasks, invoking one worker per partition, reacting to a Step Functions state — Lambda is the worker and the glue between managed services.
Glue / EMR / Spark win the heavy batch. A single job that joins two billion-row tables, does a wide shuffle, or must run for an hour belongs on a distributed engine. Forcing it into Lambda means artificial chunking, cross-invocation state, and a fifteen-minute Tetris game you will lose.
Step Functions wins the long / multi-step workflow. Anything that needs to run longer than fifteen minutes, retry per step with backoff, branch on results, or coordinate dozens of Lambdas is a Step Functions state machine that uses Lambda for the short steps.

What interviewers listen for.

Do you name the fifteen-minute limit unprompted when asked "would you use Lambda here?" — required answer.
Do you say "delivery is at-least-once, so the handler must be idempotent" before being asked about duplicates? — senior signal.
Do you frame the fit as duration × per-event volume × concurrency × cost, not "serverless is always cheaper"? — senior signal.
Do you push back on "just Lambda the whole 40 GB nightly join" with "that's a Glue/EMR job; Lambda does the trigger and the fan-out"? — senior signal.
Do you describe Lambda as "one stateless sandbox per event" rather than "a server that's always on"? — required answer.

Worked example — the fit / no-fit decision table

Detailed explanation. The single most useful artifact for a serverless-ETL design discussion is a memorised fit table: for a candidate job, score it on duration, per-event volume, concurrency, and state, then read off "Lambda", "Lambda + Step Functions", or "Glue/EMR." Walk through building the table for four real jobs that a data platform team faces in a given quarter.

Job A. Convert each uploaded CSV (~50 MB) to Parquet as it lands in S3. Short, per-object, parallel, stateless.
Job B. Upsert order events from an SQS queue into Redshift, ~500 messages/sec. Short per-batch, needs idempotency.
Job C. Nightly join of a 2 B-row orders table with a 400 M-row customers table for a full re-aggregation. Long, huge shuffle, stateful.
Job D. Reprocess 90 days of raw logs (18 TB, 120 000 objects) into a curated table. Massive fan-out, each object small.

Question. Score each job on the four axes and assign the right execution engine.

Input.

Job	Duration/unit	Volume/event	Concurrency	State
A — CSV→Parquet	~20 s/file	50 MB	bursty, high	none
B — SQS→Redshift	~2 s/batch	~2 KB×10	steady, medium	idempotency key
C — 2B×400M join	~40 min	terabytes	one big job	full shuffle
D — 90-day backfill	~15 s/object	small each	120k objects	none

Code.

# fit_engine.py — score an ETL job and pick the engine
from dataclasses import dataclass

@dataclass
class Job:
    max_unit_seconds: int      # longest a single unit of work takes
    peak_gb_in_memory: float   # data a single unit must hold at once
    units: int                 # how many units (drives fan-out)
    needs_cross_event_state: bool

def pick_engine(j: Job) -> str:
    # Hard wall: a unit that cannot finish under Lambda's ceiling is out
    if j.max_unit_seconds > 14 * 60:          # keep a 1-min safety margin
        return "Step Functions / Glue (exceeds 15-min wall)"
    if j.peak_gb_in_memory > 10:              # Lambda memory ceiling
        return "Glue / EMR (per-unit memory too large)"
    if j.needs_cross_event_state:
        return "Lambda + external state (DynamoDB) or a stream engine"
    if j.units > 10_000:
        return "Lambda fan-out via Step Functions Distributed Map"
    return "Lambda (event trigger + transform)"

jobs = {
    "A": Job(max_unit_seconds=20,   peak_gb_in_memory=0.5, units=1,       needs_cross_event_state=False),
    "B": Job(max_unit_seconds=2,    peak_gb_in_memory=0.1, units=1,       needs_cross_event_state=True),
    "C": Job(max_unit_seconds=2400, peak_gb_in_memory=40,  units=1,       needs_cross_event_state=True),
    "D": Job(max_unit_seconds=15,   peak_gb_in_memory=0.2, units=120_000, needs_cross_event_state=False),
}
for name, j in jobs.items():
    print(name, "->", pick_engine(j))

Step-by-step explanation.

The function checks the hard walls first — duration and memory. These are non-negotiable platform limits; no amount of cleverness moves them, so they short-circuit the decision. A unit over ~14 minutes or over 10 GB is disqualified from Lambda immediately.
Job A passes every check: 20 s/file, 0.5 GB, no state, one triggering event per file. It is the textbook Lambda fit — an S3 trigger runs the transform per object.
Job B needs cross-event state only in the weak sense of deduplication. That is not a shuffle; it is an idempotency key lookup, which Lambda handles by writing to a DynamoDB dedupe table. So B stays on Lambda with external state, not a stream engine.
Job C fails on both walls: 40 minutes and 40 GB in memory. This is a distributed shuffle — a Glue or EMR/Spark job. Trying to Lambda it means chunking the join by key range across thousands of invocations with cross-invocation state, which is a reimplementation of Spark you should not write.
Job D is the interesting one: each unit is tiny (15 s, 0.2 GB) but there are 120 000 of them. That is not a Lambda-vs-cluster question; it is a fan-out question. Step Functions Distributed Map iterates the object manifest and invokes one Lambda per object with bounded concurrency — the last section covers exactly this.

Output.

Job	Engine chosen	Why
A — CSV→Parquet	Lambda (S3 trigger + transform)	short, per-object, stateless
B — SQS→Redshift	Lambda + DynamoDB idempotency	short batch; dedupe externalised
C — 2B×400M join	Glue / EMR	40 min + 40 GB shuffle
D — 90-day backfill	Step Functions Distributed Map + Lambda	120k tiny units; fan-out

Rule of thumb. Never pick "serverless" because it is trendy. Score the job on (duration × per-unit memory × unit count × cross-event state). The two hard walls — fifteen minutes and 10 GB — disqualify a job from Lambda outright; everything else is a fan-out or external-state design, not a reason to reach for a cluster.

Worked example — the cost crossover versus a warm cluster

Detailed explanation. The "serverless is cheaper" claim is true only below a crossover event rate. Lambda bills per invocation and per GB-second of duration; a Glue/EMR cluster bills per hour it is up regardless of load. Below the crossover, Lambda's pay-nothing-when-idle wins by a mile; above it, the always-busy Lambda fleet loses to a right-sized cluster. Walk through the crossover for a transform job.

Lambda price. Roughly $0.0000166667 per GB-second plus $0.20 per million requests (illustrative on-demand pricing; regions vary).
Job. Each invocation uses 1 GB memory for 2 seconds → 2 GB-seconds → ~$0.0000333 compute + request cost per event.
Cluster alternative. A small always-on job that costs ~$1.50/hour whether it processes 1 event or 1 million.

Question. At what steady event rate does the always-on cluster become cheaper than the Lambda fleet?

Input.

Parameter	Value
Lambda memory	1 GB
Lambda duration/event	2 s
Lambda GB-s price	$0.0000166667
Lambda request price	$0.20 / 1M
Cluster cost	$1.50 / hour

Code.

# crossover.py — where does an always-on cluster beat the Lambda fleet?
GB_S_PRICE   = 0.0000166667
REQ_PRICE    = 0.20 / 1_000_000
MEM_GB       = 1.0
DUR_S        = 2.0
CLUSTER_HR   = 1.50

cost_per_event = MEM_GB * DUR_S * GB_S_PRICE + REQ_PRICE
# events per hour where lambda_cost == cluster_cost
crossover_eph = CLUSTER_HR / cost_per_event
print(f"cost/event      = ${cost_per_event:.8f}")
print(f"crossover       = {crossover_eph:,.0f} events/hour")
print(f"                = {crossover_eph/3600:,.1f} events/second")

Step-by-step explanation.

Each event's Lambda cost is memory_gb × duration_s × price_per_gb_s plus the flat per-request charge: 1 × 2 × 0.0000166667 + 0.0000002 ≈ $0.00003353 per event.
The cluster is a fixed $1.50/hour regardless of throughput. The crossover is where hourly Lambda spend equals hourly cluster spend: cluster_hr / cost_per_event events per hour.
1.50 / 0.00003353 ≈ 44,700 events/hour, or about 12.4 events/second, sustained, before the always-on cluster wins on pure compute price.
But the crossover is not the whole story: below the crossover, Lambda also removes cluster idle time, patching, and scaling ops — real money that does not appear in the compute line. So the practical crossover sits higher than the raw compute crossover.
Above the crossover — a steady firehose that keeps hundreds of Lambdas busy 24/7 — the fleet is both pricier and harder to reason about than a provisioned engine. That steady-high-throughput shape is the classic "you outgrew Lambda" signal.

Output.

Metric	Value
Cost per event	~$0.00003353
Crossover rate	~44,700 events/hour
Crossover rate	~12.4 events/second
Below crossover	Lambda cheaper (and no idle)
Far above crossover	Cluster cheaper; consider Glue/EMR

Rule of thumb. Compute the crossover before you commit. Bursty and low-to-medium steady rates favour Lambda decisively (you pay nothing at 3 AM); a sustained firehose well above the crossover favours a provisioned engine. Always add the operational savings of "no cluster to run" on Lambda's side of the ledger.

Worked example — the "10 GB file" streaming trap

Detailed explanation. A frequent failure is loading a whole large object into memory inside a Lambda handler — body.read() on a multi-gigabyte S3 object. It works in dev on a 20 MB sample and blows the memory ceiling (or the time budget) in production on a 4 GB file. The fix is to stream and transform line-by-line so memory stays flat regardless of object size, and to split genuinely huge objects upstream. Walk through the trap and the streaming fix.

The trap. obj["Body"].read() pulls the entire object into RAM; a 4 GB file needs >4 GB memory and risks the time wall on parse.
The fix. Iterate the streaming body in chunks/lines, transform each, and write out incrementally — constant memory.
The guard. If a single object can exceed what one invocation can stream in ~14 minutes, split it upstream (or fan out by byte-range).

Question. Rewrite a load-everything handler into a streaming transform that holds constant memory for any object size within the time budget.

Input.

Component	Before	After
Read strategy	`Body.read()` (whole file)	iterate `Body` line-by-line
Peak memory	O(file size)	O(one line + buffer)
Max safe object	~memory ceiling	~what streams in 14 min
Failure mode	OOM / timeout	none (constant memory)

Code.

# streaming_transform.py — constant-memory S3 CSV -> JSONL transform
import boto3, io, json, csv

s3 = boto3.client("s3")

def handler(event, context):
    rec    = event["Records"][0]["s3"]
    bucket = rec["bucket"]["name"]
    key    = rec["object"]["key"]

    src = s3.get_object(Bucket=bucket, Key=key)["Body"]        # streaming body
    reader = csv.DictReader(io.TextIOWrapper(src, encoding="utf-8"))

    out_key = key.rsplit(".", 1)[0] + ".jsonl"
    buf = io.BytesIO()
    n = 0
    for row in reader:                                          # one row at a time
        row["ingested_at"] = context.aws_request_id            # cheap enrichment
        buf.write((json.dumps(row) + "\n").encode("utf-8"))
        n += 1
        if buf.tell() > 8 * 1024 * 1024:                       # flush every 8 MB
            _flush(bucket, out_key, buf, part=n)
            buf = io.BytesIO()
    if buf.tell():
        _flush(bucket, out_key, buf, part=n)
    return {"rows": n, "out": out_key}

def _flush(bucket, key, buf, part):
    buf.seek(0)
    s3.upload_fileobj(buf, bucket, f"{key}.part-{part}")       # incremental write

Step-by-step explanation.

get_object(...)["Body"] returns a streaming body, not bytes. Wrapping it in TextIOWrapper and handing it to csv.DictReader means rows are pulled lazily — memory stays at one row plus the output buffer, no matter how big the object is.
The transform enriches each row cheaply (here, stamping the request id) and writes to an in-memory buffer that is flushed to S3 every 8 MB. Flushing incrementally keeps peak memory bounded and avoids holding the whole output in RAM.
Because nothing scales with file size, a 50 MB file and a 4 GB file use the same memory. The only remaining limit is time: whether the object can be fully streamed and written inside the fifteen-minute wall.
If a single object is so large it cannot stream in ~14 minutes, no in-handler trick saves you — you split the object upstream (smaller exports) or fan out by S3 byte-range so each invocation handles a slice. That is a design change, not a code change.
The anti-pattern to unlearn is Body.read() (or pandas.read_csv(whole_object)) inside a Lambda for unbounded input. It is the number-one cause of "works in dev, OOMs in prod" serverless-ETL incidents.

Output.

Object size	`Body.read()` handler	streaming handler
20 MB	fine	fine
500 MB	needs ≥512 MB mem	flat ~128 MB mem
4 GB	OOM / timeout	flat mem; time-bound only
20 GB	impossible	split upstream / byte-range fan-out

Rule of thumb. In a Lambda ETL handler, never materialise an unbounded input in memory. Stream the input, transform per record, and flush the output incrementally so memory is O(1) in object size. When even streaming can't finish in the time budget, that is your signal to split upstream or fan out — not to raise the memory dial to 10 GB and hope.

Data engineering interview question on Lambda ETL fit

A senior interviewer might ask: "A team wants to move their hourly Airflow batch — which reads new files from S3, converts them to Parquet, and loads a warehouse — onto AWS Lambda so they 'stop paying for idle EC2.' Some of the incoming files are 30 MB, but a few are 8 GB. Walk me through how you'd decide what belongs on Lambda, what doesn't, and how you'd keep the design correct under at-least-once delivery."

Solution Using a per-object fit split with a size-based routing function

# router.py — S3 event -> route each object to the right engine by size
import boto3
s3  = boto3.client("s3")
sfn = boto3.client("stepfunctions")

SMALL_LIMIT_BYTES = 1_500_000_000    # ~1.5 GB: safely streamable in one Lambda
BIG_STATE_MACHINE = "arn:aws:states:...:stateMachine:big-file-etl"

def handler(event, context):
    routed = []
    for record in event["Records"]:                     # S3 can batch notifications
        b = record["s3"]["bucket"]["name"]
        k = record["s3"]["object"]["key"]
        size = record["s3"]["object"]["size"]

        if size <= SMALL_LIMIT_BYTES:
            transform_in_lambda(b, k)                    # short, streamable
            routed.append((k, "lambda"))
        else:
            sfn.start_execution(                         # hand big file to Step Functions
                stateMachineArn=BIG_STATE_MACHINE,
                name=f"etl-{k.replace('/', '_')}-{record['s3']['object']['eTag']}",
                input='{"bucket": "%s", "key": "%s"}' % (b, k),
            )
            routed.append((k, "stepfunctions"))
    return {"routed": routed}

def transform_in_lambda(bucket, key):
    # constant-memory streaming transform (see streaming_transform.py)
    ...

Step-by-step trace.

Step	Input (S3 event)	Action
1	`orders/2026-09-05/a.csv` size 30 MB	≤ 1.5 GB → transform in Lambda
2	`orders/2026-09-05/b.csv` size 45 MB	≤ 1.5 GB → transform in Lambda
3	`orders/2026-09-05/huge.csv` size 8 GB	> 1.5 GB → start Step Functions execution
4	duplicate delivery of `a.csv`	eTag-named execution / idempotent write dedupes
5	batch of 3 records in one event	loop routes each independently

Walking the trace: the router is itself a tiny Lambda triggered by S3. For each object in the (possibly batched) notification, it reads only the metadata — crucially the size field S3 includes in the event — and routes. Small objects are transformed inline by a streaming handler; the 8 GB object is handed to a Step Functions state machine that can run a chunked or Glue-backed job well past fifteen minutes. Because S3 delivery is at-least-once, a duplicate notification for a.csv must not double-load: the execution name is derived from the object eTag, so a repeat start_execution collides on the name and is rejected, and the Lambda transform path writes idempotently by object key + eTag.

Output:

Object	Size	Engine	Duplicate-safe by
a.csv	30 MB	Lambda stream	key + eTag write
b.csv	45 MB	Lambda stream	key + eTag write
huge.csv	8 GB	Step Functions	eTag execution name
a.csv (retry)	30 MB	Lambda stream	idempotent overwrite

Why this works — concept by concept:

Per-object routing — the fit decision is made per event, not per pipeline. Small objects ride the cheap Lambda path; oversized objects are handed off. One S3 trigger serves both, so the team keeps the "no idle EC2" win without pretending an 8 GB file fits in fifteen minutes.
Metadata-only decision — the router reads object.size from the event and never downloads the object to decide. The routing Lambda is milliseconds long and megabytes-free, so it scales trivially with the notification rate.
Fifteen-minute wall respected — anything that might exceed the wall is escalated to Step Functions before a single byte is read, rather than discovered mid-transform when the invocation is killed at minute fifteen.
At-least-once safety — the eTag-derived execution name and the idempotent write make duplicate S3 notifications harmless. This is the non-negotiable half of any event-driven ETL design; a router that double-loads on retry is a correctness bug, not a performance one.
Cost — the router is O(1) per event (metadata only), the streaming transform is O(rows) with O(1) memory, and the Step Functions path is used only for the rare oversized object. Compared with a warm EC2 batch, idle cost drops to zero; compared with "Lambda everything," you never hit the wall. Net O(1) idle spend, O(rows) transform.

ETL
Topic — etl
ETL problems on serverless ingestion pipelines

Practice →

Data Topic — data-processing Data-processing problems on streaming transforms

Practice →

2. Event sources & triggers — S3 / SQS / Kinesis

S3 pushes, SQS and Kinesis are polled — the event source mapping is the knob you tune

The mental model in one line: an event-driven Lambda is only as good as its trigger, and triggers split into two families — S3 (and SNS, EventBridge) push an event straight at your function one-at-a-time-ish, while SQS and Kinesis (and DynamoDB Streams) are polled by an AWS-managed component called the event source mapping that batches records and invokes your function with a batch — so the batch size, batch window, concurrency, and failure behaviour you configure on that mapping decide throughput, latency, ordering, and how a single bad record affects the rest. Every senior serverless-ETL design lives or dies on getting the event source mapping knobs right; "it just triggers" is the answer that fails the interview.

Push versus poll — the two trigger families.

Push sources (S3, SNS, EventBridge). The service invokes Lambda asynchronously: it hands the event to Lambda's internal queue and returns. Lambda retries async failures twice and can route them to a DLQ / destination. You do not configure batch size for S3; each notification carries one (occasionally a few) records.
Poll sources (SQS, Kinesis, DynamoDB Streams). The event source mapping — an AWS-managed poller you attach to the function — reads records, forms a batch, and invokes Lambda synchronously with that batch. Throughput, latency, and failure semantics are all governed by the mapping's config, not by the function.
Why it matters. Push sources give you low config and per-event fan-in; poll sources give you batching (fewer invocations, more efficiency) but require you to handle a batch correctly, including partial failure.

S3 event notifications — the object-created trigger.

What fires. s3:ObjectCreated:* (Put, Post, CompleteMultipartUpload, Copy) and delete/restore events. You wire the bucket notification to invoke the Lambda.
Filters. Prefix and suffix filters scope the trigger — e.g. prefix raw/ and suffix .csv so only raw CSVs invoke the function. Filtering at the source is cheaper than filtering in code.
Delivery. At-least-once and not strictly ordered. A single upload occasionally yields more than one notification, and notifications can arrive out of order. Design the handler to be idempotent and order-independent.
The multipart gotcha. A large multipart upload fires CompleteMultipartUpload, not Put; scope your event types so you don't miss big files.

SQS as an event source — batching and partial failure.

Batch size & window. The mapping reads up to batchSize messages (max 10 000 for standard queues via the mapping, commonly 10) or waits up to maximumBatchingWindowInSeconds to fill a batch. Bigger batches = fewer invocations, higher per-invocation work.
Visibility timeout. While a batch is in flight, its messages are invisible. The queue visibility timeout must be ≥ the function timeout (a common rule: 6× the function timeout) or messages reappear and get double-processed.
Partial batch response. By default, if the handler throws, the whole batch returns to the queue and is retried — reprocessing the messages that already succeeded. Enabling ReportBatchItemFailures lets the handler return only the failed message IDs, so successes are not redelivered.
FIFO queues. Preserve order within a message group and give exactly-once processing only if you also dedupe; standard queues are at-least-once and unordered.

Kinesis as an event source — shards and ordering.

Shards = parallelism. A stream has N shards; the mapping runs (by default) one concurrent invocation per shard, preserving order within a shard. The partition key decides which shard a record lands on — same key, same shard, ordered.
Parallelization factor. You can raise concurrency to up to 10 invocations per shard while still preserving per-partition-key order; this lifts throughput on hot shards.
Batch size & iterator. The mapping reads up to batchSize records per shard per invocation, starting at TRIM_HORIZON (oldest) or LATEST. A batch spans one shard.
Poison-pill risk. A record the handler always fails on will block its shard forever unless you cap retries (maximumRetryAttempts), cap record age (maximumRecordAgeInSeconds), split the batch on error (bisectBatchOnFunctionError), and route failures to an on-failure destination.

Common interview probes on triggers.

"How does an S3 trigger deliver — once or at-least-once?" — at-least-once, unordered; handler must be idempotent.
"Why set visibility timeout ≥ function timeout on SQS?" — else in-flight messages reappear and double-process.
"What is partial batch response and why enable it?" — return only failed IDs so successes aren't retried.
"What guarantees ordering in Kinesis?" — per-shard order via the partition key; one invocation per shard by default.

Worked example — S3 object-created trigger to a curated Parquet copy

Detailed explanation. The canonical push trigger: a CSV lands under raw/, Lambda converts it to Parquet under curated/, idempotently. The bucket notification is scoped with a prefix/suffix filter so only the right objects fire. Build the trigger config and the handler.

Filter. Prefix raw/, suffix .csv — nothing else invokes the function.
Idempotency. Output key is derived from input key; a re-delivered notification overwrites the same Parquet object (harmless).
Events. Include Put and CompleteMultipartUpload so large uploads are not missed.

Question. Configure the S3 notification and write the idempotent transform handler.

Input.

Setting	Value
Bucket	`data-lake-prod`
Trigger events	`ObjectCreated:Put`, `ObjectCreated:CompleteMultipartUpload`
Filter	prefix `raw/`, suffix `.csv`
Output	`curated/{same-stem}.parquet`

Code.

// S3 bucket notification configuration (LambdaFunctionConfigurations)
{
  "LambdaFunctionConfigurations": [
    {
      "LambdaFunctionArn": "arn:aws:lambda:...:function:csv-to-parquet",
      "Events": ["s3:ObjectCreated:Put", "s3:ObjectCreated:CompleteMultipartUpload"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {"Name": "prefix", "Value": "raw/"},
            {"Name": "suffix", "Value": ".csv"}
          ]
        }
      }
    }
  ]
}

# handler.py — idempotent CSV -> Parquet on S3 object-created
import boto3, io
import pyarrow as pa
import pyarrow.csv as pacsv
import pyarrow.parquet as pq

s3 = boto3.client("s3")

def handler(event, context):
    results = []
    for rec in event["Records"]:                       # S3 may batch a few records
        bucket = rec["s3"]["bucket"]["name"]
        key    = rec["s3"]["object"]["key"]
        if not key.startswith("raw/") or not key.endswith(".csv"):
            continue                                    # defense in depth vs filter

        body = s3.get_object(Bucket=bucket, Key=key)["Body"].read()
        table = pacsv.read_csv(io.BytesIO(body))        # small files only (see §1 trap)

        out_key = "curated/" + key[len("raw/"):-len(".csv")] + ".parquet"
        buf = io.BytesIO()
        pq.write_table(table, buf)
        buf.seek(0)
        # Deterministic key => re-delivery overwrites the same object (idempotent)
        s3.put_object(Bucket=bucket, Key=out_key, Body=buf.getvalue())
        results.append(out_key)
    return {"written": results}

Step-by-step explanation.

The notification config scopes the trigger at the source: only objects under raw/ ending in .csv invoke Lambda. This is cheaper and cleaner than invoking on every object and filtering in code — though the handler still double-checks (defense in depth).
Including CompleteMultipartUpload in the event list is what makes large uploads reliable — the AWS SDK uploads big files as multipart, which fires that event, not Put. Miss it and your biggest files silently never trigger.
The handler loops over event["Records"] because a single notification can carry more than one record. Assuming exactly one record is a latent bug that surfaces under load.
The output key is derived deterministically from the input key. Because S3 delivery is at-least-once, the same object may fire twice; writing to the same deterministic key means the second write simply overwrites identical bytes — idempotent by construction, no dedupe table needed.
This handler uses read_csv on the whole body, which is only safe because the filter/design guarantees small raw/ files. For unbounded input you'd switch to the streaming pattern from section 1 — the trigger config is identical.

Output.

Event	Input key	Output key
Put	`raw/2026/09/a.csv`	`curated/2026/09/a.parquet`
CompleteMultipartUpload	`raw/2026/09/big.csv`	`curated/2026/09/big.parquet`
Duplicate Put	`raw/2026/09/a.csv`	`curated/2026/09/a.parquet` (overwrite)
Non-matching	`logs/x.json`	(never fires; filtered)

Rule of thumb. Scope S3 triggers with prefix/suffix filters, always include CompleteMultipartUpload for large files, loop over all records in the event, and derive the output key deterministically so at-least-once delivery is harmless. The trigger config is where correctness starts, before any transform code runs.

Worked example — SQS batch consumer with partial batch response

Detailed explanation. An SQS-triggered Lambda receives a batch of up to 10 messages. If one message's transform fails and the handler throws, the entire batch is retried by default — re-processing the 9 that succeeded. Enabling ReportBatchItemFailures and returning the failed IDs fixes this. Build the consumer.

Mapping config. batchSize: 10, functionResponseTypes: ["ReportBatchItemFailures"].
Visibility timeout. Set on the queue to ≥ 6× the function timeout so in-flight batches don't reappear.
Contract. The handler returns {"batchItemFailures": [{"itemIdentifier": messageId}, ...]} for only the messages that failed.

Question. Write an SQS consumer that processes each message independently and reports only the failures.

Input.

Setting	Value
Queue	`orders-ingest`
Batch size	10
Function timeout	30 s
Visibility timeout	180 s (6×)
Response type	`ReportBatchItemFailures`

Code.

# sqs_consumer.py — process each message; report only failures
import json

def handler(event, context):
    failures = []
    for msg in event["Records"]:
        try:
            body = json.loads(msg["body"])
            process_order(body)                      # your transform / upsert
        except Exception as e:
            # Do NOT re-raise: that would fail the WHOLE batch
            print(f"failed messageId={msg['messageId']}: {e}")
            failures.append({"itemIdentifier": msg["messageId"]})
    # Only these IDs return to the queue for retry; the rest are deleted
    return {"batchItemFailures": failures}

def process_order(order):
    if order.get("total_cents") is None:             # a poison-pill example
        raise ValueError("missing total_cents")
    # ... idempotent upsert into the warehouse ...

Step-by-step explanation.

The mapping is configured with functionResponseTypes = ["ReportBatchItemFailures"]. This changes the contract: instead of "throw to fail the batch," the handler returns a list of the message IDs that failed.
The loop processes each message inside its own try/except. Critically, the handler never re-raises — re-raising would signal batch-level failure and force SQS to redeliver all 10 messages, double-processing the successful ones.
Successful messages are implicitly acknowledged (SQS deletes them) because they are not in the returned batchItemFailures. Failed messages are returned by ID and become visible again after the visibility timeout for another attempt.
The queue's visibility timeout (180 s) is set to well above the function timeout (30 s). If it were lower, a slow batch's messages would reappear while the batch was still processing — the classic double-processing bug. The 6× rule bakes in headroom for retries within Lambda.
A message that fails every time (a poison pill — here, missing total_cents) will retry until the queue's maxReceiveCount redrive policy ships it to a DLQ. Partial batch response ensures that poison pill never drags its 9 healthy batch-mates down with it.

Output.

Message	Result	Returned in batchItemFailures?
m1 (valid)	processed, deleted	no
m2 (missing total)	raised → recorded	yes
m3..m10 (valid)	processed, deleted	no
batch outcome	9 done, 1 retried	only m2 redelivered

Rule of thumb. For any SQS-triggered Lambda, enable ReportBatchItemFailures, catch per-message inside the loop, never re-raise, and return only the failed IDs. Pair it with a queue visibility timeout ≥ 6× the function timeout and a redrive policy to a DLQ. This is the difference between "one bad message poisons the batch" and "one bad message is quarantined."

Worked example — Kinesis shard consumer with ordering and parallelization

Detailed explanation. A Kinesis-triggered Lambda reads a batch of records from a single shard, in order. Records with the same partition key always land on the same shard, so per-key ordering holds. The parallelizationFactor lifts throughput on hot shards without breaking per-key order. Build the consumer.

Ordering. Guaranteed within a shard; the partition key routes records to shards.
Parallelization factor. Up to 10 concurrent batches per shard, still ordered per partition key.
Checkpointing. The mapping advances the shard iterator only after a successful invocation (or per partial-failure config).

Question. Write a Kinesis consumer that transforms records preserving per-key order and explain how parallelization factor keeps order.

Input.

Setting	Value
Stream	`clickstream`
Shards	8
Batch size	500
Starting position	TRIM_HORIZON
Parallelization factor	4

Code.

# kinesis_consumer.py — ordered per-key transform of a shard batch
import base64, json

def handler(event, context):
    failures = []
    for rec in event["Records"]:                          # all from ONE shard, in order
        try:
            payload = base64.b64decode(rec["kinesis"]["data"])
            click = json.loads(payload)
            # partition key == user_id => all of a user's events are ordered here
            transform_click(click, seq=rec["kinesis"]["sequenceNumber"])
        except Exception as e:
            print(f"failed seq={rec['kinesis']['sequenceNumber']}: {e}")
            failures.append({"itemIdentifier": rec["kinesis"]["sequenceNumber"]})
    return {"batchItemFailures": failures}                 # ReportBatchItemFailures on the ESM

def transform_click(click, seq):
    # idempotent write keyed by (user_id, event_ts, seq)
    ...

Step-by-step explanation.

Every record in event["Records"] comes from one shard and is delivered in sequence-number order. The handler can therefore rely on order within the batch — e.g. to fold a user's clicks in the order they happened.
The Kinesis partition key (here user_id) is hashed to a shard. All of one user's events hit the same shard, so per-user ordering is preserved end to end even across many shards and many concurrent invocations.
parallelizationFactor: 4 lets up to 4 batches from the same shard run concurrently — but Kinesis still guarantees that records sharing a partition key are processed in order, because it groups by key across those concurrent batches. You get more throughput on a hot shard without sacrificing per-key ordering.
The consumer uses partial batch response too (returning failed sequenceNumbers). Combined with bisectBatchOnFunctionError on the mapping, a single poison record is isolated rather than blocking the shard — covered in section 4.
Checkpointing is automatic: the mapping advances the shard iterator past the batch only when the invocation succeeds (or past the successful prefix under partial-failure config). A thrown, unhandled error would re-read the same batch — the reason idempotency is mandatory.

Output.

Aspect	Behaviour
Order within shard	preserved (sequence order)
Order across shards	none (independent)
Same partition key	always same shard, ordered
parallelizationFactor 4	4× throughput, key order kept
Failed record	returned by sequenceNumber

Rule of thumb. Treat a Kinesis batch as an ordered, single-shard slice. Route related records with a stable partition key so per-key order holds, lift throughput with parallelizationFactor (not more shards, when the shard isn't the bottleneck), and always process idempotently because a failure re-reads the batch. Ordering is a partition-key property, not a wish.

Data engineering interview question on event source mappings

A senior interviewer might ask: "You have an SQS queue feeding a Lambda that upserts events into Redshift. Under a traffic spike, you see the same event applied twice and occasional 'message became visible again mid-processing' warnings, and one malformed message keeps failing the whole batch. Design the event source mapping and handler so successes aren't retried, in-flight messages don't reappear, and the poison message is quarantined."

Solution Using ReportBatchItemFailures with a tuned visibility timeout and DLQ redrive

# robust_sqs_consumer.py
import json

def handler(event, context):
    failures = []
    for msg in event["Records"]:
        mid = msg["messageId"]
        try:
            body = json.loads(msg["body"])
            upsert_idempotent(body, dedupe_key=body["event_id"])   # safe under retry
        except Exception as e:
            print(f"quarantine-candidate {mid}: {e}")
            failures.append({"itemIdentifier": mid})
    return {"batchItemFailures": failures}

def upsert_idempotent(body, dedupe_key):
    # INSERT ... ON CONFLICT (event_id) DO UPDATE — Redshift/Postgres upsert
    ...

# Event source mapping + queue configuration (conceptual)
SQS queue: orders-ingest
  VisibilityTimeout          = 180        # >= 6 x function timeout (30s)
  RedrivePolicy:
    maxReceiveCount          = 5          # after 5 tries -> DLQ
    deadLetterTargetArn      = orders-ingest-dlq

Event source mapping (queue -> lambda):
  BatchSize                  = 10
  MaximumBatchingWindowInSec = 5
  FunctionResponseTypes      = [ReportBatchItemFailures]
  ScalingConfig.MaximumConcurrency = 20   # cap concurrency to protect Redshift

Step-by-step trace.

Step	Condition	Behaviour
1	batch of 10 arrives	handler processes each in its own try/except
2	m4 malformed, throws	recorded in batchItemFailures; not re-raised
3	m1–m3, m5–m10 succeed	deleted from queue (not in failures list)
4	m4 returned by ID	reappears after 180 s visibility timeout
5	m4 fails 5 times	redrive policy ships it to `orders-ingest-dlq`
6	duplicate of m1 delivered	`event_id` conflict → upsert is a no-op

Walking the trace: the visibility timeout of 180 s (6× the 30 s function timeout) means a batch stays invisible for the whole time it could plausibly be processing and retrying, so the "became visible mid-processing" double-apply disappears. Partial batch response deletes the 9 healthy messages and only returns the malformed one, so successes are never reprocessed. The malformed message retries up to maxReceiveCount = 5 and then lands in the DLQ, quarantined for a human — the batch is never blocked. And because the upsert dedupes on event_id, even a genuine at-least-once duplicate is a harmless no-op.

Output:

Symptom before	Fix	Result
Event applied twice	idempotent upsert on `event_id`	duplicate is a no-op
Message reappears mid-processing	visibility timeout 180 s (6×)	no premature redelivery
One message fails whole batch	ReportBatchItemFailures	only that ID retried
Poison message loops forever	redrive maxReceiveCount 5 → DLQ	quarantined after 5 tries
Spike stampedes Redshift	ScalingConfig max concurrency 20	bounded parallel upserts

Why this works — concept by concept:

Partial batch response — returning batchItemFailures instead of throwing tells SQS to delete the successes and redeliver only the failed IDs. It converts "one bad message poisons ten" into "one bad message retries alone."
Visibility timeout ≥ 6× function timeout — a batch stays invisible for the entire window it could be processing plus retries, which removes the race where a message reappears and a second invocation double-applies it.
Idempotent upsert on a dedupe key — because delivery is at-least-once, correctness cannot depend on "exactly once." The ON CONFLICT upsert makes any duplicate a no-op, so retries and redeliveries are safe.
Redrive policy to a DLQ — a poison message is bounded to maxReceiveCount attempts and then parked in a dead-letter queue for inspection, so it never blocks the pipeline indefinitely.
Cost — O(batch) per invocation with O(1) dedupe lookups; capped concurrency bounds downstream load. Compared with the naive "throw on any error" consumer, this reprocesses only failures (not whole batches), so wasted re-work drops from O(batch × retries) to O(failures × retries).

Event
Topic — event-processing
Event-processing problems on queue and stream consumers

Practice →

ETL Topic — etl ETL problems on batch trigger design

Practice →

3. Concurrency, memory & cold starts

One sandbox per event — concurrency, memory, and cold starts are the three dials that decide throughput, latency, and blast radius

The mental model in one line: Lambda scales by running one isolated execution environment per concurrent event, so your throughput is concurrency = arrival_rate × average_duration, your per-invocation speed is set by the memory dial (more memory buys proportionally more CPU), and your tail latency is set by cold starts (the one-time init cost paid whenever a new sandbox is created) — and the three dials interact, because raising concurrency can stampede a downstream, capping it can throttle the source, and a big deployment package or a VPC attachment lengthens every cold start. Senior serverless-ETL tuning is choosing reserved concurrency to bound blast radius, provisioned concurrency to kill cold starts where latency matters, and the memory setting that minimises cost-per-invocation.

The concurrency model — three kinds of concurrency.

Unreserved (on-demand) concurrency. By default all functions in an account share a regional pool (commonly 1 000, raisable). A burst spins up sandboxes until the pool or the burst rate limit is hit, then throttles with 429 TooManyRequestsException.
Reserved concurrency. A cap you set per function. It does two things at once: it guarantees that function at least that many slots, and it limits it to at most that many — protecting downstreams and protecting other functions from being starved by this one.
Provisioned concurrency. Pre-initialised sandboxes kept warm so there is no cold start for the first N concurrent requests. You pay for them whether used or not; you use it where p99 latency matters (synchronous APIs, latency-sensitive steps), rarely for pure batch ETL.
The formula. Steady-state concurrency ≈ requests_per_second × average_duration_seconds. 100 rps × 0.2 s = 20 concurrent sandboxes. This is the number you reserve around.

Cold start anatomy — what the first invocation pays.

Init phase. Creating a new sandbox downloads your code/layers, starts the runtime, and runs your module-level (init) code before the handler. That one-time cost is the cold start; subsequent invocations on the same warm sandbox skip it.
What lengthens it. A large deployment package or many layers; heavy import-time work (loading a big model, opening connections at import); a VPC attachment (historically ENI setup, now much faster but still non-zero); and runtime choice (interpreted vs compiled startup).
What shortens it. Smaller packages, lazy imports, moving one-time setup into init and reusing it across invocations, and provisioned concurrency for the paths that can't tolerate the penalty.
Warm reuse. A sandbox is reused for subsequent events; connections, clients, and cached data created at init are reused — which is exactly why you create the DB client once at module scope, not per invocation.

Memory equals CPU — the power dial.

The coupling. Memory is the only performance dial; CPU (and network) scale proportionally with it. At ~1 769 MB you get roughly one full vCPU; more memory gives more vCPUs.
The counter-intuitive part. Raising memory can lower cost: if doubling memory more than halves duration (CPU-bound work), the GB-second product drops. Power-tuning finds the sweet spot.
The method. Sweep memory settings, measure duration and cost per setting, pick the minimum-cost (or minimum-latency) point. AWS Lambda Power Tuning automates this sweep.

Throttling and back-pressure.

Throttling. When concurrency hits the cap, sync invocations get 429; async invocations are retried internally; poll-based mappings slow their polling (natural back-pressure onto SQS/Kinesis).
Back-pressure as a feature. Reserving low concurrency on a Lambda that writes to a small RDS instance means SQS simply holds messages when the cap is hit, smoothing the load instead of overwhelming the database.
Account blast radius. Without per-function reserved concurrency, one runaway function can consume the whole account pool and throttle everything else. Reserve critical functions.

Common interview probes on concurrency.

"How does Lambda scale?" — one concurrent sandbox per in-flight event; concurrency = rate × duration.
"Reserved vs provisioned concurrency?" — reserved caps/guarantees slot count; provisioned pre-warms sandboxes to remove cold starts.
"How do you make Lambda go faster?" — raise the memory dial (more CPU); it can even lower cost.
"How do you stop Lambda from overwhelming a database?" — reserved concurrency cap → SQS back-pressure.

Worked example — reserved concurrency to protect a downstream database

Detailed explanation. A Lambda upserts into a small RDS Postgres that tolerates ~50 concurrent connections. Under a spike, unbounded Lambda concurrency opens hundreds of connections and topples the database. Reserved concurrency caps the function so it can never exceed the DB's connection budget; the SQS source absorbs the overflow. Build it.

DB budget. ~50 safe concurrent connections.
Cap. Reserve 40 concurrency on the function (headroom below 50).
Overflow. SQS holds messages; the mapping polls only as fast as slots free up.

Question. Choose a reserved concurrency value and explain the back-pressure behaviour under a 10× spike.

Input.

Parameter	Value
Safe DB connections	50
Reserved concurrency	40
Connections per invocation	1 (pooled/reused)
Source	SQS (poll)
Spike	10× normal arrival

Code.

# db_writer.py — one pooled connection per warm sandbox
import os, psycopg2

# Created ONCE at init and reused across invocations on this warm sandbox.
_conn = psycopg2.connect(os.environ["DB_DSN"])

def handler(event, context):
    with _conn.cursor() as cur:
        for msg in event["Records"]:
            cur.execute(
                "INSERT INTO orders(id, total_cents) VALUES (%s, %s) "
                "ON CONFLICT (id) DO UPDATE SET total_cents = EXCLUDED.total_cents",
                (msg_id(msg), total(msg)),
            )
    _conn.commit()
    return {"ok": True}

# Reserved concurrency (set on the function)
aws lambda put-function-concurrency \
  --function-name db-writer \
  --reserved-concurrent-executions 40

Step-by-step explanation.

The database connection is created at module scope (init), so each warm sandbox holds exactly one reused connection. With reserved concurrency 40, at most 40 sandboxes exist, so at most 40 DB connections — comfortably under the 50 budget.
put-function-concurrency 40 both guarantees and caps: the function always has up to 40 slots and can never exceed them. This is the hard ceiling that protects the database.
Under a 10× spike, arrivals exceed 40 concurrent capacity. Because the source is SQS (poll-based), the event source mapping simply stops pulling faster than slots free up — messages stay in the queue, visible-timeout-protected, and drain as invocations complete. This is back-pressure, not failure.
Had the cap been absent, Lambda would have scaled to hundreds of sandboxes, opened hundreds of connections, and exhausted the DB's connection limit — a cascading outage. The cap trades a little latency (queue depth grows briefly) for stability.
The one subtlety: connection reuse depends on the sandbox staying warm. On a cold start a fresh connection is opened, which is why 40 (not 50) leaves headroom for the brief overlap while old sandboxes drain and new ones warm.

Output.

Load	Concurrency	DB connections	Queue depth
Normal	~8	~8	~0
10× spike	capped at 40	≤ 40	grows, then drains
No cap (hypothetical)	400+	400+ → DB down	0 (but outage)
After spike	falls to ~8	~8	back to ~0

Rule of thumb. When a Lambda writes to a capacity-limited downstream, set reserved concurrency below the downstream's safe limit and let a poll-based source (SQS/Kinesis) absorb overflow as back-pressure. Create the connection once at init and reuse it. Concurrency is your rate-limiter; the queue is your shock absorber.

Worked example — provisioned concurrency for a latency-sensitive step

Detailed explanation. A Lambda sits on a synchronous path where p99 latency matters (an enrichment call in a request flow). Cold starts add hundreds of milliseconds to the first request after idle. Provisioned concurrency keeps N sandboxes pre-initialised so those requests never pay init cost. Build it and reason about the trade-off.

Problem. p99 spikes to ~800 ms on cold starts vs ~40 ms warm.
Fix. Provision 10 warm sandboxes for the expected concurrent load.
Trade-off. You pay for provisioned concurrency whether used or not; size it to steady-state, not peak.

Question. Configure provisioned concurrency and explain when it is (and isn't) worth it.

Input.

Parameter	Value
Warm duration	~40 ms
Cold start penalty	~760 ms
Steady concurrency	~10
Provisioned concurrency	10
Path	synchronous enrichment

Code.

# Provisioned concurrency on a published version/alias
aws lambda put-provisioned-concurrency-config \
  --function-name enrich \
  --qualifier prod \
  --provisioned-concurrent-executions 10

# Optional: autoscale provisioned concurrency on a schedule / utilization
aws application-autoscaling register-scalable-target \
  --service-namespace lambda \
  --resource-id function:enrich:prod \
  --scalable-dimension lambda:function:ProvisionedConcurrency \
  --min-capacity 5 --max-capacity 50

# enrich.py — heavy init done ONCE, reused by every warm/provisioned sandbox
import json
MODEL = load_lookup_table()          # expensive; runs during init, not per request

def handler(event, context):
    key = event["key"]
    return {"enriched": MODEL.get(key, "unknown")}   # ~40 ms, no init cost

Step-by-step explanation.

Provisioned concurrency is attached to a published version/alias (prod), not $LATEST. AWS keeps 10 sandboxes fully initialised — init code already run — so the first 10 concurrent requests hit warm environments and skip the ~760 ms penalty.
The expensive load_lookup_table() runs during init, once per sandbox. With provisioned concurrency, that init happens ahead of traffic, so no request ever waits for it. This is the whole point: move the cost off the request path.
Sizing is to steady-state concurrency (~10), not peak. Beyond the provisioned count, extra load spills to on-demand sandboxes that do pay cold start — acceptable for the rare overflow, and you can autoscale provisioned concurrency to track it.
The trade-off is money: provisioned concurrency bills for the warm sandboxes continuously. It is worth it on latency-sensitive synchronous paths; it is usually not worth it for pure asynchronous batch ETL, where a few hundred ms of cold start on a multi-second job is irrelevant.
This is why most ETL Lambdas skip provisioned concurrency entirely: batch jobs care about throughput and cost, not tail latency. Reserve provisioned concurrency for the request-flow steps where p99 is a user-facing SLO.

Output.

Metric	On-demand only	Provisioned (10)
p50 latency	~40 ms	~40 ms
p99 latency (idle→spike)	~800 ms	~40 ms
Cost when idle	~$0	pay for 10 warm
Best for	batch ETL	latency-sensitive sync

Rule of thumb. Use provisioned concurrency only where cold-start tail latency is a user-facing SLO, size it to steady-state concurrency, and keep expensive setup in init so the pre-warmed sandboxes actually absorb it. For batch ETL, a cold start is a rounding error — don't pay to remove it.

Worked example — memory power-tuning to minimise cost

Detailed explanation. Because CPU scales with memory, a CPU-bound transform can get cheaper at higher memory if the speedup outpaces the price increase. Sweep memory settings, measure duration and GB-second cost, and pick the minimum-cost point. Walk through a power-tuning sweep.

Workload. A CPU-bound Parquet compression step.
Sweep. 512 MB → 1024 → 1769 → 3008 MB.
Pick. The memory with the lowest duration × memory (cost), or lowest duration if latency-bound.

Question. Given a sweep, choose the cost-optimal memory setting.

Input.

Memory (MB)	Duration (ms)	GB-seconds
512	4200	2.10
1024	2100	2.10
1769	1150	1.98
3008	1000	2.94

Code.

# power_tune.py — pick the cheapest memory from a sweep
GB_S_PRICE = 0.0000166667
sweep = [
    (512,  4200),
    (1024, 2100),
    (1769, 1150),
    (3008, 1000),
]
def gb_seconds(mem_mb, dur_ms):
    return (mem_mb / 1024) * (dur_ms / 1000)

for mem, dur in sweep:
    gbs  = gb_seconds(mem, dur)
    cost = gbs * GB_S_PRICE
    print(f"{mem:>4} MB  {dur:>5} ms  {gbs:.2f} GB-s  ${cost:.8f}")

best = min(sweep, key=lambda md: gb_seconds(*md))
print("cheapest:", best[0], "MB")

Step-by-step explanation.

Cost is memory_gb × duration_s × price. At 512 MB the job is slow (4200 ms) and costs 2.10 GB-s; doubling to 1024 MB roughly halves duration, so cost is identical — proof the work is CPU-bound and scaling linearly with CPU.
At 1769 MB (≈ one full vCPU), duration drops to 1150 ms and GB-seconds fall to 1.98 — the minimum. Here the speedup slightly outpaces the price increase, so this is the cost-optimal point.
At 3008 MB the job is only marginally faster (1000 ms) but you pay for ~1.7× the memory, so GB-seconds jump to 2.94 — more expensive. Past the point where extra CPU stops helping, higher memory only raises cost.
The sweet spot is workload-specific: I/O-bound jobs barely speed up with memory (so the smallest memory that fits is cheapest), while CPU-bound jobs often minimise around one vCPU. You must measure, not guess.
If latency (not cost) is the objective, pick 3008 MB for its 1000 ms; if cost, pick 1769 MB. Power-tuning makes the trade explicit instead of leaving memory at a default 128 MB where CPU-bound jobs crawl.

Output.

Memory	Duration	GB-s	Verdict
512 MB	4200 ms	2.10	slow, same cost
1024 MB	2100 ms	2.10	faster, same cost
1769 MB	1150 ms	1.98	cost-optimal
3008 MB	1000 ms	2.94	fastest, pricier

Rule of thumb. Never leave a CPU-bound Lambda at 128 MB. Sweep memory, plot duration and GB-seconds, and pick the minimum-cost point (often near one vCPU) or the minimum-latency point if that's the objective. Memory is a performance dial, not just a capacity dial.

Data engineering interview question on concurrency and cold starts

A senior interviewer might ask: "Your S3-triggered transform Lambda writes to a Redshift cluster and a small RDS metadata DB. During a batch of 5 000 files landing at once, Redshift is fine but the RDS DB hits its connection limit and the whole pipeline stalls, and you also see elevated p99 from cold starts. Design the concurrency and connection strategy so the RDS DB is protected, throughput stays high, and cold starts stop hurting."

Solution Using reserved concurrency, connection reuse, and SQS back-pressure

# transform.py — S3 events buffered through SQS, capped concurrency, reused conn
import os, psycopg2, boto3

_rds = psycopg2.connect(os.environ["RDS_DSN"])   # once per warm sandbox
_redshift_data = boto3.client("redshift-data")

def handler(event, context):
    for msg in event["Records"]:                 # SQS batch (S3 -> SQS -> Lambda)
        obj = parse(msg["body"])
        rows = transform_object(obj)             # streaming, constant memory
        load_to_redshift(_redshift_data, rows)   # scales fine
        with _rds.cursor() as cur:               # metadata write: the bottleneck
            cur.execute(
                "INSERT INTO file_audit(key, rows) VALUES (%s, %s) "
                "ON CONFLICT (key) DO UPDATE SET rows = EXCLUDED.rows",
                (obj["key"], len(rows)),
            )
    _rds.commit()
    return {"ok": True}

# Wiring + limits
S3 (ObjectCreated) ---> SQS (buffer) ---> Lambda transform
  Reserved concurrency (transform)      = 40    # < RDS safe conn limit (50)
  SQS VisibilityTimeout                 = 180   # >= 6 x function timeout
  SQS RedrivePolicy.maxReceiveCount     = 5 -> DLQ
  RDS connection                        = created at init, reused
  Redshift writes                       = redshift-data API (no held conn)

Step-by-step trace.

Step	Event	Behaviour
1	5 000 files land	S3 fans notifications into an SQS buffer
2	mapping polls SQS	invokes transform up to reserved cap 40
3	40 sandboxes warm	≤ 40 RDS connections (reused per sandbox)
4	overflow	remaining messages wait in SQS (back-pressure)
5	cold starts	init opens RDS conn once; reused for the batch
6	queue drains	concurrency falls, connections released

Walking the trace: inserting an SQS buffer between S3 and the transform turns a 5 000-file thundering herd into a smoothly polled queue. Reserved concurrency 40 caps sandboxes below the RDS 50-connection limit, and because the connection is created at init and reused, 40 sandboxes mean at most 40 connections — the DB is safe. Redshift, which scales fine, is written via the connectionless redshift-data API so it never competes for the RDS budget. Cold starts still happen, but each sandbox pays init once and then processes many messages warm, so the amortised p99 impact across a 5 000-file batch is negligible — no provisioned concurrency needed for a batch job.

Output:

Symptom before	Fix	Result
RDS connection exhaustion	reserved concurrency 40 + reuse	≤ 40 connections
Thundering herd of 5 000	S3 → SQS buffer	smooth polling
Pipeline stalls	back-pressure in queue	drains, no failure
Cold-start p99	init once, reuse warm	amortised away
Redshift contention	redshift-data API	no held connections

Why this works — concept by concept:

Reserved concurrency as a rate limiter — capping the function below the RDS connection budget makes it impossible to exhaust the database, converting a spike into bounded parallelism.
Connection reuse at init — creating the connection at module scope means one connection per sandbox, reused across the batch, so concurrency (not invocation count) bounds connections.
SQS buffer for back-pressure — placing a queue between the bursty S3 source and the capped consumer absorbs the herd; the mapping polls only as fast as slots free, so overflow waits instead of failing.
Connectionless downstream for the scalable store — routing Redshift writes through the redshift-data API keeps the scalable warehouse off the scarce-connection path entirely.
Cold-start amortisation — for batch ETL, each warm sandbox handles many messages, so the one-time init cost is spread thin; provisioned concurrency would be wasted spend here.
Cost — O(files) work at O(cap) parallelism with O(cap) reused connections; the SQS buffer adds a cheap per-message poll. Compared with unbounded concurrency, this trades a little queue latency for a database that never falls over, at no extra compute cost.

Data
Topic — data-processing
Data-processing problems on parallel throughput tuning

Practice →

ETL Topic — etl ETL problems on concurrency and back-pressure

Practice →

4. Error handling — DLQ, retries, idempotency

At-least-once means retries are guaranteed — so handlers must be idempotent and poison pills must have somewhere to go

The mental model in one line: every Lambda event source delivers at-least-once and retries on failure, but how it retries depends on the invocation type — synchronous callers retry themselves, asynchronous sources (S3, SNS, EventBridge) retry twice then send to a dead-letter queue or on-failure destination, and poll sources (SQS, Kinesis) retry the batch until success, DLQ redrive, or an age/attempt cap — which means correct error handling is three disciplines working together: make the handler idempotent so retries are safe, cap and route failures so a poison pill lands in a DLQ instead of blocking forever, and bisect batches so one bad record doesn't sink its neighbours. Skip any one and an at-least-once world turns a transient blip into duplicated rows or a wedged partition.

Retry behaviour by invocation type.

Synchronous (API Gateway, direct invoke). No automatic Lambda retry — the caller decides whether to retry. Errors surface to the caller immediately.
Asynchronous (S3, SNS, EventBridge). Lambda's internal queue retries a failed invocation up to 2 more times (configurable) with delay, then routes the event to an on-failure destination (SQS/SNS/EventBridge/Lambda) or a legacy DLQ. The event is not lost — it is parked.
Poll-based (SQS, Kinesis, DynamoDB Streams). The event source mapping retries the batch. For SQS, the queue's redrive policy sends a message to a DLQ after maxReceiveCount. For streams, maximumRetryAttempts and maximumRecordAgeInSeconds cap retries and an on-failure destination captures the failed batch metadata.
The consequence. Because retries are guaranteed, the same event will be processed more than once at some point. Idempotency is not optional.

Dead-letter queues and destinations.

DLQ (legacy). An SQS queue or SNS topic attached to the function for async invocations; failed events land there after retries. Still supported but destinations are richer.
Lambda destinations. For async invocations, configure on-success and on-failure destinations that receive the full invocation record (request + response/error) — better than a bare DLQ because you get context.
SQS redrive. For SQS sources, the DLQ is a property of the queue (RedrivePolicy → deadLetterTargetArn), not the function. After maxReceiveCount failed receives, the message moves to the DLQ.
The DLQ is not a graveyard. It needs an alarm (messages > 0) and a redrive/replay path once the bug is fixed. An unwatched DLQ silently loses data to expiry.

Idempotency — the correctness backbone.

Why. At-least-once + retries = duplicates. A non-idempotent write (blind INSERT, += amount) double-applies on retry; an idempotent write is a no-op the second time.
Techniques. Deterministic keys + upsert (ON CONFLICT DO NOTHING/UPDATE); a dedupe/idempotency table with a conditional write (PutItem with attribute_not_exists); content-addressed output keys (write to s3://.../{hash}); dedupe on a natural business key (event_id).
The idempotency table. A DynamoDB table keyed by idempotency_key with a TTL; the handler conditionally records "processed this key," and on a duplicate the conditional write fails, so the handler skips the side effect.
Scope carefully. Idempotency must cover the side effect, not just the record. "Recorded processed" and "did the work" should commit together (or be ordered so a crash between them is recoverable).

Poison-pill handling on streams.

Bisect on error. bisectBatchOnFunctionError: true splits a failing batch in half and retries each half, narrowing down to the single bad record instead of failing the whole batch repeatedly.
Caps. maximumRetryAttempts and maximumRecordAgeInSeconds bound how long a bad record blocks the shard.
On-failure destination. After caps, the failed batch's metadata (not the payload) goes to an SQS/SNS destination so you can investigate; the shard advances.
Why it matters. Without these, one un-processable record blocks its Kinesis shard forever, halting all downstream processing for that partition.

Common interview probes on error handling.

"What happens when an S3-triggered Lambda fails?" — async retry ×2, then on-failure destination / DLQ.
"How do you make an ETL handler idempotent?" — deterministic key + upsert, or a conditional-write dedupe table.
"How do you stop one bad Kinesis record blocking a shard?" — bisect on error + retry/age caps + on-failure destination.
"Where does an SQS DLQ get configured?" — on the queue's redrive policy, after maxReceiveCount.

Worked example — async S3 invocation with retries and an on-failure destination

Detailed explanation. An S3-triggered (async) Lambda occasionally fails on a transient downstream blip. Lambda retries twice automatically; if it still fails, the event should land in an on-failure destination for investigation rather than vanish. Configure retries and the destination.

Retries. maximumRetryAttempts: 2 (the async default) with maximumEventAgeInSeconds cap.
Destination. On-failure → an SQS DLQ that carries the full invocation record.
Alarm. CloudWatch alarm on DLQ depth > 0.

Question. Configure the async retry policy and on-failure destination for an S3-triggered transform.

Input.

Setting	Value
Invocation type	asynchronous (S3)
Max retry attempts	2
Max event age	3600 s
On-failure destination	`transform-dlq` (SQS)

Code.

# Async invocation config + destination
aws lambda put-function-event-invoke-config \
  --function-name csv-to-parquet \
  --maximum-retry-attempts 2 \
  --maximum-event-age-in-seconds 3600 \
  --destination-config '{
    "OnFailure": {"Destination": "arn:aws:sqs:...:transform-dlq"}
  }'

# handler.py — raise on real failure so retries + destination engage
def handler(event, context):
    for rec in event["Records"]:
        try:
            transform(rec)
        except TransientError:
            raise                     # let Lambda retry (×2), then -> on-failure dest
        except PermanentError as e:
            # unrecoverable: record context and DO NOT retry uselessly
            park_to_quarantine(rec, reason=str(e))
    return {"ok": True}

Step-by-step explanation.

For async sources like S3, Lambda manages retries internally. maximum-retry-attempts 2 means an initial attempt plus two retries (three total) before the event is considered failed and routed to the on-failure destination.
The handler raises on a transient error so Lambda's retry machinery engages — swallowing the exception would mark the invocation successful and skip retries entirely. Raising is the signal that says "please try again."
After retries are exhausted (or maximum-event-age passes), the full invocation record — event plus error — lands in transform-dlq. Unlike a bare DLQ, a destination carries the response/error context, so you know why it failed.
For a permanent error (malformed record that will never succeed), the handler parks it to a quarantine store immediately rather than burning three attempts on a hopeless case. Distinguishing transient from permanent avoids wasted retries.
A CloudWatch alarm on transform-dlq depth > 0 turns the destination into an actionable signal. Without the alarm, failures accumulate silently and expire — the DLQ becomes a data-loss trap, not a safety net.

Output.

Attempt	Outcome	Next
1	TransientError	retry
2	TransientError	retry
3	TransientError	→ on-failure destination
any	PermanentError	quarantined immediately
success	done	event acknowledged

Rule of thumb. For async (S3/SNS) Lambdas, set explicit retry attempts and an on-failure destination (richer than a bare DLQ), raise on transient errors so retries engage, quarantine permanent errors without wasteful retries, and alarm on DLQ depth. A failure must always have somewhere to go and someone to notice.

Worked example — idempotent S3-to-warehouse upsert with a dedupe table

Detailed explanation. An S3-triggered Lambda loads a file's rows into a warehouse. Because delivery is at-least-once, the same file can be processed twice, double-loading rows. A DynamoDB idempotency table keyed by the object's eTag records "this exact object version was loaded," and a conditional write makes the second attempt a no-op. Build it.

Key. idempotency_key = bucket/key@eTag — unique per object version.
Guard. PutItem with attribute_not_exists(pk); success = first time, ConditionalCheckFailed = duplicate.
TTL. Expire keys after, say, 30 days.

Question. Make the S3-to-warehouse load idempotent so duplicate deliveries don't double-load.

Input.

Component	Value
Idempotency store	DynamoDB `etl_idempotency`
Key	`bucket/key@eTag`
Guard	conditional put (`attribute_not_exists`)
TTL	30 days

Code.

# idempotent_load.py
import boto3, time
from botocore.exceptions import ClientError

ddb = boto3.client("dynamodb")
TABLE = "etl_idempotency"

def already_done(key: str) -> bool:
    try:
        ddb.put_item(
            TableName=TABLE,
            Item={"pk": {"S": key}, "ttl": {"N": str(int(time.time()) + 30*86400)}},
            ConditionExpression="attribute_not_exists(pk)",   # only if new
        )
        return False                # we just claimed it -> first time
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return True             # someone already claimed it -> duplicate
        raise

def handler(event, context):
    for rec in event["Records"]:
        b   = rec["s3"]["bucket"]["name"]
        k   = rec["s3"]["object"]["key"]
        tag = rec["s3"]["object"]["eTag"]
        key = f"{b}/{k}@{tag}"
        if already_done(key):
            print(f"skip duplicate {key}")
            continue
        rows = read_and_transform(b, k)
        load_to_warehouse(rows)     # the side effect, guarded by the claim above
    return {"ok": True}

Step-by-step explanation.

The idempotency key includes the eTag, so it identifies a specific version of the object. Re-uploading changed content produces a new eTag (correctly reprocessed); a duplicate notification for the same bytes produces the same key (correctly skipped).
put_item with ConditionExpression="attribute_not_exists(pk)" atomically claims the key. On the first delivery the put succeeds and returns False (not done). On a duplicate the condition fails with ConditionalCheckFailedException, returning True (already done) — a single atomic operation, no read-then-write race.
Only after successfully claiming the key does the handler perform the side effect (load to warehouse). The claim gates the work, so two concurrent duplicate deliveries can't both load — exactly one wins the conditional put.
The ttl attribute lets DynamoDB expire old keys automatically, bounding the table's size. Thirty days comfortably exceeds any redelivery window while keeping storage flat.
One subtlety: if the claim succeeds but the load then crashes, the key is marked done while the work isn't. For strict correctness, either make the load itself idempotent (upsert) as a second layer, or record the claim after the load with an in-progress marker — belt and braces for critical pipelines.

Output.

Delivery	Key	Conditional put	Action
1st of file A	`.../a@etag1`	succeeds	load rows
duplicate of A	`.../a@etag1`	fails	skip
A re-uploaded	`.../a@etag2`	succeeds	load new version
concurrent dup	`.../a@etag1`	one wins	one loads, one skips

Rule of thumb. Guard every non-idempotent side effect with an atomic conditional write on a deterministic idempotency key (object version, event_id, or content hash), TTL the dedupe table, and layer an upsert underneath for the crash-after-claim edge. In an at-least-once world, the dedupe key is as important as the transform itself.

Worked example — Kinesis bisect-on-error to isolate a poison record

Detailed explanation. A single malformed record in a Kinesis batch makes the handler throw; the mapping retries the whole batch, fails again, and blocks the shard. bisectBatchOnFunctionError splits the batch to isolate the bad record; retry/age caps and an on-failure destination bound the damage. Configure it.

Bisect. On error, split the batch and retry halves — narrows to the single bad record.
Caps. maximumRetryAttempts and maximumRecordAgeInSeconds stop infinite blocking.
Destination. On-failure → SQS with the failed batch metadata; shard advances.

Question. Configure the Kinesis event source mapping to isolate poison records and keep the shard moving.

Input.

Setting	Value
bisectBatchOnFunctionError	true
maximumRetryAttempts	5
maximumRecordAgeInSeconds	3600
on-failure destination	`kinesis-failures` (SQS)
functionResponseTypes	ReportBatchItemFailures

Code.

# Kinesis event source mapping (poison-pill safe)
aws lambda create-event-source-mapping \
  --function-name clickstream-consumer \
  --event-source-arn arn:aws:kinesis:...:stream/clickstream \
  --batch-size 500 \
  --maximum-batching-window-in-seconds 5 \
  --parallelization-factor 4 \
  --bisect-batch-on-function-error \
  --maximum-retry-attempts 5 \
  --maximum-record-age-in-seconds 3600 \
  --function-response-types ReportBatchItemFailures \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:...:kinesis-failures"}}'

# consumer.py — report the exact failing record; let bisect narrow it
import base64, json

def handler(event, context):
    failures = []
    for rec in event["Records"]:
        seq = rec["kinesis"]["sequenceNumber"]
        try:
            data = json.loads(base64.b64decode(rec["kinesis"]["data"]))
            transform(data)
        except Exception as e:
            print(f"bad record seq={seq}: {e}")
            failures.append({"itemIdentifier": seq})
    return {"batchItemFailures": failures}

Step-by-step explanation.

With ReportBatchItemFailures, the handler returns the exact failing sequenceNumbers, so the mapping retries only from the first failure — successes before it are checkpointed and never reprocessed.
If the whole invocation throws instead (or the reported failure persists), bisectBatchOnFunctionError splits the 500-record batch into two 250s and retries each. Repeated bisection homes in on the single poison record in ~log₂(500) ≈ 9 splits, instead of failing all 500 forever.
maximumRetryAttempts 5 and maximumRecordAgeInSeconds 3600 bound the blocking: after 5 attempts or once the record is an hour old, the mapping gives up on it rather than blocking the shard indefinitely.
When a record is finally abandoned, its batch metadata (shard, sequence range) is sent to the kinesis-failures SQS destination — not the payload, which you re-read from the stream if needed — and the shard iterator advances past it. Downstream processing resumes.
The combination is what keeps a stream healthy: partial failure reporting minimises reprocessing, bisection isolates the culprit, caps bound the blast radius in time, and the destination preserves a breadcrumb for investigation. Miss any one and a single bad record halts a partition.

Output.

Mechanism	Without it	With it
Report failures	whole batch reread	only tail reread
Bisect on error	500 fail forever	narrows to 1 record
Retry/age caps	shard blocked forever	bounded (5 tries / 1 h)
On-failure destination	no breadcrumb	metadata captured
Net	shard wedged	shard keeps moving

Rule of thumb. For Kinesis/DynamoDB-Streams ETL, always set bisectBatchOnFunctionError, retry and record-age caps, ReportBatchItemFailures, and an on-failure destination. A stream without poison-pill handling is one malformed record away from a silently stalled partition.

Data engineering interview question on idempotency and DLQs

A senior interviewer might ask: "Your SQS-triggered Lambda credits customer wallets from a payments event stream. Because delivery is at-least-once, you occasionally double-credit a wallet, and one malformed event retries endlessly. Design the handler and infrastructure so each event is applied exactly once in effect, and the poison event is quarantined without blocking the queue."

Solution Using an idempotency table with a conditional write plus a DLQ redrive

# wallet_credit.py — effect-once wallet credit under at-least-once delivery
import boto3, json, time
from botocore.exceptions import ClientError

ddb = boto3.client("dynamodb")
IDEMP = "wallet_idempotency"

def claim(event_id: str) -> bool:
    """Return True if we are the first to process this event_id."""
    try:
        ddb.put_item(
            TableName=IDEMP,
            Item={"pk": {"S": event_id}, "ttl": {"N": str(int(time.time()) + 30*86400)}},
            ConditionExpression="attribute_not_exists(pk)",
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise

def handler(event, context):
    failures = []
    for msg in event["Records"]:
        mid = msg["messageId"]
        try:
            body = json.loads(msg["body"])
            event_id = body["event_id"]
            if not claim(event_id):
                continue                          # duplicate -> effect-once skip
            credit_wallet(body["wallet_id"], body["amount_cents"])   # side effect
        except KeyError as e:
            failures.append({"itemIdentifier": mid})   # malformed -> retry -> DLQ
        except Exception:
            failures.append({"itemIdentifier": mid})   # transient -> retry
    return {"batchItemFailures": failures}

# Infra
SQS wallet-events:
  VisibilityTimeout   = 180
  RedrivePolicy       = { maxReceiveCount: 5, deadLetterTargetArn: wallet-events-dlq }
Event source mapping: FunctionResponseTypes = [ReportBatchItemFailures], BatchSize = 10
DynamoDB wallet_idempotency: pk = event_id, TTL enabled
CloudWatch alarm: wallet-events-dlq ApproximateNumberOfMessagesVisible > 0

Step-by-step trace.

Step	Input	Behaviour
1	event_id=E1, credit $5	`claim(E1)` succeeds → credit applied
2	duplicate event_id=E1	`claim(E1)` fails → skip (no double-credit)
3	event_id=E2, malformed (no wallet_id)	KeyError → reported failure
4	E2 retried 5×	still malformed → redrive to DLQ
5	transient DB blip on E3	reported failure → retried, then succeeds
6	DLQ depth > 0	CloudWatch alarm pages on-call

Walking the trace: the claim conditional put is the exactly-once effect gate — the first delivery of E1 claims the key and credits the wallet; every subsequent delivery of E1 fails the conditional write and skips the credit, so the wallet is credited once no matter how many times SQS delivers the message. The malformed E2 is reported (not re-raised), so only it is retried; after maxReceiveCount = 5 it moves to the DLQ, quarantined, while healthy messages flow. A transient blip on E3 is retried and eventually succeeds because the claim + credit are safe to repeat until they stick. The DLQ alarm makes the parked poison event visible to a human.

Output:

Event	Deliveries	Credits applied	Final location
E1 ($5)	3 (at-least-once)	1	done
E2 (malformed)	5	0	DLQ
E3 (transient)	2	1	done
wallet balance	—	+$10 (E1+E3)	correct

Why this works — concept by concept:

Conditional-write claim — the atomic PutItem with attribute_not_exists is the exactly-once effect primitive: only the first delivery claims the event_id, so the credit runs once even though delivery is at-least-once.
Effect-once, not delivery-once — you cannot make SQS deliver exactly once, so you make the side effect idempotent. The claim gates the credit; duplicates become no-ops.
Partial batch response — reporting only failed IDs means a malformed event retries alone and never forces healthy credits to reprocess.
DLQ redrive + alarm — bounding the malformed event to five attempts parks it in a DLQ instead of blocking the queue, and the alarm turns "parked" into "noticed."
Cost — one O(1) conditional write per event plus the credit; duplicates short-circuit before the side effect. Compared with a non-idempotent handler, financial correctness is guaranteed at the cost of a single small DynamoDB write per unique event.

Event
Topic — event-processing
Event-processing problems on exactly-once effects

Practice →

Data Topic — data-processing Data-processing problems on dedupe and retries

Practice →

5. Patterns & limits — fan-out, Step Functions handoff

Fan-out to scale horizontally, hand off at the fifteen-minute wall — the two moves that take Lambda past its limits

The mental model in one line: Lambda's two structural limits — a single invocation is time-boxed to fifteen minutes and processes one event's worth of work — are overcome by two complementary patterns: fan-out, where one event is sprayed to many parallel Lambdas (via SNS, or a Map over a manifest) so a big workload becomes thousands of small independent ones, and handoff, where a job too long or too multi-step for one invocation is escalated to Step Functions (or its Distributed Map) which orchestrates many Lambda steps with per-step retries, branching, and no overall time cap — and knowing which to reach for, plus the hard limits (payload size, /tmp, deployment package) that bound each, is the senior signal. Fan-out scales the width; Step Functions scales the length and coordination.

Fan-out patterns — one event, many workers.

SNS fan-out. Publish one event to an SNS topic; many Lambdas (and SQS queues) subscribe and each gets a copy. Decouples producer from N independent consumers — the classic pub/sub fan-out.
SQS fan-out with a splitter. A Lambda reads a manifest and enqueues one SQS message per work item; a fleet of consumers drains the queue with bounded concurrency. Good for back-pressure-controlled fan-out.
Map-style parallel fan-out. Step Functions Map (inline) or Distributed Map iterates a collection (e.g. every object under an S3 prefix, every line of a manifest) and runs one Lambda per item, with configurable max concurrency.
When to fan out. When the workload is a large set of independent small units (thousands of files, partitions, or shards) — fan-out turns O(N) serial work into O(N / concurrency) wall-clock.

The fifteen-minute wall and the Step Functions handoff.

The wall. No single invocation runs past fifteen minutes. A backfill, a multi-hour export, or a chained multi-step job cannot live in one Lambda.
The handoff. Escalate to a Step Functions state machine that calls Lambda for each short step and orchestrates the long-running whole — retries per state, Wait/Choice/Parallel states, and a total runtime up to a year (standard workflows).
Chunking under orchestration. A long job becomes many short Lambda steps (process chunk, checkpoint, loop) coordinated by Step Functions, each step comfortably under fifteen minutes.
When to hand off. Long duration, ordered multi-step pipelines, human-approval waits, or anything needing per-step retry/branch logic that would be ugly to hand-roll inside one Lambda.

Distributed Map — large-scale S3 fan-out.

What it is. A Step Functions state that reads a large dataset (millions of S3 objects or CSV/JSON lines) and runs a child workflow / Lambda per item at very high concurrency (up to ~10 000), with batching and tolerated-failure thresholds.
Why not plain Map. Inline Map is limited (bounded array, lower concurrency, in-state payload). Distributed Map is built for millions of items and offloads iteration to the service.
ETL use. The go-to for "reprocess every object in this prefix" backfills — it enumerates the objects, fans out, bounds concurrency to protect downstreams, and aggregates results.

Hard limits to remember.

Payload. Sync invocation request/response ≤ 6 MB; async event ≤ 256 KB. Big payloads go by reference (an S3 key), never inline.
/tmp ephemeral storage. 512 MB by default, configurable up to 10 GB. Scratch space for a single invocation; not shared or durable.
Deployment package. 50 MB zipped / 250 MB unzipped for zip; up to 10 GB for container images. Big dependencies push you to container images or layers.
Timeout & memory. 15 min max timeout; 128 MB–10 GB memory. Environment variables ≤ 4 KB total.

Common interview probes on patterns and limits.

"How do you process 100 000 S3 objects with Lambda?" — fan-out via Step Functions Distributed Map with bounded concurrency.
"A job takes 40 minutes — what do you do?" — hand off to Step Functions; chunk into sub-15-minute Lambda steps.
"How do you pass a large payload between steps?" — by reference (S3 key), not inline; mind the 6 MB / 256 KB limits.
"SNS vs SQS for fan-out?" — SNS for pub/sub to N independent consumers; SQS for buffered, back-pressured fan-out to a worker pool.

Worked example — SNS fan-out to independent consumers

Detailed explanation. An order event must trigger three independent things: update the warehouse, refresh a search index, and notify a fraud service. Coupling them in one Lambda makes one slow consumer block the others. SNS fan-out publishes once and lets three Lambdas each react independently. Build it.

Topic. order-events SNS topic.
Subscribers. Three Lambdas (warehouse, search, fraud), each with its own retry/DLQ.
Isolation. A failure in one consumer doesn't affect the others.

Question. Wire an SNS fan-out so one order event drives three independent consumers.

Input.

Component	Value
Topic	`order-events` (SNS)
Consumer 1	`warehouse-loader` Lambda
Consumer 2	`search-indexer` Lambda
Consumer 3	`fraud-scorer` Lambda
Delivery	each gets its own copy

Code.

# producer.py — publish ONCE; SNS fans out to all subscribers
import boto3, json
sns = boto3.client("sns")
TOPIC = "arn:aws:sns:...:order-events"

def on_order(order):
    sns.publish(
        TopicArn=TOPIC,
        Message=json.dumps(order),
        MessageAttributes={
            "event_type": {"DataType": "String", "StringValue": "OrderPlaced"}
        },
    )

# warehouse_loader.py — one of three independent subscribers
import json
def handler(event, context):
    for record in event["Records"]:                 # SNS -> Lambda event shape
        order = json.loads(record["Sns"]["Message"])
        upsert_order(order)                          # idempotent
    return {"ok": True}
# search_indexer.py and fraud_scorer.py subscribe to the same topic independently.

Step-by-step explanation.

The producer publishes the order once to the SNS topic. SNS delivers an independent copy to each subscribed Lambda, so the producer neither knows nor cares how many consumers exist — new consumers subscribe without touching the producer.
Each subscriber Lambda has its own retry policy and on-failure destination/DLQ. If search-indexer is failing, warehouse-loader and fraud-scorer proceed unaffected — the failure is isolated to one branch.
SNS→Lambda is asynchronous, so each consumer gets async retry semantics (retry then DLQ). Each consumer must still be idempotent because SNS delivery is at-least-once.
MessageAttributes allow subscription filter policies — a consumer can subscribe only to event_type = OrderPlaced and ignore other event types on the same topic, filtering at SNS instead of in code.
For consumers that need buffering/back-pressure rather than direct invocation, subscribe an SQS queue to the topic and let a worker pool drain it — SNS fan-out and SQS buffering compose. This is the standard "fan-out then back-pressure" shape.

Output.

Consumer	Gets event?	Failure isolated?	Retry/DLQ
warehouse-loader	yes	yes	own
search-indexer	yes	yes	own
fraud-scorer	yes	yes	own
producer	publishes once	N/A	N/A

Rule of thumb. For one-event-to-many-independent-reactions, publish once to SNS and let each consumer subscribe with its own retry/DLQ and filter policy. Add an SQS queue between SNS and a consumer when that consumer needs buffering. Never couple independent consumers into one Lambda — one slow branch shouldn't block the rest.

Worked example — Step Functions handoff for a multi-hour backfill

Detailed explanation. A backfill must reprocess a month of data — far more than fifteen minutes of work. A single Lambda can't do it, so a Step Functions state machine drives a loop: each iteration invokes a Lambda that processes one day (well under fifteen minutes), checkpoints progress, and the state machine loops until done. Build the workflow.

Structure. A Choice/Map loop over days; each day is one Lambda step.
Checkpointing. Each step records the last completed day so a failure resumes, not restarts.
No time cap. The state machine can run for hours/days (standard workflow) while each step stays short.

Question. Design a Step Functions workflow that backfills a month of data via short Lambda steps.

Input.

Component	Value
Workflow type	Standard (long-running)
Step	`process-day` Lambda (≤ 15 min)
Iteration	one per day, 30 days
Checkpoint	last completed day in DynamoDB

Code.

// backfill.asl.json — Step Functions state machine (simplified)
{
  "StartAt": "NextDay",
  "States": {
    "NextDay": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:...:function:process-day",
      "Retry": [
        {"ErrorEquals": ["TransientError"], "MaxAttempts": 4, "BackoffRate": 2.0, "IntervalSeconds": 10}
      ],
      "Catch": [
        {"ErrorEquals": ["States.ALL"], "Next": "ParkFailure"}
      ],
      "Next": "MoreDays?"
    },
    "MoreDays?": {
      "Type": "Choice",
      "Choices": [
        {"Variable": "$.done", "BooleanEquals": false, "Next": "NextDay"}
      ],
      "Default": "Succeed"
    },
    "ParkFailure": {"Type": "Task", "Resource": "arn:aws:lambda:...:function:park-failure", "Next": "Fail"},
    "Fail":    {"Type": "Fail"},
    "Succeed": {"Type": "Succeed"}
  }
}

# process_day.py — one short step; returns whether more days remain
import boto3
ddb = boto3.client("dynamodb")

def handler(event, context):
    day = next_unprocessed_day(event)      # read checkpoint
    process_one_day(day)                   # << 15 minutes of work
    checkpoint(day)                        # durable progress
    remaining = days_left(day)
    return {"last_day": day, "done": remaining == 0}

Step-by-step explanation.

Step Functions owns the loop and the clock. Each NextDay task invokes process-day for a single day's data — a unit deliberately sized to finish well under fifteen minutes — and returns done: false until the last day.
The MoreDays? Choice state loops back to NextDay while work remains. The overall backfill can run for hours across 30 iterations, but no individual Lambda ever approaches the time wall.
Per-step Retry with exponential backoff handles transient errors at the orchestration layer, so the Lambda stays simple. A permanent failure is caught and routed to ParkFailure, preserving context.
Each step checkpoints the last completed day durably. If the workflow fails and is restarted, next_unprocessed_day resumes from the checkpoint rather than reprocessing from the start — the backfill is restartable.
This is the canonical "job too long for Lambda" answer: don't fight the fifteen-minute wall, decompose the job into short idempotent steps and let Step Functions orchestrate duration, retries, and branching.

Output.

Iteration	Day processed	done	Next state
1	2026-08-01	false	NextDay
2	2026-08-02	false	NextDay
…	…	false	NextDay
30	2026-08-30	true	Succeed
on error	(parked)	—	ParkFailure → Fail

Rule of thumb. When a job exceeds fifteen minutes, hand it to Step Functions and decompose it into short, idempotent, checkpointed Lambda steps. Put retries and branching in the state machine, keep each step small, and make progress durable so a restart resumes. Orchestration beats fighting the time wall.

Worked example — Distributed Map over an S3 manifest

Detailed explanation. A backfill must reprocess 120 000 S3 objects. A serial loop would take hours and a single Lambda can't hold the list. Step Functions Distributed Map enumerates the objects and runs one Lambda per object at high, bounded concurrency, tolerating a small failure rate. Build it.

Source. An S3 prefix (or a manifest file) of 120 000 objects.
Concurrency. Max 500 concurrent child executions (bounded to protect downstreams).
Tolerated failure. Continue if < 1% of items fail; collect failures.

Question. Configure a Distributed Map to reprocess 120 000 objects with bounded concurrency and a failure threshold.

Input.

Setting	Value
Item source	S3 prefix `raw/2026/`
Max concurrency	500
Tolerated failure %	1
Per-item worker	`reprocess-object` Lambda

Code.

// distributed_map.asl.json — Distributed Map over an S3 prefix
{
  "StartAt": "ReprocessAll",
  "States": {
    "ReprocessAll": {
      "Type": "Map",
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": {"Bucket": "data-lake-prod", "Prefix": "raw/2026/"}
      },
      "ItemProcessor": {
        "ProcessorConfig": {"Mode": "DISTRIBUTED", "ExecutionType": "STANDARD"},
        "StartAt": "Reprocess",
        "States": {
          "Reprocess": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:...:function:reprocess-object",
            "End": true
          }
        }
      },
      "MaxConcurrency": 500,
      "ToleratedFailurePercentage": 1,
      "ResultWriter": {
        "Resource": "arn:aws:states:::s3:putObject",
        "Parameters": {"Bucket": "data-lake-prod", "Prefix": "backfill-results/"}
      },
      "End": true
    }
  }
}

# reprocess_object.py — one object per invocation, idempotent
def handler(event, context):
    bucket = event["Bucket"]
    key    = event["Key"]                 # provided per item by the Map
    reprocess(bucket, key)                # streaming, constant memory, idempotent
    return {"key": key, "status": "ok"}

Step-by-step explanation.

The ItemReader with s3:listObjectsV2 makes Step Functions enumerate the 120 000 objects itself — the list never lives inside a Lambda, sidestepping payload and memory limits.
Mode: DISTRIBUTED runs each item as an independent child execution, so the Map scales to very high item counts (millions) that inline Map cannot handle.
MaxConcurrency: 500 bounds how many reprocess-object Lambdas run at once — protecting downstreams (warehouse, RDS) exactly like reserved concurrency did in section 3, but at the orchestration layer.
ToleratedFailurePercentage: 1 lets the backfill continue even if a handful of objects fail, rather than aborting the whole 120 000-object job on the first bad file. Failures are recorded; you fix and rerun just those.
The ResultWriter aggregates per-item results to S3, giving a durable manifest of successes/failures for audit and targeted retry. Each worker processes exactly one object idempotently, so a retried item is safe. This is the production answer to "reprocess everything in this prefix."

Output.

Aspect	Value
Objects enumerated	120,000 (by the service)
Peak concurrency	500 workers
Wall-clock	~ (120k × 15 s) / 500 ≈ 1 hour
Tolerated failures	up to 1,200 (1%) before abort
Results	written to `backfill-results/`

Rule of thumb. For massive S3 fan-out, use Step Functions Distributed Map: let the service enumerate items, bound MaxConcurrency to protect downstreams, set a ToleratedFailurePercentage so a few bad objects don't abort the run, and write results for audit. It turns a many-hour serial backfill into a bounded-concurrency parallel one — the modern replacement for hand-rolled fan-out.

Data engineering interview question on scaling past Lambda limits

A senior interviewer might ask: "You need to reprocess 2 million small JSON files (each ~1 MB) from an S3 prefix into a partitioned Parquet table, protecting a downstream warehouse that tolerates ~300 concurrent writers, and the whole job must be restartable and take hours if needed. A single Lambda obviously can't do it. Design the solution and justify each choice against Lambda's limits."

Solution Using a Step Functions Distributed Map handoff with bounded concurrency

// reprocess_2m.asl.json — Distributed Map, bounded, restartable
{
  "StartAt": "Backfill",
  "States": {
    "Backfill": {
      "Type": "Map",
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": {"Bucket": "lake", "Prefix": "raw/json/"}
      },
      "ItemBatcher": {"MaxItemsPerBatch": 20},
      "ItemProcessor": {
        "ProcessorConfig": {"Mode": "DISTRIBUTED", "ExecutionType": "STANDARD"},
        "StartAt": "ToParquet",
        "States": {
          "ToParquet": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:...:function:json-to-parquet",
            "Retry": [{"ErrorEquals": ["TransientError"], "MaxAttempts": 4, "BackoffRate": 2.0}],
            "End": true
          }
        }
      },
      "MaxConcurrency": 300,
      "ToleratedFailurePercentage": 1,
      "ResultWriter": {
        "Resource": "arn:aws:states:::s3:putObject",
        "Parameters": {"Bucket": "lake", "Prefix": "backfill-manifests/"}
      },
      "End": true
    }
  }
}

# json_to_parquet.py — process a small BATCH of objects per invocation
import boto3, io, json
import pyarrow as pa, pyarrow.parquet as pq
s3 = boto3.client("s3")

def handler(event, context):
    # ItemBatcher hands us up to 20 items -> fewer invocations, still short
    for item in event["Items"]:
        b, k = item["Bucket"], item["Key"]
        rows = [json.loads(l) for l in s3.get_object(Bucket=b, Key=k)["Body"].iter_lines()]
        part = "curated/" + partition_path(rows) + "/" + k.split("/")[-1] + ".parquet"
        buf = io.BytesIO(); pq.write_table(pa.Table.from_pylist(rows), buf); buf.seek(0)
        s3.put_object(Bucket=b, Key=part, Body=buf.getvalue())   # deterministic key = idempotent
    return {"ok": True}

Step-by-step trace.

Step	Choice	Justification vs Lambda limit
1	Distributed Map enumerates 2M keys	list never enters a Lambda (payload/memory limit)
2	ItemBatcher 20/invocation	20 × ~1 MB × short = well under 15 min & 10 GB
3	MaxConcurrency 300	≤ warehouse's 300-writer budget (back-pressure)
4	deterministic Parquet key	at-least-once safe (idempotent overwrite)
5	ToleratedFailurePercentage 1	a few bad files don't abort 2M-file run
6	ResultWriter manifest	restartable: rerun only failed items

Walking the trace: the 2-million-object list is enumerated by Step Functions, never materialised in a Lambda, so the payload and memory limits are irrelevant. ItemBatcher groups 20 small objects per invocation, keeping each invocation short (seconds) and small (tens of MB) — comfortably inside the fifteen-minute and 10 GB walls. MaxConcurrency 300 caps parallel writers at the warehouse's tolerance, applying back-pressure exactly where section 3 applied reserved concurrency, but at the orchestration layer. Deterministic output keys make each object's write idempotent, so retries and at-least-once execution never duplicate rows. The tolerated-failure threshold keeps a handful of malformed files from aborting the whole run, and the ResultWriter manifest records every outcome so a rerun targets only the failures — the restartable requirement satisfied.

Output:

Requirement	Mechanism	Met?
2M files, single Lambda can't	Distributed Map fan-out	yes
≤ 300 concurrent writers	MaxConcurrency 300	yes
Restartable / hours OK	Standard workflow + manifest	yes
At-least-once safe	deterministic keys	yes
A few bad files tolerated	ToleratedFailurePercentage 1	yes

Why this works — concept by concept:

Distributed Map fan-out — the service enumerates and iterates millions of items, so no Lambda ever holds the list; the payload/memory limits that would sink a hand-rolled splitter simply don't apply.
Item batching for right-sized units — grouping 20 small objects per invocation keeps each unit short and small, respecting the fifteen-minute and 10 GB walls while cutting invocation count 20×.
Bounded MaxConcurrency as back-pressure — capping concurrent workers at the downstream's tolerance protects the warehouse, the orchestration-layer analogue of reserved concurrency.
Deterministic keys for idempotency — content-addressed output keys make every write a safe overwrite, so the guaranteed retries of an at-least-once system never duplicate data.
Cost — O(N / concurrency) wall-clock at O(concurrency) parallelism, with a result manifest enabling O(failures) targeted reruns instead of reprocessing all N. Compared with a single hand-rolled fan-out Lambda, it removes the payload, memory, and time walls entirely and makes the backfill restartable by construction.

Event
Topic — event-processing
Event-processing problems on fan-out and orchestration

Practice →

ETL
Topic — etl
ETL problems on large-scale backfills

Practice →

Cheat sheet — AWS Lambda ETL recipes

Fit / no-fit rule. Lambda fits per-event ETL when a unit of work finishes in well under 15 minutes and holds under 10 GB memory, is stateless (or externalises state to DynamoDB), and the event rate sits below the cost crossover with Glue/EMR. Score every job on duration × per-unit volume × unit count × cross-event state; the two hard walls disqualify a unit outright, and huge unit counts mean fan-out, not a cluster.
Push vs poll triggers. S3/SNS/EventBridge push async (retry ×2 → DLQ/destination, one event each, no batch knob). SQS/Kinesis/DynamoDB-Streams are polled by the event source mapping which batches and invokes sync — tune batchSize, maximumBatchingWindowInSeconds, and failure handling on the mapping.
S3 trigger config. Scope with prefix + suffix filters; include both ObjectCreated:Put and ObjectCreated:CompleteMultipartUpload (large uploads fire the latter); loop over all event["Records"]; derive output keys deterministically so at-least-once re-delivery overwrites idempotently.
SQS ESM knobs. batchSize (≤10 typical), maximumBatchingWindowInSeconds, visibility timeout ≥ 6× function timeout, FunctionResponseTypes=[ReportBatchItemFailures] (return only failed IDs — never re-raise), and a redrive policy (maxReceiveCount → DLQ). Cap load with ScalingConfig.MaximumConcurrency.
Kinesis ESM knobs. batchSize per shard, parallelizationFactor up to 10 (more throughput, per-partition-key order preserved), TRIM_HORIZON vs LATEST, plus poison-pill guards: bisectBatchOnFunctionError, maximumRetryAttempts, maximumRecordAgeInSeconds, on-failure destination. Order is a partition-key property, per shard.
Concurrency types. Unreserved = shared regional pool (throttles with 429). Reserved = per-function cap that both guarantees and limits slots (protects downstreams). Provisioned = pre-warmed sandboxes that remove cold starts (pay always; use for latency-sensitive sync, rarely batch). Steady concurrency ≈ rps × avg_duration_s.
Cold-start fixes. Shrink the deployment package; lazy-import heavy libs; do one-time setup (DB clients, model loads) at init and reuse across warm invocations; avoid needless VPC attachment; use provisioned concurrency only where p99 is a user-facing SLO. For batch ETL, cold starts amortise away — don't pay to remove them.
Memory = CPU (power tuning). CPU/network scale with the memory dial (~1 vCPU at ~1769 MB). Sweep memory, measure duration and GB-seconds, pick the minimum-cost point (often ~1 vCPU for CPU-bound work) or minimum-latency point. Never leave a CPU-bound job at 128 MB.
Error handling stack. At-least-once ⇒ retries guaranteed ⇒ handlers must be idempotent. Async: maximum-retry-attempts + on-failure destination (richer than DLQ). SQS: redrive → DLQ. Streams: bisect + caps + on-failure destination. Alarm on every DLQ depth > 0 and build a redrive/replay path.
Idempotency template. Deterministic key (bucket/key@eTag, event_id, or content hash) + atomic conditional write (PutItem with attribute_not_exists) to a TTL'd DynamoDB dedupe table; run the side effect only after claiming the key; layer an ON CONFLICT upsert underneath for the crash-after-claim edge. Make the effect exactly-once, not the delivery.
Fan-out patterns. SNS for pub/sub to N independent consumers (each with its own retry/DLQ and filter policy). SQS splitter for buffered, back-pressured fan-out to a worker pool. Step Functions Map / Distributed Map for iterating a manifest or S3 prefix at bounded concurrency.
Step Functions handoff. When a job exceeds 15 minutes or needs multi-step retry/branch logic, hand off to a Standard state machine that calls Lambda for short, idempotent, checkpointed steps. Use Distributed Map for million-object S3 backfills: service-side enumeration, MaxConcurrency back-pressure, ToleratedFailurePercentage, and a ResultWriter manifest for restartable reruns.
Hard limits. Sync request/response ≤ 6 MB, async event ≤ 256 KB (pass big payloads by S3 reference); /tmp 512 MB–10 GB; zip package 50 MB/250 MB (containers up to 10 GB); env vars ≤ 4 KB; timeout ≤ 15 min; memory 128 MB–10 GB.
Cost crossover. Lambda wins bursty and low-to-medium steady rates (nothing paid at idle, no cluster ops); a sustained firehose well above the compute crossover (cluster_hourly / lambda_cost_per_event) favours a provisioned engine. Always add "no cluster to run" to Lambda's side of the ledger.

Frequently asked questions

Is AWS Lambda good for ETL?

Yes, for the right shape of ETL. aws lambda etl shines on event-driven, per-event work — transform a file the moment it lands, enrich a message off a queue, route a streamed record — because it scales from zero to thousands of parallel invocations with no cluster to run and you pay only per invocation. It is a poor fit for a single long-running job that needs a big in-memory shuffle or must chew through terabytes end to end: those hit the 15-minute wall or the 10 GB memory ceiling and belong on Glue, EMR, or a Step Functions-orchestrated set of short Lambda steps. The rule of thumb is: Lambda for the trigger, transform, and fan-out; a distributed engine for the heavy batch in the middle.

What is the 15-minute Lambda limit and how do I work around it?

Every Lambda invocation is hard-capped at fifteen minutes of wall-clock time; when it's reached, the invocation is killed regardless of progress. You do not work around it by trying to make one invocation faster past a point — you decompose the job. Split the work into units that each finish comfortably under fifteen minutes (smaller files, record batches, per-day chunks), and orchestrate the whole with Step Functions, which has no overall time cap and calls Lambda for each short, idempotent, checkpointed step. For massive parallel work like reprocessing millions of S3 objects, Step Functions Distributed Map fans the units out at bounded concurrency. If a single indivisible unit genuinely can't fit, that unit belongs on Glue/EMR, not Lambda.

How do S3 triggers work with Lambda?

You attach an S3 event notification to a bucket so that object events (s3:ObjectCreated:*, deletes, restores) invoke your Lambda asynchronously, scoped by prefix and suffix filters so only the objects you care about fire it. Delivery is at-least-once and not strictly ordered, so the same object can occasionally trigger the function more than once — your handler must be idempotent (derive output keys deterministically or dedupe on the object eTag). Two gotchas: include ObjectCreated:CompleteMultipartUpload (large uploads fire that, not Put), and loop over every record in event["Records"] because a notification can carry more than one. Because it's an async source, failures retry twice and then route to an on-failure destination or DLQ.

What is the difference between reserved and provisioned concurrency?

Reserved concurrency is a per-function cap on the number of concurrent executions: it both guarantees the function at least that many slots and limits it to at most that many, which is how you protect a fragile downstream (cap below its connection budget) and stop one function from starving the account pool. Provisioned concurrency is different — it keeps a set number of execution environments pre-initialised and warm so those invocations skip the cold start entirely; you pay for them whether used or not. Use reserved concurrency to shape blast radius and back-pressure (nearly always in ETL), and provisioned concurrency only where cold-start tail latency is a user-facing SLO (rarely for pure batch ETL, where a cold start is a rounding error).

How do I make a Lambda ETL job idempotent?

Because delivery is at-least-once and retries are guaranteed, the same event will be processed more than once, so you make the effect idempotent rather than hoping for exactly-once delivery. The core techniques: derive deterministic output keys (content-addressed S3 objects, or bucket/key@eTag) so a re-run overwrites identical bytes; use upserts (INSERT ... ON CONFLICT DO NOTHING/UPDATE) instead of blind inserts; and for side effects that can't be made naturally idempotent, gate them with an atomic conditional write to a DynamoDB dedupe table keyed by event_id (PutItem with attribute_not_exists) — the first delivery claims the key and does the work, every duplicate fails the condition and skips it. TTL the dedupe table and, for critical paths, layer an upsert under the claim to cover a crash between claiming and doing the work.

When should I use Step Functions instead of Lambda?

Reach for Step Functions whenever the work is longer than fifteen minutes, has multiple steps that each need their own retry/backoff, must branch on results, waits for something (a human approval, a poll), or fans out over a huge collection. Step Functions is the orchestrator; Lambda is the worker it calls for each short step, so you get per-step retries, Choice/Parallel/Wait states, durable checkpointing, and no overall time cap — without hand-rolling that logic inside one function. For large-scale S3 ETL, Distributed Map iterates millions of objects at bounded concurrency with a tolerated-failure threshold and a result manifest. If your job is a single short per-event transform, plain Lambda is simpler and cheaper; the moment you need duration, coordination, or branching, hand off.

Practice on PipeCode

Drill the ETL practice library → for the serverless ingestion, S3-trigger transform, batch-consumer, and large-scale backfill problems senior interviewers love.
Rehearse on the event-processing practice library → for queue/stream consumers, partial batch response, fan-out, and exactly-once-effect scenarios.
Sharpen the transform axis with the data-processing practice library → for streaming transforms, dedupe, and concurrency-tuning patterns.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the fit/no-fit decision and the event-driven design trade-offs against real graded inputs.

Lock in event-driven ETL muscle memory

Docs explain the services. PipeCode drills explain the decision — when Lambda fits ETL and when the 15-minute wall sends you to Step Functions, when partial batch response saves a queue, when reserved concurrency protects a database, when an idempotency key turns at-least-once into effect-once. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice ETL problems →
Practice event-processing problems →

Redis for Data Engineers: Beyond Caching

Gowtham Potureddi — Wed, 09 Sep 2026 11:37:48 +0000

Redis for data engineers is a mental-model upgrade, not a new tool to learn — the same in-memory server most teams reach for as a dumb key-value cache is actually a data-structure server whose sorted sets, append-only streams, and probabilistic sketches quietly power leaderboards, rate limiters, deduplicated event ingestion, and billion-row unique-visitor counts inside pipelines you already run. The reason so many engineers stall at GET/SET is that the caching use case is so obvious it hides everything else: the moment you treat a Redis key as a shape — a ranked set ordered by score, a log you can replay with consumer groups, a 12-kilobyte cardinality estimator — the same box that memoizes your query results also becomes the cheapest way to solve an entire class of streaming and analytics problems.

This guide is the walkthrough you wished existed the first time an interviewer asked "you have Redis in the stack already — how would you build a sliding-window rate limiter without hammering Postgres?", or "count the daily unique users across two billion events without storing two billion IDs", or "why would you pick a stream over a list for an ingestion buffer?". It opens the toolbox in layers: the five core value types and the O(1)/O(log N) primitives that make them fast, Redis Streams with consumer groups for durable at-least-once fan-out, sorted sets as the Swiss-army structure behind leaderboards, time-series windows and rate limiting, the HyperLogLog and Bloom filters that trade exactness for constant memory, and finally the persistence, eviction, and atomicity model that decides whether your Redis survives a restart. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the data-structures practice library →, rehearse storage internals on the database practice library →, and sharpen the ingestion axis with the streaming practice library →.

On this page

Redis core data structures — strings, hashes, lists, sets, sorted sets
Redis Streams — the append-only log with consumer groups
Sorted sets — leaderboards, time-series windows, rate limiting
Probabilistic structures — HyperLogLog and Bloom filters
Persistence, patterns and pitfalls
Cheat sheet — Redis for data engineers recipes
Frequently asked questions
Practice on PipeCode

1. Redis core data structures — strings, hashes, lists, sets, sorted sets

The value type you pick is the optimization — five shapes, each with O(1) or O(log N) primitives

The one-sentence invariant: Redis is not a key-value store that happens to have extras — it is a data-structure server where every key holds one of a handful of typed values (string, hash, list, set, sorted set, stream, and a few specialised sketches), and choosing the right shape gives you the exact algorithmic complexity you need for free, because the command set for each type is a curated list of the operations that structure supports cheaply. A counter is a string you INCR; a session is a hash you update field-by-field; a capped event buffer is a list you LTRIM; a de-duplicated membership test is a set you SISMEMBER; a ranked scoreboard is a sorted set you ZADD. The engineer who reaches for GET/SET and serialises a JSON blob into a single string is leaving every one of those primitives on the table.

The five value types and what each is for.

String. The atom — bytes, an integer, or a float up to 512 MB. But the killer feature is atomic numeric mutation: INCR, INCRBY, DECR, INCRBYFLOAT mutate a counter without a read-modify-write race. Strings back counters, rate-limit tallies, feature flags, and cache entries. SETEX key ttl value writes value + TTL in one command.
Hash. A field-to-value map under one key — HSET user:42 name Ada plan pro. You read or write individual fields (HGET, HSET, HINCRBY) without deserialising the whole object. This is the correct shape for a "row" — a session, a user profile, a feature-store record — because partial updates are O(1) and memory is far tighter than N separate string keys.
List. An ordered, doubly-linked sequence — LPUSH/RPUSH at the ends, LPOP/RPOP to drain, LRANGE to read a window, LTRIM to cap length. Lists back simple queues, most-recent-N feeds, and capped log buffers. BLPOP blocks for producer/consumer hand-off.
Set. An unordered collection of unique members — SADD, SISMEMBER (O(1) membership), SCARD (count), and set algebra (SINTER, SUNION, SDIFF). Sets back de-duplication, tag membership, "has this user seen X", and audience intersections.
Sorted set (zset). A set where every member carries a floating-point score, kept ordered by score — ZADD, ZRANGE/ZREVRANGE, ZRANK, ZINCRBY, ZRANGEBYSCORE. This is the most powerful data-engineering structure in Redis; section 3 is entirely about it.

The encoding internals interviewers love to probe.

Small collections are packed. A small hash, list, set, or zset is stored as a listpack (a compact contiguous byte array) — tiny memory, linear scan, but the collection is small so scans are cheap. Above configurable thresholds (hash-max-listpack-entries, zset-max-listpack-entries, etc.) Redis converts to the full structure: a hashtable for hashes/sets, a skiplist + hashtable for zsets.
Why it matters. The conversion changes both memory footprint and complexity. A 100-field hash in listpack encoding is compact; a 100,000-field hash is a hashtable with O(1) field access but far more overhead. Knowing the threshold explains "why did my memory jump 4× when the collection grew".
Integer sets. A set of only integers uses the intset encoding — a sorted integer array — until it grows or gains a non-integer member. This is why a set of user IDs is dramatically smaller than a set of UUIDs.

TTL — a first-class property of every key.

Any key can expire. EXPIRE key seconds, PEXPIRE (ms), EXPIREAT (absolute), and the write-plus-TTL shortcuts SETEX/SET key val EX 60. TTL key reads the remaining life; PERSIST removes it.
Expiration is lazy + sampled. Redis does not scan for expired keys; it evicts on access (lazy) and via a background sampler that probabilistically clears expired keys. The practical consequence: an expired key stops answering immediately, but its memory is reclaimed slightly later.
Data-engineering use. TTL is how you build rolling windows, idempotency guards, and self-cleaning caches without a cron job. A rate-limit key that expires after the window, a "processed event ID" set member that expires after the dedup horizon — TTL removes the reconcile job.

What interviewers listen for.

Do you name the structure, not the command? — "that's a sorted set" beats "I'd use ZADD".
Do you reach for a hash for object rows instead of a serialised JSON string? — senior signal on memory + partial-update awareness.
Do you know INCR is atomic and eliminates the read-modify-write race? — required answer for any counter question.
Do you mention the listpack → hashtable/skiplist conversion when asked about memory? — senior signal.

Worked example — atomic counters with `INCR` and `EXPIRE`

Detailed explanation. The most common Redis anti-pattern is GET count, increment in application code, SET count. Under concurrency this loses updates: two workers read 10, both write 11, one increment vanishes. Redis's INCR collapses read-modify-write into a single atomic server-side operation. Pair it with EXPIRE and you have a windowed counter — page views this hour, API calls this minute — that cleans itself up.

Atomicity. INCR is executed by Redis's single-threaded command loop; no two increments interleave.
First-write TTL. Set the TTL only on the first increment so the window starts on the first event, not on every event.
Read. GET returns the current tally; the key vanishes when the window ends.

Question. Build an hourly page-view counter for an article that resets every hour and never loses an increment under concurrent writers.

Input.

Parameter	Value
Key pattern	`views:article:{id}:{yyyymmddHH}`
Increment op	`INCR`
Window	1 hour
TTL policy	set on first increment only

Code.

import redis
from datetime import datetime, timezone

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

def record_view(article_id: int) -> int:
    # Bucket key by hour so each hour is a fresh counter
    hour = datetime.now(timezone.utc).strftime("%Y%m%d%H")
    key  = f"views:article:{article_id}:{hour}"

    # INCR is atomic and creates the key at 0 if absent
    count = r.incr(key)

    # Only arm the TTL on the first increment (count == 1),
    # so the window is anchored to the first view of the hour.
    if count == 1:
        r.expire(key, 3600)   # key self-destructs after 1 hour

    return count

def views_this_hour(article_id: int) -> int:
    hour = datetime.now(timezone.utc).strftime("%Y%m%d%H")
    val  = r.get(f"views:article:{article_id}:{hour}")
    return int(val) if val else 0

Step-by-step explanation.

The key embeds the hour bucket (...:2026090514), so each hour is a physically separate counter. There is no "reset" step — the new hour simply writes to a new key, and the old key expires on its own.
r.incr(key) atomically creates the key as 0 if it does not exist, adds one, and returns the new value — all inside Redis's single-threaded execution, so concurrent callers never lose an increment.
The if count == 1 guard arms the TTL exactly once. Calling EXPIRE on every increment would keep pushing the expiry forward (a sliding TTL), which is not what an hourly bucket wants — we want a fixed 1-hour life from the first view.
views_this_hour reads the current bucket with GET; a missing key (no views yet, or the hour already expired) returns 0. There is no cleanup job — expiry is handled by Redis.
Because the counter lives in Redis, the hot write path never touches the primary database. A nightly job can SCAN the day's buckets and roll them into the warehouse if you need durable history.

Output.

Event	Command	Return	TTL after
1st view, 14:03	`INCR views:...:2026090514`	1	`EXPIRE 3600` armed
2nd view, 14:07	`INCR`	2	~3360 s remaining
3rd view, 14:59	`INCR`	3	~40 s remaining
view at 15:00	`INCR views:...:2026090515`	1	new bucket, TTL armed

Rule of thumb. For any counter, use INCR — never GET + application-side add + SET. Bucket by time in the key name for windowed counts, and arm EXPIRE only on the first increment so the window is anchored, not sliding.

Worked example — a hash-backed session / feature-store row

Detailed explanation. A common mistake is storing a user object as a serialised JSON string: every field read deserialises the whole blob, every field write re-serialises and rewrites it, and two concurrent field updates clobber each other. A Redis hash stores the object as independent fields under one key — read one field, write one field, atomically increment a numeric field — with far tighter memory than N separate string keys.

Partial reads/writes. HGET, HSET, HDEL, HINCRBY touch single fields in O(1).
Whole-object read. HGETALL returns every field when you genuinely need the full row.
Atomic field math. HINCRBY user:42 login_count 1 bumps a counter inside the row without a read.

Question. Model a user session as a hash that stores profile fields, tracks a login counter, and expires 30 minutes after the last activity.

Input.

Field	Type	Example
`user_id`	string	`42`
`plan`	string	`pro`
`login_count`	integer	`7`
`last_seen`	string (iso)	`2026-09-05T14:03:00Z`

Code.

import redis
from datetime import datetime, timezone

r = redis.Redis(decode_responses=True)
SESSION_TTL = 1800  # 30 minutes

def start_session(session_id: str, user_id: int, plan: str) -> None:
    key = f"session:{session_id}"
    # HSET writes multiple fields in one round trip
    r.hset(key, mapping={
        "user_id": user_id,
        "plan": plan,
        "login_count": 0,
        "last_seen": datetime.now(timezone.utc).isoformat(),
    })
    r.expire(key, SESSION_TTL)

def touch(session_id: str) -> None:
    key = f"session:{session_id}"
    # Atomic field increment + field update, then slide the TTL forward
    r.hincrby(key, "login_count", 1)
    r.hset(key, "last_seen", datetime.now(timezone.utc).isoformat())
    r.expire(key, SESSION_TTL)   # sliding expiry: 30 min from last touch

def plan_for(session_id: str) -> str | None:
    # Read ONE field without deserialising the whole object
    return r.hget(f"session:{session_id}", "plan")

Step-by-step explanation.

HSET ... mapping={...} writes all four fields in a single round trip. Compared to four separate SET session:{id}:plan ... keys, the hash uses one key's overhead and keeps the fields co-located for cheap HGETALL.
touch uses HINCRBY to bump login_count atomically — no read of the old value, no lost-update race — and HSET to refresh last_seen. Both mutate single fields; the rest of the row is untouched.
The EXPIRE in touch implements a sliding session: every activity resets the 30-minute clock, so an active user never expires and an idle session is reclaimed automatically. This is the opposite policy from the hourly counter, and the choice is deliberate.
plan_for reads exactly one field with HGET. A JSON-string design would fetch and parse the entire object to read one attribute — wasteful on a hot path.
Because the whole session is one key, it participates cleanly in TTL, DEL, and replication as a unit. There is no risk of the plan key surviving while the login_count key expired — a classic bug when you spread one object across many keys with independent TTLs.

Output.

Step	Command	Effect
login	`HSET session:abc ...`	row created, 4 fields
activity	`HINCRBY session:abc login_count 1`	`login_count` → 1
activity	`EXPIRE session:abc 1800`	TTL reset to 30 min
read plan	`HGET session:abc plan`	`"pro"` (one field)
30 min idle	(lazy expiry)	key removed

Rule of thumb. Store an object as a hash, not a serialised string, whenever you read or write individual fields. Use HINCRBY for in-row counters and reset the TTL on activity for a sliding session, or set it once for a fixed window.

Worked example — a list as a capped "last N events" buffer

Detailed explanation. You often need the most-recent-N of something — last 100 log lines for a job, last 50 actions for a user, a bounded in-memory queue that never grows without limit. A Redis list with LPUSH + LTRIM gives you exactly that: push onto the head, then trim the list to the newest N, in two O(1)/O(N-window) commands.

Push newest to head. LPUSH key value prepends; the newest item is index 0.
Cap length. LTRIM key 0 N-1 keeps only indices 0..N-1, discarding the tail.
Read the window. LRANGE key 0 -1 returns the whole capped buffer, newest first.

Question. Keep the 100 most recent audit events for each pipeline run, discarding older events automatically.

Input.

Parameter	Value
Key	`audit:run:{run_id}`
Push op	`LPUSH` (newest first)
Cap	100
Trim op	`LTRIM 0 99`

Code.

import redis, json
r = redis.Redis(decode_responses=True)

CAP = 100

def record_audit(run_id: str, event: dict) -> None:
    key = f"audit:run:{run_id}"
    # Pipeline: push + trim atomically-ish in one round trip
    pipe = r.pipeline()
    pipe.lpush(key, json.dumps(event))     # newest at head
    pipe.ltrim(key, 0, CAP - 1)            # keep only newest 100
    pipe.expire(key, 86400)                # keep the buffer 1 day
    pipe.execute()

def recent_audits(run_id: str, n: int = 100) -> list[dict]:
    raw = r.lrange(f"audit:run:{run_id}", 0, n - 1)  # newest first
    return [json.loads(x) for x in raw]

Step-by-step explanation.

LPUSH prepends the new event, so index 0 is always the most recent. The complementary choice — RPUSH + LTRIM 0 99 — would keep the oldest 100, which is rarely what "recent events" means.
LTRIM key 0 99 keeps only the first 100 elements and drops the rest in one command. Without it, the list would grow unbounded — the single most common list-related memory leak.
Wrapping LPUSH + LTRIM + EXPIRE in a pipeline sends all three commands in one network round trip. (For strict atomicity you would use MULTI/EXEC or a Lua script — covered in section 5 — but a pipeline is fine here because trim-after-push is self-correcting.)
EXPIRE key 86400 bounds the buffer's lifetime to a day, so completed runs eventually vanish even if nobody deletes them.
LRANGE key 0 n-1 returns the newest-first window for display. Because the list is capped at 100, reads are cheap and predictable — no scanning a million-element list.

Output.

Operation	List state (head → tail)
push e1	`[e1]`
push e2	`[e2, e1]`
push e3..e101	`[e101 ... e2, e1]` (101 items)
after `LTRIM 0 99`	`[e101 ... e2]` (100 items; e1 dropped)

Rule of thumb. For a "last N" buffer, always pair LPUSH with LTRIM 0 N-1 in the same round trip. A list without an LTRIM is an unbounded memory leak waiting to happen.

Data engineering interview question on choosing the right structure

A senior interviewer might ask: "You need a server-side store for web sessions: read and write individual attributes cheaply, atomically bump a per-session request counter, expire a session 30 minutes after the last activity, and keep memory tight across ten million concurrent sessions. Which Redis structure do you pick, how do you key it, and how do you avoid the classic 'partial object' bugs?"

Solution Using a hash-per-session with a sliding TTL and atomic field counters

import redis
from datetime import datetime, timezone

r = redis.Redis(decode_responses=True)
SESSION_TTL = 1800  # 30 min sliding

def create(session_id: str, user_id: int, plan: str) -> None:
    key = f"sess:{session_id}"
    with r.pipeline() as pipe:
        pipe.hset(key, mapping={
            "user_id": user_id,
            "plan": plan,
            "req_count": 0,
            "created_at": datetime.now(timezone.utc).isoformat(),
        })
        pipe.expire(key, SESSION_TTL)
        pipe.execute()

def on_request(session_id: str) -> int:
    key = f"sess:{session_id}"
    with r.pipeline() as pipe:
        pipe.hincrby(key, "req_count", 1)   # atomic in-row counter
        pipe.expire(key, SESSION_TTL)       # slide the window
        count, _ = pipe.execute()
    return count

def read_field(session_id: str, field: str) -> str | None:
    return r.hget(f"sess:{session_id}", field)

Step-by-step trace.

Step	Session state	Reasoning
`create("abc", 42, "pro")`	`{user_id:42, plan:pro, req_count:0}` TTL 1800	one key, all fields, tight memory
`on_request("abc")` @ t=0	`req_count → 1`, TTL reset to 1800	atomic bump; sliding window
`read_field("abc","plan")`	returns `"pro"`	single-field O(1) read
`on_request("abc")` @ t=1200	`req_count → 2`, TTL reset	still active; never expires while used
idle 1800 s	key evicted	self-cleaning; no cron

After deployment, each session is exactly one hash key: field reads and writes are O(1), the request counter can never lose an increment because HINCRBY runs server-side, and idle sessions reclaim their own memory via TTL. Ten million sessions of a handful of small fields each stay in listpack encoding until they grow, keeping per-session overhead minimal.

Output:

Metric	Value
Keys per session	1 (hash)
Field read/write	O(1)
Counter safety	atomic (`HINCRBY`)
Expiry	30 min sliding, self-cleaning
Encoding (small row)	listpack (compact)

Why this works — concept by concept:

Hash-per-object — one key holds the whole session, so reads and writes target individual fields (HGET/HSET) without deserialising a blob, and the object expires, replicates, and deletes as one unit. This eliminates the "some fields expired, some survived" bug of spreading an object across many keys.
HINCRBY atomicity — the per-session request counter is mutated server-side in Redis's single-threaded loop, so concurrent requests can never lose an increment the way a read-modify-write in application code would.
Sliding TTL via EXPIRE on activity — resetting the TTL on every request keeps active sessions alive and lets idle ones self-destruct, replacing a session-reaper cron job with a Redis primitive.
Listpack encoding for small rows — a session with a few small fields stays in the compact listpack encoding, so ten million sessions cost far less memory than ten million multi-key objects; the structure only converts to a hashtable if a single session grows past the entry threshold.
Cost — O(1) per field operation and O(1) per session for expiry sampling; memory is O(fields) per session. Compared to a relational session table with row locks and a TTL sweep job, the Redis model is a constant-time hot path with zero cleanup infrastructure.

Data structures
Topic — data-structures
Data-structure selection and complexity problems

Practice →

Database Topic — database Database and key-value store design problems

Practice →

2. Redis Streams — the append-only log with consumer groups

`redis streams` turn Redis into a durable, replayable log with at-least-once consumer groups — the ingestion buffer you already have

The mental model in one line: a Redis Stream is an append-only, ID-ordered log (like a single-partition Kafka topic) living inside Redis — producers XADD entries that get a monotonic ms-seq ID, consumers read ranges with XRANGE/XREAD, and consumer groups (XREADGROUP + XACK + XPENDING + XCLAIM) give you at-least-once delivery with per-consumer load balancing and failure recovery, so a stream is the right shape whenever you need a buffer that survives restarts, supports replay, and fans work out to a pool of workers. Unlike pub/sub (fire-and-forget, no history) and unlike a plain list (no acknowledgement, no groups), a stream remembers what has and has not been processed.

The stream primitives.

XADD stream * field value ... appends an entry; the * tells Redis to assign the ID <millisecondsTime>-<sequence>, which is monotonically increasing and globally ordered. You can pass an explicit ID for idempotent producers.
XLEN / XRANGE / XREVRANGE. XLEN is O(1) length; XRANGE stream - + reads all entries in ID order; ranges accept (id for exclusive bounds and COUNT for paging.
XREAD BLOCK ms STREAMS stream $. A blocking tail read — $ means "only entries added after I started blocking". This is the simple single-consumer tail; it does not track acknowledgements.
Trimming. XADD stream MAXLEN ~ 100000 * ... caps the stream to ~100k entries (the ~ allows efficient approximate trimming); XTRIM trims after the fact. Without trimming a stream grows forever.

Consumer groups — the reliability layer.

XGROUP CREATE stream grp $ creates a consumer group starting at the current end ($) or from the beginning (0). The group tracks a last-delivered ID and a Pending Entries List (PEL) per consumer.
XREADGROUP GROUP grp consumer COUNT n STREAMS stream > delivers new (>) entries to this consumer and records them as pending (delivered-but-not-acked). Two consumers in the same group split the entries — load balancing for free.
XACK stream grp id marks an entry done and removes it from the PEL. An entry read but never acked stays pending forever — that is the at-least-once guarantee.
XPENDING / XCLAIM / XAUTOCLAIM. XPENDING inspects stuck entries (which consumer, how long idle); XCLAIM/XAUTOCLAIM reassign entries idle longer than a threshold to a healthy consumer — this is how you recover work from a crashed worker.

Stream vs pub/sub vs list — the three-way interview question.

Pub/sub (PUBLISH/SUBSCRIBE). Fire-and-forget broadcast. Zero history — a subscriber that was offline misses everything. Use it for live fan-out notifications where losing a message on disconnect is acceptable (cache invalidation broadcasts, live dashboards).
List (LPUSH/BRPOP). A durable queue, but no consumer groups and no acknowledgement — once BRPOP pops an entry, if the worker crashes before finishing, the entry is gone. Fine for best-effort work queues where a retry mechanism lives elsewhere.
Stream. Durable, ordered, replayable, with groups and acknowledgement. The right choice when you need "process every event at least once, recover from worker crashes, and be able to replay history". The cost is more moving parts (PEL management, trimming).

Common interview probes on streams.

"Why a stream over a list for ingestion?" — acknowledgement + consumer groups + replay.
"How do you recover a crashed consumer's in-flight work?" — XAUTOCLAIM idle entries to a live consumer.
"How do you stop a stream growing forever?" — MAXLEN ~ capped XADD or a periodic XTRIM.
"Stream vs Kafka?" — a stream is a single-node, single-partition-per-key log; Kafka is a distributed, partitioned, replicated log. Streams win for "I already run Redis and need a buffer"; Kafka wins for multi-consumer, multi-TB, cross-datacentre.

Worked example — ingesting events with `XADD` and a capped stream

Detailed explanation. The producer side of a stream is a single command. You append structured events (field/value pairs) and let Redis assign ordered IDs. The one thing you must not forget is MAXLEN — a stream without a cap grows until it eats all memory. Approximate trimming (MAXLEN ~ N) is nearly free because Redis only trims at listpack-node boundaries.

Append. XADD events * type click user 42 ts 1725545000 — * assigns the ID.
Cap. MAXLEN ~ 1000000 keeps roughly the last million entries.
Ordering. IDs are ms-seq, monotonic, so XRANGE always returns time order.

Question. Write a producer that appends click events to a capped stream and returns the assigned entry ID.

Input.

Parameter	Value
Stream key	`events:clicks`
Fields	`type`, `user`, `url`, `ts`
Cap	`MAXLEN ~ 1000000`
ID assignment	`*` (server-assigned)

Code.

import redis, time
r = redis.Redis(decode_responses=True)

def emit_click(user_id: int, url: str) -> str:
    entry_id = r.xadd(
        "events:clicks",
        {"type": "click", "user": user_id, "url": url, "ts": int(time.time())},
        maxlen=1_000_000,     # cap the stream ...
        approximate=True,     # ... with cheap approximate trimming (the '~')
    )
    return entry_id   # e.g. "1725545000123-0"

# Inspect
def stream_stats():
    return {
        "length": r.xlen("events:clicks"),
        "first":  r.xrange("events:clicks", count=1),
        "last":   r.xrevrange("events:clicks", count=1),
    }

Step-by-step explanation.

XADD with * assigns the entry an ID of <ms>-<seq>. Two entries added in the same millisecond get sequence -0, -1, … so ordering is total even under bursts.
maxlen=1_000_000, approximate=True emits MAXLEN ~ 1000000. The ~ lets Redis trim whole macro-nodes rather than counting to an exact boundary — trimming becomes O(1) amortised instead of O(entries-removed).
The event is stored as field/value pairs, not a serialised blob, so consumers can read individual fields and downstream tooling (XRANGE) shows a readable structure.
XLEN is O(1) — Redis tracks the length — so monitoring stream depth is cheap. XRANGE ... COUNT 1 and XREVRANGE ... COUNT 1 cheaply fetch the oldest and newest entries for lag calculations.
The producer knows nothing about consumers. Any number of consumer groups can attach later and read from 0 (all history still in the stream) or $ (only new entries) — the producer path is unchanged.

Output.

Call	Result
`emit_click(42, "/pricing")`	`"1725545000123-0"`
`emit_click(42, "/docs")`	`"1725545000481-0"`
`XLEN events:clicks`	`2` (then grows)
after 1.2M adds	length capped near `1000000`

Rule of thumb. Every XADD in production carries a MAXLEN ~ cap. An uncapped stream is an unbounded memory leak; approximate trimming makes the cap essentially free.

Worked example — a consumer-group worker loop with `XREADGROUP` and `XACK`

Detailed explanation. The consumer side is where streams earn their keep. A consumer group gives each worker a slice of new entries, tracks what each worker has in flight (the PEL), and holds unacknowledged entries until you XACK. The canonical loop: read new entries with >, process, ack. Add a second phase that first drains anything already pending for this consumer (ID 0) so a restarted worker finishes its in-flight work before taking new work.

Create the group once. XGROUP CREATE events:clicks loaders $ MKSTREAM.
Read new work. XREADGROUP GROUP loaders w1 COUNT 100 BLOCK 5000 STREAMS events:clicks >.
Ack on success. XACK events:clicks loaders <id> per processed entry.

Question. Implement a worker in the loaders group that processes click events in batches and acknowledges only after the sink write succeeds.

Input.

Parameter	Value
Stream	`events:clicks`
Group	`loaders`
Consumer	`w1`
Batch	`COUNT 100`, `BLOCK 5000` ms

Code.

import redis
r = redis.Redis(decode_responses=True)

STREAM, GROUP, CONSUMER = "events:clicks", "loaders", "w1"

def ensure_group():
    try:
        r.xgroup_create(STREAM, GROUP, id="$", mkstream=True)
    except redis.ResponseError as e:
        if "BUSYGROUP" not in str(e):   # group already exists is fine
            raise

def run_worker():
    ensure_group()
    while True:
        # '>' = entries never delivered to any consumer in this group
        resp = r.xreadgroup(GROUP, CONSUMER, {STREAM: ">"},
                            count=100, block=5000)
        if not resp:
            continue                      # timed out; loop and block again
        _, entries = resp[0]
        for entry_id, fields in entries:
            try:
                write_to_sink(fields)     # e.g. buffer into a warehouse loader
                r.xack(STREAM, GROUP, entry_id)   # ack ONLY after success
            except Exception:
                # do NOT ack: entry stays in the PEL and will be reclaimed
                # by XAUTOCLAIM (see next example) or retried on restart
                log_failure(entry_id, fields)

Step-by-step explanation.

XGROUP CREATE ... $ MKSTREAM creates the group at the current end (only new entries) and creates the stream if it does not exist. Catching BUSYGROUP makes group creation idempotent across worker restarts.
XREADGROUP ... STREAMS events:clicks > delivers new entries to w1 and records them in w1's PEL. If a second worker w2 runs the same loop, Redis splits entries between them — automatic load balancing with no coordinator.
BLOCK 5000 makes the read wait up to 5 seconds for new entries instead of busy-looping, so an idle worker costs almost nothing.
The worker XACKs each entry only after write_to_sink succeeds. This is the at-least-once contract: if the process crashes between read and ack, the entry stays pending and is redelivered — never silently lost.
On an exception, the worker deliberately does not ack. The entry remains in the PEL, visible to XPENDING, and is later reclaimed by XAUTOCLAIM (next example) — so a bad sink write becomes a retry, not a data loss.

Output.

Phase	PEL for w1	Action
read 100 new	100 pending	process batch
sink ok for 98	2 pending	98 acked, 2 failed
next loop	2 pending + new	failures await claim/retry
crash + restart	2 still pending	redelivered via `0`/autoclaim

Rule of thumb. Acknowledge with XACK only after the downstream write commits. An entry read but unacked is your safety net — the stream will redeliver it, which is exactly what at-least-once means.

Worked example — recovering a dead consumer with `XAUTOCLAIM`

Detailed explanation. When a worker crashes mid-batch, its in-flight entries sit in the PEL forever unless someone claims them. XAUTOCLAIM (Redis 6.2+) scans the group's pending entries, finds those idle longer than a threshold, and reassigns them to a healthy consumer in one command — the modern replacement for the older XPENDING + XCLAIM dance.

Detect. Entries idle > min-idle-time belong to a stalled consumer.
Reassign. XAUTOCLAIM stream grp new_consumer min_idle 0 transfers them.
Process + ack. The rescuer processes and acks reclaimed entries like normal work.

Question. Add a recovery routine that reclaims entries idle for more than 60 seconds and re-processes them under a live consumer.

Input.

Parameter	Value
Min idle	60000 ms
Rescuer consumer	`w1`
Start cursor	`0-0`
Batch	`COUNT 100`

Code.

import redis
r = redis.Redis(decode_responses=True)
STREAM, GROUP, CONSUMER = "events:clicks", "loaders", "w1"

def reclaim_stalled(min_idle_ms: int = 60_000):
    cursor = "0-0"
    while True:
        # Claim entries idle > min_idle_ms for THIS consumer to finish
        cursor, claimed, _deleted = r.xautoclaim(
            STREAM, GROUP, CONSUMER,
            min_idle_time=min_idle_ms,
            start_id=cursor,
            count=100,
        )
        for entry_id, fields in claimed:
            try:
                write_to_sink(fields)
                r.xack(STREAM, GROUP, entry_id)
            except Exception:
                log_failure(entry_id, fields)  # leave pending; try again later
        if cursor == "0-0":     # cursor wraps to 0-0 when the scan completes
            break

def pending_overview():
    # summary: total pending, min/max id, per-consumer counts
    return r.xpending(STREAM, GROUP)

Step-by-step explanation.

XPENDING (summary form) tells you the group's health: how many entries are pending and which consumers hold them. A consumer with a growing pending count that never shrinks is a crashed or stuck worker.
XAUTOCLAIM ... min_idle_time=60000 transfers only entries that have been pending (unacked) for over 60 seconds — long enough to be confident the original consumer is not merely slow. This avoids stealing work from a healthy-but-busy worker.
The command returns a cursor; you loop, passing the cursor back, until it wraps to 0-0, meaning the whole PEL has been scanned. This pages through large backlogs without loading them all at once.
Reclaimed entries are processed and acked exactly like fresh entries. The delivery count for each entry increments, so you can route entries redelivered too many times (a "poison message") to a dead-letter stream.
The net effect: a crashed worker's in-flight work is automatically picked up by a survivor within the idle threshold, so the pipeline self-heals without operator intervention or lost events.

Output.

State	Pending (w2, crashed)	After `XAUTOCLAIM` by w1
before	40 entries, idle 90s	—
claim	—	40 moved to w1
process + ack	—	40 acked, PEL drains
poison (delivery > 5)	—	routed to `events:clicks:dead`

Rule of thumb. Run an XAUTOCLAIM sweep on a timer in every consumer-group deployment. Pending entries with a high delivery count are poison messages — dead-letter them instead of retrying forever.

Data engineering interview question on Redis Streams

A senior interviewer might ask: "You need a durable ingestion buffer between a high-throughput producer and a warehouse loader that occasionally goes down for maintenance. Events must not be lost while the loader is offline, multiple loader workers should share the load, and a crashed worker's in-flight events must be retried. Design this with Redis Streams — the produce path, the consumer group, failure recovery, trimming, and idempotency at the sink."

Solution Using a capped stream + consumer group + XAUTOCLAIM + idempotent sink

import redis, json
r = redis.Redis(decode_responses=True)
STREAM, GROUP = "ingest:events", "warehouse-loaders"

# --- producer ---
def produce(event: dict) -> str:
    return r.xadd(STREAM, {"body": json.dumps(event)},
                  maxlen=5_000_000, approximate=True)

# --- consumer worker ---
def worker(name: str):
    try:
        r.xgroup_create(STREAM, GROUP, id="0", mkstream=True)
    except redis.ResponseError:
        pass
    while True:
        # 1) finish this worker's own pending work first (crash recovery)
        _drain(name, "0")
        # 2) then take new work
        _drain(name, ">")
        # 3) reclaim entries abandoned by dead workers
        _reclaim(name)

def _drain(name: str, cursor: str):
    resp = r.xreadgroup(GROUP, name, {STREAM: cursor}, count=200, block=2000)
    if not resp:
        return
    for entry_id, fields in resp[0][1]:
        event = json.loads(fields["body"])
        if sink_idempotent(entry_id, event):   # dedupe by entry_id at the sink
            r.xack(STREAM, GROUP, entry_id)

def _reclaim(name: str):
    cursor = "0-0"
    while True:
        cursor, claimed, _ = r.xautoclaim(STREAM, GROUP, name,
                                          min_idle_time=60_000,
                                          start_id=cursor, count=200)
        for entry_id, fields in claimed:
            event = json.loads(fields["body"])
            if sink_idempotent(entry_id, event):
                r.xack(STREAM, GROUP, entry_id)
        if cursor == "0-0":
            break

Step-by-step trace.

Step	Input	Effect
producer `XADD`	`{order:42}` → stream, capped 5M	durable, ordered, id `t-0`
loader offline	producer keeps `XADD`-ing	events accumulate in stream (bounded by MAXLEN)
worker reads `>`	200 entries → PEL	delivered, not yet acked
sink write ok	`sink_idempotent` dedupes by id	`XACK` removes from PEL
worker crash	200 entries stuck in PEL, idle 60s	survivor `XAUTOCLAIM`s them
survivor re-processes	idempotent sink skips already-written	`XACK`; no double-load

After deployment, the producer never blocks on the loader — events land in the capped stream regardless of loader health. Multiple loader workers in one group split the backlog; when one crashes, its in-flight entries are reclaimed after 60 seconds and retried; and because the sink dedupes by the stream entry ID, at-least-once delivery is made effectively-once at the warehouse.

Output:

Metric	Value
Delivery guarantee	at-least-once (effectively-once with idempotent sink)
Loader-outage tolerance	until stream hits `MAXLEN ~ 5M`
Load balancing	automatic across group consumers
Crash recovery	`XAUTOCLAIM` after 60 s idle
Ordering	total, by stream ID

Why this works — concept by concept:

Capped stream as the buffer — XADD ... MAXLEN ~ 5M decouples producer from consumer: the producer writes at full speed even while the loader is down, and approximate trimming bounds memory. The stream is the durable, replayable backlog a list or pub/sub could never be.
Consumer group load balancing — XREADGROUP ... > hands each worker a disjoint slice of new entries with no external coordinator, so adding a worker linearly increases throughput.
Pending-first drain (0 before >) — reading cursor 0 first makes a restarted worker finish its own in-flight entries before taking new work, closing the window where a crash could strand entries.
XAUTOCLAIM recovery — entries idle past 60 seconds are reassigned to a live worker, so a crashed consumer's work is retried automatically; delivery-count tracking lets you dead-letter poison messages.
Idempotent sink keyed by entry ID — because at-least-once can redeliver, the sink deduplicates on the immutable stream ID, upgrading the guarantee to effectively-once without distributed transactions.
Cost — O(1) XADD, O(1) XACK, O(log N) internal indexing, memory bounded by MAXLEN. Compared to standing up Kafka for a buffer you could serve from the Redis you already run, this is dramatically less operational surface for the same at-least-once semantics.

Streaming
Topic — streaming
Streaming ingestion and consumer-group problems

Practice →

Database Topic — database Durable buffer and delivery-semantics problems

Practice →

3. Sorted sets — leaderboards, time-series windows, rate limiting

`sorted sets` are the Swiss-army structure — one score-ordered set solves leaderboards, time-series windows, and rate limiting

The mental model in one line: a sorted set stores unique members each tagged with a floating-point score and keeps them permanently ordered by that score, so ZADD inserts in O(log N), ZRANGE/ZREVRANGE read ranked slices, ZRANK finds a member's position, and ZRANGEBYSCORE/ZREMRANGEBYSCORE operate on score windows — and because the score can be anything (points for a leaderboard, a Unix timestamp for a time-series or rate-limit window, a priority for a queue), the same structure covers a startling range of data-engineering problems. When an interviewer asks for a leaderboard, a sliding-window rate limiter, or a "give me events in this time range" index, the answer is almost always a sorted set.

The sorted-set primitives.

ZADD key score member inserts or updates a member's score in O(log N). ZADD ... GT/LT update only if the new score is greater/less; ZADD ... NX/XX insert-only / update-only.
ZRANGE / ZREVRANGE read members by rank (position) — ZREVRANGE lb 0 9 WITHSCORES is "top 10 with scores". ZRANGEBYSCORE / ZRANGEBYLEX read by score / lexical window.
ZRANK / ZREVRANK / ZSCORE. A member's position (ascending / descending) and its score — O(log N). This is how you show "you are rank 4,217".
ZINCRBY key delta member atomically bumps a member's score — the leaderboard "add points" primitive.
ZREMRANGEBYSCORE / ZREMRANGEBYRANK delete a score / rank window — the trimming primitive behind sliding windows and capped leaderboards. ZCARD counts members; ZCOUNT counts a score range.

The three canonical uses — one structure, three problems.

Leaderboards. Member = player, score = points. ZINCRBY to award points, ZREVRANGE 0 N-1 WITHSCORES for the top N, ZREVRANK for a player's rank. O(log N) writes and O(log N + M) top-M reads at any scale.
Time-series windows. Member = event ID, score = event timestamp (ms). ZADD to record, ZRANGEBYSCORE key from to to fetch a time window, ZREMRANGEBYSCORE key -inf (cutoff to expire old events. A per-entity sorted set becomes a queryable, self-pruning time index.
Sliding-window rate limiting. Member = a unique request marker, score = request timestamp. Drop entries older than now - window, count what remains, allow if under the limit, record the new request — all atomically. This is the textbook precise rate limiter, and it is a sorted set.

Leaderboard subtleties interviewers probe.

Tie-breaking. Equal scores are ordered lexicographically by member. If you need "earlier submission wins ties", encode a timestamp into the score's fractional part or into the member.
Descending display. Redis stores ascending; use ZREVRANGE/ZREVRANK for high-score-first, or store negated scores if you frequently need ascending semantics.
Bounded leaderboards. To keep only the top 10,000, periodically ZREMRANGEBYRANK key 0 -10001 (drop everyone below rank 10,000). This caps memory on a viral leaderboard.

Common interview probes on sorted sets.

"How do you get a user's rank in O(log N)?" — ZREVRANK; never scan.
"How do you build a sliding-window rate limiter?" — sorted set scored by timestamp + ZREMRANGEBYSCORE + ZCARD, atomically.
"How do you fetch events between two times?" — score by timestamp, ZRANGEBYSCORE.
"How do you cap a leaderboard's memory?" — ZREMRANGEBYRANK to drop low ranks.

Worked example — a top-N leaderboard with `ZINCRBY` and `ZREVRANGE`

Detailed explanation. A leaderboard is the poster child for sorted sets. Awarding points is ZINCRBY (atomic, no lost updates); reading the top N is ZREVRANGE ... WITHSCORES; showing a player their rank is ZREVRANK. Every operation is O(log N) or better, so a leaderboard with ten million players is as fast as one with ten.

Award points. ZINCRBY game:lb 50 player:42.
Top N. ZREVRANGE game:lb 0 9 WITHSCORES.
My rank. ZREVRANK game:lb player:42 (0-based; add 1 for display).

Question. Implement award-points, top-10, and get-my-rank for a game leaderboard.

Input.

Parameter	Value
Key	`game:lb`
Member	`player:{id}`
Score	cumulative points
Top-N read	`ZREVRANGE 0 9 WITHSCORES`

Code.

import redis
r = redis.Redis(decode_responses=True)
LB = "game:lb"

def award(player_id: int, points: int) -> float:
    # Atomic: no read-modify-write race, returns the new total score
    return r.zincrby(LB, points, f"player:{player_id}")

def top_n(n: int = 10) -> list[tuple[str, float]]:
    # Highest score first, with scores
    return r.zrevrange(LB, 0, n - 1, withscores=True)

def my_rank(player_id: int) -> int | None:
    rank = r.zrevrank(LB, f"player:{player_id}")
    return None if rank is None else rank + 1   # 1-based for humans

def cap_leaderboard(keep_top: int = 10_000):
    # Drop everyone below the top `keep_top` to bound memory
    r.zremrangebyrank(LB, 0, -keep_top - 1)

Step-by-step explanation.

ZINCRBY adds points to a player's score, creating the member at that score if absent, and returns the new total — atomically, so two concurrent point awards never lose one.
ZREVRANGE LB 0 9 WITHSCORES returns the ten highest-scoring members with their scores in one O(log N + 10) call. The set is always sorted, so there is no sort step at read time.
ZREVRANK returns a player's 0-based position from the top in O(log N); adding 1 gives the human-facing rank. This is how "You are #4,217 of 2.3M" renders instantly.
cap_leaderboard uses ZREMRANGEBYRANK LB 0 -(keep_top+1) to delete every member below the top keep_top. On a viral leaderboard this bounds memory without affecting the visible ranks.
Because scores are floats, you can encode tie-breaks: score = points - timestamp*1e-13 makes an earlier achiever outrank a later one at equal points, since the tiny timestamp term breaks ties deterministically.

Output.

Call	Result
`award(42, 50)`	`50.0`
`award(42, 30)`	`80.0`
`award(7, 100)`	`100.0`
`top_n(2)`	`[("player:7",100.0),("player:42",80.0)]`
`my_rank(42)`	`2`

Rule of thumb. A leaderboard is a sorted set: ZINCRBY to score, ZREVRANGE WITHSCORES for the top, ZREVRANK for a position. Never fetch all members and sort in application code — that throws away the whole point of the structure.

Worked example — a precise sliding-window rate limiter

Detailed explanation. Fixed-window counters (an INCR per minute bucket) allow bursts at window edges — 100 requests at 11:59:59 and 100 more at 12:00:00 pass a "100/min" limit but are 200 requests in one second. A sliding-window limiter using a sorted set scored by timestamp is precise: it counts exactly the requests in the trailing window at every instant. The four steps — drop old, count, add new, expire — must run atomically, which a Lua script or a MULTI/EXEC transaction guarantees.

Drop old. ZREMRANGEBYSCORE key -inf (now-window) removes requests outside the window.
Count. ZCARD key is how many requests remain in the window.
Admit + record. If under the limit, ZADD key now unique_id.

Question. Build a 100-requests-per-60-seconds per-user rate limiter that is precise at window edges and atomic under concurrency.

Input.

Parameter	Value
Key	`rl:user:{id}`
Limit	100
Window	60 s
Score	request timestamp (ms)

Code.

import redis, time, uuid
r = redis.Redis(decode_responses=True)

# Atomic sliding-window limiter as a single Lua script (runs server-side)
_LUA = """
local key    = KEYS[1]
local now    = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit  = tonumber(ARGV[3])
local member = ARGV[4]

-- 1) drop entries older than the window
redis.call('ZREMRANGEBYSCORE', key, '-inf', now - window)
-- 2) count what remains
local count = redis.call('ZCARD', key)
if count < limit then
    -- 3) admit: record this request
    redis.call('ZADD', key, now, member)
    redis.call('PEXPIRE', key, window)   -- self-clean idle keys
    return 1        -- allowed
else
    redis.call('PEXPIRE', key, window)
    return 0        -- rejected
end
"""
_check = r.register_script(_LUA)

def allow_request(user_id: int, limit: int = 100, window_ms: int = 60_000) -> bool:
    now = int(time.time() * 1000)
    member = f"{now}-{uuid.uuid4().hex[:8]}"   # unique so ZADD never collides
    return _check(keys=[f"rl:user:{user_id}"],
                  args=[now, window_ms, limit, member]) == 1

Step-by-step explanation.

The entire check runs as one Lua script, which Redis executes atomically — no other command interleaves between the "drop old", "count", and "add new" steps, so two concurrent requests can never both slip past the limit on a stale count.
Step 1, ZREMRANGEBYSCORE key -inf (now-window), evicts every request timestamped before the window start. After this line the set contains exactly the requests in the trailing 60 seconds.
Step 2, ZCARD, counts the survivors. Because old entries were just removed, this count is the precise current window occupancy — no edge-burst loophole.
Step 3 admits the request only if count < limit, recording it with ZADD key now member. The member embeds a UUID so two requests in the same millisecond are distinct members (a plain timestamp would collide and undercount).
PEXPIRE key window sets the key to expire one window after the last activity, so a user who stops making requests has their rate-limit key reclaimed automatically — no key accumulation for idle users.

Output.

Request	Window count before	Decision	Set size after
#1..#100 in 60s	0..99	allow	grows to 100
#101 at +30s	100	reject (0)	still 100
#102 at +61s	old dropped → 40	allow	41

Rule of thumb. For a precise rate limiter, use a sorted set scored by timestamp and run drop-count-add as one Lua script. Fixed-window INCR counters are cheaper but allow 2× bursts at window edges — know which precision your SLA needs.

Worked example — time-bucketed metrics with score = timestamp

Detailed explanation. A per-entity sorted set scored by event time is a lightweight time-series index: record each event at score = ts, query any time range with ZRANGEBYSCORE, and prune history with ZREMRANGEBYSCORE. This gives you "last 24 hours of events for device 7" without a time-series database, and the pruning keeps memory bounded.

Record. ZADD ts:device:7 <ts> <event_id>.
Query range. ZRANGEBYSCORE ts:device:7 <from> <to>.
Prune. ZREMRANGEBYSCORE ts:device:7 -inf (now-24h).

Question. Store device telemetry events and answer "how many events in the last hour" and "give me events between t1 and t2", pruning anything older than 24 hours.

Input.

Parameter	Value
Key	`ts:device:{id}`
Score	event timestamp (ms)
Member	event id
Retention	24 h

Code.

import redis, time
r = redis.Redis(decode_responses=True)

def record_event(device_id: int, event_id: str, ts_ms: int | None = None):
    ts = ts_ms or int(time.time() * 1000)
    key = f"ts:device:{device_id}"
    pipe = r.pipeline()
    pipe.zadd(key, {event_id: ts})
    pipe.zremrangebyscore(key, "-inf", f"({ts - 24*3600*1000}")  # prune >24h
    pipe.execute()

def count_last_hour(device_id: int) -> int:
    now = int(time.time() * 1000)
    return r.zcount(f"ts:device:{device_id}", now - 3600*1000, now)

def events_between(device_id: int, t1: int, t2: int) -> list[str]:
    return r.zrangebyscore(f"ts:device:{device_id}", t1, t2)

Step-by-step explanation.

ZADD key {event_id: ts} records the event with its timestamp as the score, so the set is automatically ordered by time — no separate index needed.
The ZREMRANGEBYSCORE key -inf (now-24h on every write prunes events older than the retention horizon incrementally, so the per-device set stays bounded instead of growing forever.
count_last_hour uses ZCOUNT key (now-1h) now to count members whose score (timestamp) falls in the last hour — O(log N), no scan.
events_between uses ZRANGEBYSCORE key t1 t2 to return exactly the event IDs in an arbitrary time window, in time order. This is the "range query" a time-series store would give you, from a sorted set.
Because both the query and the prune operate on score windows, the structure doubles as a self-maintaining rolling buffer: hot recent data stays, cold data is dropped, and every read is a logarithmic range operation.

Output.

Operation	Result
record e1@t, e2@t+5m, e3@t+2h	set has 3 members
`count_last_hour` (now=t+2h5m)	`1` (only e3 in last hour)
`events_between(t, t+10m)`	`["e1","e2"]`
prune (>24h old)	old members removed on write

Rule of thumb. Score a sorted set by timestamp to get a queryable, self-pruning time index for free. Prune on write with ZREMRANGEBYSCORE so the set never outgrows your retention window.

Data engineering interview question on sorted sets

A senior interviewer might ask: "Design a per-user API rate limiter that enforces exactly 1,000 requests per hour with no burst loophole at the hour boundary, works correctly under many concurrent requests to the same user, cleans up state for idle users automatically, and adds negligible load to the primary database. Walk me through the data structure, the atomicity, and the failure behaviour if Redis restarts."

Solution Using a sorted-set sliding window driven by one atomic Lua script

import redis, time, uuid
r = redis.Redis(decode_responses=True)

_LIMITER = """
local key    = KEYS[1]
local now    = tonumber(ARGV[1])
local window = tonumber(ARGV[2])   -- ms
local limit  = tonumber(ARGV[3])
local member = ARGV[4]

redis.call('ZREMRANGEBYSCORE', key, '-inf', now - window)
local used = redis.call('ZCARD', key)
if used < limit then
    redis.call('ZADD', key, now, member)
    redis.call('PEXPIRE', key, window)
    return {1, limit - used - 1}      -- allowed, remaining
else
    redis.call('PEXPIRE', key, window)
    return {0, 0}                     -- rejected, 0 remaining
end
"""
_run = r.register_script(_LIMITER)

def rate_limit(user_id: int, limit=1000, window_ms=3_600_000):
    now = int(time.time() * 1000)
    member = f"{now}:{uuid.uuid4().hex[:8]}"
    allowed, remaining = _run(keys=[f"rl:{user_id}"],
                              args=[now, window_ms, limit, member])
    return bool(allowed), int(remaining)

Step-by-step trace.

Step	Input	Effect
req at t=0	window empty	`ZREM` (noop) → `ZCARD` 0 → admit → size 1
999 more by t=+50m	count climbs 1→999	all admitted; size 1000
req #1001 at t=+55m	`ZCARD` 1000 ≥ limit	rejected, remaining 0
req at t=+61m	entries < t+1m dropped	window shrinks → admit again
concurrent reqs	Lua runs atomically	no two admit on a stale count
Redis restart (no AOF)	key lost	limiter resets to empty (fail-open)

After deployment, each user's rate state is one sorted set keyed rl:{user_id}. The Lua script makes the drop-count-admit sequence indivisible, so concurrency cannot over-admit; the sliding window has no edge-burst loophole because old entries are removed before every count; and PEXPIRE reclaims idle users' keys. The database is never touched on the limit-check path.

Output:

Metric	Value
Precision	exact sliding window (no edge burst)
Concurrency safety	atomic (single Lua script)
Idle cleanup	`PEXPIRE` after one window
DB load on check	zero
Restart behaviour	fail-open unless AOF persists state

Why this works — concept by concept:

Sorted set scored by timestamp — each request is a member scored by its arrival time, so "requests in the last hour" is a score-range operation and pruning old requests is ZREMRANGEBYSCORE — the structure is the sliding window.
Atomic Lua script — bundling remove-count-add into one server-side script makes it indivisible under Redis's single-threaded execution, so two simultaneous requests cannot both read a stale count and both be admitted past the limit.
UUID-tagged members — embedding a UUID in the member guarantees uniqueness within the same millisecond, so ZADD never overwrites a concurrent request's entry and the count stays exact.
PEXPIRE self-cleanup — expiring the key one window after the last request means idle users cost no memory, replacing a reaper job with a TTL.
Cost — O(log N) per ZADD/ZREM where N is requests-in-window (bounded by the limit), so the check is effectively O(log limit) and constant memory per active user. Compared to a database-backed limiter with a row lock per check, this offloads the entire hot path to Redis at microsecond latency.

Data structures
Topic — data-structures
Sorted-set, ranking, and window problems

Practice →

Database Topic — database Rate-limiting and time-index design problems

Practice →

4. Probabilistic structures — HyperLogLog and Bloom filters

`hyperloglog` and Bloom filters trade exactness for constant memory — count billions of uniques in 12 KB, test membership in kilobytes

The mental model in one line: probabilistic data structures answer "how many distinct?" and "have I seen this before?" using a fixed, tiny amount of memory that does not grow with the data, at the cost of a small, bounded, tunable error — HyperLogLog estimates cardinality (unique count) in ~12 KB regardless of whether you feed it a thousand items or a billion, and a Bloom filter tests set membership in a few kilobytes with a controllable false-positive rate and zero false negatives. For data engineers these are the difference between "store two billion user IDs to count uniques" (impossible on one box) and "store a 12 KB sketch" (trivial).

HyperLogLog — cardinality in constant memory.

PFADD key element [element ...] adds elements; duplicates are absorbed — adding the same user a million times counts as one. Returns 1 if the estimated cardinality changed.
PFCOUNT key [key ...] returns the estimated number of distinct elements, with a standard error of about 0.81%. Given multiple keys, it returns the cardinality of their union without materialising it.
PFMERGE dest src1 src2 ... merges several HLLs into one — the union sketch. This is how you roll daily unique-visitor HLLs into a monthly one without re-scanning events.
The magic. A Redis HLL is capped at 12 KB and estimates cardinalities up to ~2^64. Memory is constant — it does not grow with the number of distinct items. That fixed cost is the whole point.

Bloom filters — membership in bounded memory.

BF.ADD key item / BF.MADD add items (RedisBloom module). BF.EXISTS key item tests membership.
The guarantee. A Bloom filter answers "definitely not in the set" or "probably in the set" — it has zero false negatives (if it says no, it means no) and a tunable false-positive rate (if it says yes, there is a small chance it is wrong). BF.RESERVE key error_rate capacity sizes it.
Why it fits. A Bloom filter for 100 million items at a 1% false-positive rate is roughly 120 MB — versus storing 100M full keys. It shines as a pre-filter: "have we probably seen this event / URL / email before?" before an expensive exact check.

Accuracy vs memory — the engineering decision.

When approximate is correct. Dashboards ("~2.1M daily uniques"), cache-miss guards, dedup pre-filters, and "roughly how big" analytics rarely need exactness. A 0.81% error on a unique-visitor chart is invisible.
When it is not. Billing, compliance counts, and anything where a single miscount has legal or financial consequences need exact structures. Never bill customers from a HyperLogLog.
The false-positive contract. A Bloom "yes" must be safe to be occasionally wrong. Using it as a cache-miss filter is fine (a false "seen" just triggers a redundant check); using it as the sole gate on "have we already charged this card" is not.

Common interview probes.

"Count daily unique visitors across two billion events cheaply." — HyperLogLog, one per day, PFCOUNT.
"How much memory does HLL use?" — ~12 KB, constant, ~0.81% error.
"Roll daily uniques into monthly without double-counting." — PFMERGE the daily HLLs.
"Difference between a Bloom filter and a set?" — Bloom is constant-ish memory with false positives and no deletes (standard Bloom); a set is exact but O(n) memory.

Worked example — daily unique visitors with HyperLogLog

Detailed explanation. Counting unique visitors exactly means storing every distinct visitor ID — billions of them. HyperLogLog replaces that with a 12 KB sketch per day: PFADD every visitor (duplicates are free), PFCOUNT for the estimate. The memory does not grow with traffic, so a site with a billion daily events uses the same 12 KB as one with a thousand.

Add. PFADD uv:2026-09-05 user:42.
Count. PFCOUNT uv:2026-09-05 → ~unique visitors that day.
Union. PFCOUNT uv:2026-09-05 uv:2026-09-06 → uniques across both days, de-duplicated.

Question. Track daily unique visitors and report both a single day and a two-day de-duplicated total.

Input.

Parameter	Value
Key pattern	`uv:{yyyy-mm-dd}`
Add op	`PFADD`
Count op	`PFCOUNT`
Memory per day	≤ 12 KB

Code.

import redis
from datetime import date
r = redis.Redis(decode_responses=True)

def track_visit(user_id: int, day: str | None = None):
    key = f"uv:{day or date.today().isoformat()}"
    # PFADD absorbs duplicates: same user added twice counts once
    r.pfadd(key, f"user:{user_id}")

def uniques_for_day(day: str) -> int:
    return r.pfcount(f"uv:{day}")

def uniques_across(days: list[str]) -> int:
    # PFCOUNT over multiple keys = cardinality of the UNION (deduped)
    keys = [f"uv:{d}" for d in days]
    return r.pfcount(*keys)

Step-by-step explanation.

PFADD uv:{day} user:{id} hashes the visitor into the day's HLL sketch. Adding the same visitor again does not change the estimate — de-duplication is intrinsic, so you never store the raw IDs.
The sketch is capped at 12 KB no matter how many distinct visitors it has seen, so memory per day is constant and predictable — you can keep years of daily HLLs in a few megabytes.
PFCOUNT uv:{day} returns the estimated distinct count with ~0.81% standard error — for a dashboard reading "2,143,000 daily uniques", the true value is within a few thousand, which is invisible at that scale.
PFCOUNT key1 key2 ... computes the cardinality of the union of several sketches on the fly, so "uniques across Saturday and Sunday" correctly counts a visitor who came both days only once — something you cannot do by adding two daily counts.
Because the operations are add-and-count only, the ingestion path is a single O(1) PFADD per event — cheap enough to call on every request without touching the database.

Output.

Operation	Result
`PFADD uv:2026-09-05 user:42` (×3)	counts as 1
2.1M distinct users added	sketch still ≤ 12 KB
`PFCOUNT uv:2026-09-05`	`~2,143,118` (±~0.8%)
`PFCOUNT uv:09-05 uv:09-06`	union, deduped

Rule of thumb. For "how many distinct" at scale, use one HyperLogLog per bucket and PFCOUNT. Never store raw IDs to count uniques — a 12 KB sketch replaces gigabytes and the 0.81% error is invisible on any dashboard.

Worked example — rolling daily HLLs into weekly/monthly with `PFMERGE`

Detailed explanation. You want monthly unique visitors, but summing 30 daily counts double-counts anyone who visited on multiple days. PFMERGE combines the daily sketches into a union sketch whose PFCOUNT is the true monthly distinct count — no event re-scan, no double counting.

Merge. PFMERGE uv:2026-09 uv:2026-09-01 ... uv:2026-09-30.
Count the union. PFCOUNT uv:2026-09 → monthly uniques.
Idempotent. Re-merging is safe; HLL union is associative and commutative.

Question. Produce a monthly unique-visitor count from the daily HLLs without re-reading raw events.

Input.

Parameter	Value
Source keys	`uv:2026-09-01 .. uv:2026-09-30`
Merge op	`PFMERGE`
Dest key	`uv:2026-09`
Count	`PFCOUNT uv:2026-09`

Code.

import redis
from datetime import date, timedelta
r = redis.Redis(decode_responses=True)

def month_days(year: int, month: int) -> list[str]:
    d = date(year, month, 1)
    out = []
    while d.month == month:
        out.append(d.isoformat())
        d += timedelta(days=1)
    return out

def rollup_month(year: int, month: int) -> int:
    dest = f"uv:{year}-{month:02d}"
    srcs = [f"uv:{d}" for d in month_days(year, month)]
    # Union all daily sketches into one monthly sketch
    r.pfmerge(dest, *srcs)
    return r.pfcount(dest)   # true monthly distinct count (deduped across days)

Step-by-step explanation.

PFMERGE dest src1 ... srcN computes the union of all source HLLs into dest. Because HLL union takes the max register value across sketches, a visitor present on ten days contributes to the union exactly once.
The result PFCOUNT uv:2026-09 is the genuine monthly distinct count — not the sum of daily counts, which would over-count multi-day visitors, often by a large factor for sticky products.
The rollup reads only the 30 daily sketches (≤ 12 KB each), never the raw event log, so a month's rollup is kilobytes of I/O regardless of how many billions of events occurred.
The operation is idempotent and order-independent: re-running the merge, or merging weeks first and then months, yields the same count because HLL union is associative and commutative.
You can build a hierarchy — daily → weekly → monthly → yearly — each level a PFMERGE of the level below, giving cheap drill-down uniques at every granularity from a few kilobytes of sketches.

Output.

Operation	Result
sum of 30 daily counts	over-counts multi-day visitors
`PFMERGE uv:2026-09 <30 days>`	union sketch, ≤ 12 KB
`PFCOUNT uv:2026-09`	true monthly uniques
weekly rollups	`PFMERGE` of 7 daily keys each

Rule of thumb. Never sum daily unique counts for a monthly total — that double-counts returning visitors. PFMERGE the daily sketches and PFCOUNT the union for the true distinct count.

Worked example — a Bloom filter as a dedup / cache-miss pre-filter

Detailed explanation. Before an expensive check ("is this the first time we have seen this event ID?" or "is this URL already crawled?"), a Bloom filter gives a cheap probabilistic pre-answer. If it says "definitely not seen", you skip the expensive lookup entirely; if it says "probably seen", you do the exact check. Because it has zero false negatives, a "no" is trustworthy and safe to act on.

Reserve. BF.RESERVE seen 0.001 100000000 — 0.1% false positives, 100M capacity.
Add + test. BF.ADD seen <id>, BF.EXISTS seen <id>.
Contract. "no" is exact; "yes" is probably-yes (do the exact check).

Question. Use a Bloom filter to skip the expensive "already processed?" database lookup for events that are definitely new.

Input.

Parameter	Value
Filter key	`seen:events`
Error rate	0.001 (0.1%)
Capacity	100,000,000
Ops	`BF.ADD`, `BF.EXISTS`

Code.

import redis
r = redis.Redis(decode_responses=True)   # requires the RedisBloom module

def ensure_filter():
    try:
        r.execute_command("BF.RESERVE", "seen:events", 0.001, 100_000_000)
    except redis.ResponseError:
        pass   # already exists

def process_event(event_id: str, payload: dict) -> str:
    # 1) cheap probabilistic pre-check
    maybe_seen = r.execute_command("BF.EXISTS", "seen:events", event_id)
    if not maybe_seen:
        # definitely NEW (zero false negatives) -> skip the exact DB lookup
        handle_new_event(event_id, payload)
        r.execute_command("BF.ADD", "seen:events", event_id)
        return "processed-new"

    # 2) 'probably seen' -> confirm with the authoritative store
    if db_already_processed(event_id):
        return "skipped-duplicate"
    else:                       # rare false positive: was NOT actually seen
        handle_new_event(event_id, payload)
        r.execute_command("BF.ADD", "seen:events", event_id)
        return "processed-after-fp"

Step-by-step explanation.

BF.EXISTS is the fast path. When it returns false, the event is definitely new — Bloom filters never have false negatives — so we process it and skip the expensive database dedup lookup entirely.
The overwhelming majority of genuinely new events take this fast path, so the expensive exact check runs only for events the filter thinks it has probably seen — a small fraction of traffic plus the ~0.1% false-positive rate.
When BF.EXISTS returns true ("probably seen"), we fall back to the authoritative db_already_processed check because a Bloom "yes" can be wrong. This preserves correctness: the filter optimises the common case, the database guarantees the answer.
The rare false positive (filter says seen, database says new) is handled by processing the event and adding it — correctness is never sacrificed, only a redundant lookup is occasionally spent.
BF.RESERVE seen 0.001 100000000 sizes the filter for 100M items at 0.1% false positives — roughly 180 MB, versus storing 100M raw IDs. The pre-filter turns most dedup checks into a single in-memory bit test.

Output.

Event	`BF.EXISTS`	Path taken	Result
brand-new id	false	fast path	processed-new
true duplicate	true	DB confirms	skipped-duplicate
false positive (~0.1%)	true	DB says new	processed-after-fp
memory (100M @ 0.1%)	—	~180 MB	vs GBs of raw ids

Rule of thumb. Use a Bloom filter as a pre-filter in front of an exact store, never as the sole source of truth. Its zero-false-negative guarantee makes "no" trustworthy; treat every "yes" as "probably — go confirm".

Data engineering interview question on probabilistic structures

A senior interviewer might ask: "You need daily and monthly unique-visitor counts for a site doing two billion events a day. Storing raw visitor IDs is out of the question on memory grounds, dashboards tolerate a sub-1% error, and the monthly number must not double-count visitors who came on many days. Design the counting system, explain the memory footprint, and state exactly where an approximate answer is acceptable and where it is not."

Solution Using one HyperLogLog per day and PFMERGE rollups

import redis
from datetime import date, timedelta
r = redis.Redis(decode_responses=True)

# --- ingest: O(1) per event, constant memory per day ---
def on_event(visitor_id: str, day: str | None = None):
    key = f"uv:{day or date.today().isoformat()}"
    r.pfadd(key, visitor_id)          # duplicates absorbed; sketch <= 12 KB

# --- daily count ---
def daily(day: str) -> int:
    return r.pfcount(f"uv:{day}")     # ~0.81% standard error

# --- monthly rollup: union of daily sketches, deduped ---
def monthly(year: int, month: int) -> int:
    d, srcs = date(year, month, 1), []
    while d.month == month:
        srcs.append(f"uv:{d.isoformat()}")
        d += timedelta(days=1)
    dest = f"uv:{year}-{month:02d}"
    r.pfmerge(dest, *srcs)            # union across all days
    return r.pfcount(dest)            # true monthly uniques

Step-by-step trace.

Step	Input	Effect
2B events/day → `PFADD`	many dup visitor ids	each day's sketch stays ≤ 12 KB
`daily("2026-09-05")`	one sketch	`PFCOUNT` → ~uniques, ±0.81%
visitor on 10 days	10 daily sketches	`PFMERGE` counts them once
`monthly(2026, 9)`	30 daily sketches	union sketch → true monthly uniques
sum of daily counts (wrong)	30 numbers	over-counts returning visitors
billing needs exact	—	do NOT use HLL; use exact store

After deployment, ingestion is a single PFADD per event with constant per-day memory, so two billion events a day cost 12 KB of unique-count state per day. Dashboards read PFCOUNT for any day and PFMERGE+PFCOUNT for any week or month, always de-duplicating returning visitors. The only place the approximation is disallowed — billing or compliance counts — is served by a separate exact pipeline.

Output:

Metric	Value
Memory per day	≤ 12 KB (constant)
Ingest cost	O(1) `PFADD` per event
Count error	~0.81% standard error
Monthly dedup	correct via `PFMERGE`
Exactness boundary	dashboards yes; billing no

Why this works — concept by concept:

HyperLogLog constant memory — an HLL estimates cardinality from the distribution of hash-value leading zeros, so it needs only ~12 KB of registers regardless of how many distinct items it has seen, replacing gigabytes of raw IDs with a fixed sketch.
PFADD idempotence — adding the same visitor repeatedly leaves the estimate unchanged, so the ingest path can fire on every event without any application-side dedup.
PFMERGE union for rollups — merging daily sketches takes the register-wise maximum, which is exactly set union, so a multi-day visitor is counted once and monthly totals are correct without re-scanning events.
Bounded, known error — the ~0.81% standard error is quantified and stable, so you can state up front that dashboards tolerate it and billing does not — the accuracy budget is an explicit engineering decision, not an accident.
Cost — O(1) time per add and per count, O(1) (12 KB) memory per bucket, O(days) per rollup merge. Compared to an exact COUNT(DISTINCT) over two billion rows — which needs to hold or sort the distinct set — this is a constant-memory, constant-time alternative that is wrong by less than a percent.

Data structures
Topic — data-structures
Probabilistic structure and cardinality problems

Practice →

Database Topic — database Approximate analytics and dedup design problems

Practice →

5. Persistence, patterns and pitfalls

Redis is durable if you configure it — RDB, AOF, eviction, and atomicity decide whether your data survives a restart

The one-sentence invariant: Redis holds data in memory, so its durability, memory bounds, and atomicity are configuration choices you must make deliberately — RDB snapshots and the append-only file (AOF) decide what survives a crash, maxmemory plus an eviction policy decide what happens when you run out of RAM, and MULTI/EXEC, WATCH, and Lua scripts decide whether multi-step operations are atomic — and the difference between a Redis you can trust as a durable buffer and one that silently loses data is entirely in these settings. Treating Redis as "just a cache" is fine until you use it as a stream buffer or a rate-limit store, at which point persistence and eviction stop being optional.

Persistence — RDB vs AOF.

RDB (snapshotting). Point-in-time binary dumps written every N seconds / M changes (save 900 1). Compact, fast to load, great for backups — but a crash loses everything since the last snapshot (seconds to minutes of data). BGSAVE forks and writes in the background.
AOF (append-only file). Logs every write command; on restart Redis replays the log. Durability is governed by appendfsync: always (fsync every write — safest, slowest), everysec (fsync once a second — the standard trade-off, ≤ 1 s loss), no (let the OS decide). AOF files are larger and rewrite-compacted periodically (BGREWRITEAOF).
Hybrid (recommended default). aof-use-rdb-preamble yes writes an RDB snapshot as the AOF's base plus the command tail — fast loads and ≤ 1 s durability. Most production Redis runs AOF everysec with the RDB preamble.

Eviction — what happens at maxmemory.

Set a bound. maxmemory 8gb caps memory; without it, Redis grows until the OS OOM-kills it. Always set maxmemory in production.
Pick a policy. noeviction (reject writes when full — correct for a durable buffer you must not silently drop), allkeys-lru / allkeys-lfu (evict least-recently/frequently-used across all keys — correct for a pure cache), volatile-lru / volatile-ttl (evict only keys that have a TTL, preferring shortest TTL — correct when cache and durable data share one instance).
The trap. Running a cache-style allkeys-lru on an instance that also holds your only copy of stream/rate-limit data means Redis will happily evict that data under pressure. Separate durable and cache workloads, or use volatile-* so only expendable (TTL'd) keys are evicted.

Atomicity — transactions, scripts, and pipelines.

MULTI/EXEC. Queues commands and runs them as one atomic unit — no other client interleaves. But queued commands cannot see each other's results (no conditional logic mid-transaction).
WATCH (optimistic locking). WATCH key before MULTI; if the key changes before EXEC, the transaction aborts — the check-and-set primitive for "increment only if unchanged".
Lua scripts (EVAL). A script runs atomically server-side and can branch on intermediate results — the right tool when you need read-decide-write in one indivisible step (the rate limiter in section 3).
Pipelining is not a transaction. A pipeline batches commands into one round trip for throughput, but they are not atomic — other clients' commands can interleave. Use pipelines for speed, MULTI/Lua for atomicity.

The pitfalls that page you at 3 AM.

Big keys. A single 5 GB list or a hash with 50M fields makes every operation on it slow and blocks the single-threaded server; DEL of a huge key can stall Redis (use UNLINK for async free). Keep collections bounded.
Hot keys. One key taking the majority of traffic can saturate a single core / shard. Shard hot counters across N sub-keys and sum on read.
KEYS in production. KEYS pattern scans the entire keyspace and blocks the server — never run it in prod. Use SCAN (cursor-based, incremental) instead.
TTL-less keys. Keys written without a TTL live forever and are the most common cause of slow memory creep. Default to a TTL unless the data is genuinely permanent.
Cache stampede. When a hot cached key expires, thousands of requests miss simultaneously and hammer the database. Mitigate with a short lock ("one rebuilder"), a probabilistic early refresh, or staggered TTLs.

Common interview probes on operations.

"RDB vs AOF — which and why?" — hybrid (AOF everysec + RDB preamble) for durability + fast load.
"What eviction policy for a mixed cache + durable instance?" — volatile-* so only TTL'd keys go.
"How do you make read-decide-write atomic?" — Lua script (or WATCH/MULTI).
"Why is KEYS dangerous?" — O(N) blocking scan; use SCAN.

Worked example — choosing RDB vs AOF for a durable buffer

Detailed explanation. When Redis backs a stream ingestion buffer, a restart must not lose acknowledged-but-not-yet-loaded events. RDB alone loses everything since the last snapshot; AOF everysec caps loss at one second; the hybrid gives fast restart loads plus that one-second bound. This example shows the config and the reasoning.

RDB only. Fast load, up to save-interval data loss on crash.
AOF everysec. ≤ 1 s loss, slower load without a preamble.
Hybrid. RDB preamble + AOF tail = fast load and ≤ 1 s loss.

Question. Configure a Redis instance used as a durable stream buffer for ≤ 1 second of data loss and fast restart.

Input.

Requirement	Choice
Max data loss on crash	≤ 1 s
Restart load speed	fast
Persistence mode	AOF everysec + RDB preamble
Eviction	`noeviction` (must not drop buffer)

Code.

# redis.conf — durable buffer profile

# --- AOF for ≤ 1s durability ---
appendonly            yes
appendfsync           everysec        # fsync once per second (≤ 1s loss)
aof-use-rdb-preamble  yes             # RDB snapshot as AOF base -> fast load

# --- RDB snapshots as backups (also the AOF preamble source) ---
save 900 1
save 300 100
save 60  10000

# --- memory: never silently drop the buffer ---
maxmemory             12gb
maxmemory-policy      noeviction      # reject writes when full, do NOT evict

# --- safety ---
rename-command        KEYS ""         # disable KEYS in production

Step-by-step explanation.

appendonly yes + appendfsync everysec turns on the append-only log with a one-second fsync cadence, capping worst-case data loss at the last second of writes — the standard durability/throughput trade-off.
aof-use-rdb-preamble yes makes Redis write a compact RDB snapshot as the base of the AOF file and append recent commands after it, so restart loads the snapshot fast and replays only the short command tail — you get AOF durability without slow full-log replay.
maxmemory-policy noeviction is the critical choice for a durable buffer: when memory is full, Redis rejects new writes with an error rather than silently evicting buffered events. The producer sees backpressure instead of data loss.
The save lines keep RDB snapshots as an independent backup channel and feed the AOF preamble. They are cheap insurance and enable point-in-time restores.
rename-command KEYS "" disables the KEYS command entirely, removing the single most common way an operator accidentally blocks the whole server during an incident.

Output.

Failure	RDB only	AOF everysec	Hybrid
crash	lose since last snapshot	lose ≤ 1 s	lose ≤ 1 s
restart load	fast	slow (replay log)	fast (preamble)
memory full	evict/OOM	evict/OOM	reject (noeviction)

Rule of thumb. For anything you cannot afford to lose on restart, run AOF everysec with the RDB preamble and maxmemory-policy noeviction. RDB-only is for caches and backups, not for buffers that hold your only copy of the data.

Worked example — cache-aside with stampede protection

Detailed explanation. The cache-aside pattern (read cache; on miss, load from DB and populate cache) has a classic failure: when a hot key expires, thousands of concurrent requests all miss and all hit the database at once — a stampede that can topple the DB. The fix is to let exactly one request rebuild the value while others briefly wait or serve stale, using a short SET NX lock.

Read. GET key; hit → return.
Miss with lock. SET lock:key 1 NX EX 5 — only one request wins the lock and rebuilds.
Losers. Wait briefly and re-read, or serve the last-known value.

Question. Implement cache-aside with single-flight rebuild so a hot-key expiry cannot stampede the database.

Input.

Parameter	Value
Value key	`cache:{id}`
Lock key	`lock:{id}` (NX, EX 5)
Value TTL	300 s
Loser behaviour	short backoff + re-read

Code.

import redis, time, json
r = redis.Redis(decode_responses=True)

def get_with_stampede_guard(obj_id: str, ttl: int = 300):
    key, lock = f"cache:{obj_id}", f"lock:{obj_id}"

    cached = r.get(key)
    if cached is not None:
        return json.loads(cached)              # fast path: cache hit

    # Cache miss: try to become the single rebuilder
    if r.set(lock, "1", nx=True, ex=5):        # only ONE request wins
        try:
            value = load_from_db(obj_id)        # expensive
            r.set(key, json.dumps(value), ex=ttl)
            return value
        finally:
            r.delete(lock)
    else:
        # Lost the lock: another request is rebuilding. Back off + re-read.
        for _ in range(10):
            time.sleep(0.05)
            cached = r.get(key)
            if cached is not None:
                return json.loads(cached)
        return load_from_db(obj_id)             # last-resort fallback

Step-by-step explanation.

The fast path is a plain GET; on a hit it returns immediately with no locking overhead — the overwhelming majority of requests take this path.
On a miss, SET lock NX EX 5 atomically grants the rebuild lock to exactly one request. NX means "only if absent", so concurrent missers race and only the winner proceeds to the expensive load_from_db.
The winner loads from the database, repopulates the cache with a fresh TTL, and releases the lock in a finally so a crash mid-rebuild cannot leave the lock stuck (the EX 5 also auto-expires it as a backstop).
The losers do not hit the database. They back off in short sleeps and re-read the cache, picking up the value the winner just wrote — so N concurrent missers produce exactly one database load, not N.
The EX 5 lock TTL and the bounded retry loop guarantee liveness: even if the rebuilder dies, the lock frees within 5 seconds and a fallback load prevents an indefinite stall.

Output.

Scenario	DB loads	Behaviour
cache hit	0	immediate return
single miss	1	rebuild + populate
1000 concurrent misses	1	one rebuilds, 999 re-read
rebuilder crash	≤ 1 + fallback	lock expires in 5 s

Rule of thumb. Protect hot cached keys with a single-flight SET NX rebuild lock. Without it, a popular key's expiry turns into a synchronized thundering herd against your database.

Worked example — atomic read-decide-write with a Lua script

Detailed explanation. Some operations must read a value, decide based on it, and write — atomically. MULTI/EXEC cannot branch on intermediate results, and WATCH retries can starve under contention. A Lua script runs on the server as one indivisible unit and can branch, making it the cleanest tool for conditional atomic mutations like "decrement inventory only if stock remains".

Read. GET stock:{sku} inside the script.
Decide. Branch: enough stock or not.
Write. DECRBY only on the success branch — all atomic.

Question. Implement "reserve K units of a SKU only if at least K are in stock" atomically.

Input.

Parameter	Value
Key	`stock:{sku}`
Reserve amount	K
Success	`DECRBY` and return remaining
Failure	return -1, no change

Code.

import redis
r = redis.Redis(decode_responses=True)

_RESERVE = """
local stock = tonumber(redis.call('GET', KEYS[1]) or '0')
local want  = tonumber(ARGV[1])
if stock >= want then
    return redis.call('DECRBY', KEYS[1], want)   -- reserve, return remaining
else
    return -1                                     -- insufficient stock
end
"""
_reserve = r.register_script(_RESERVE)

def reserve(sku: str, qty: int) -> int:
    # returns remaining stock, or -1 if not enough
    return int(_reserve(keys=[f"stock:{sku}"], args=[qty]))

Step-by-step explanation.

The whole read-decide-write runs inside one Lua script, which Redis executes atomically — no other client can change the stock between the GET and the DECRBY, eliminating the oversell race entirely.
The script reads the current stock, coercing a missing key to 0, then compares against the requested quantity. All logic runs server-side, so there is no network round trip between read and decide.
On the success branch, DECRBY reserves the units and returns the new remaining count in the same atomic step. Two concurrent reservations cannot both see the same "enough stock" and both decrement past zero.
On the failure branch, the script returns -1 and makes no change, so an over-request is a clean rejection, not a partial or negative decrement.
This is strictly stronger than WATCH/MULTI, which would abort-and-retry under contention (wasting round trips) and can livelock on a hot SKU; the Lua approach resolves each request in one server-side pass.

Output.

Stock before	`reserve(sku, 3)`	Stock after	Return
10	ok	7	7
2	insufficient	2	-1
concurrent (10, two ×6)	one wins	4 then -1	4, -1

Rule of thumb. For read-decide-write that must be atomic and conditional, use a Lua script — it is indivisible, branch-capable, and contention-proof, unlike MULTI/EXEC (no branching) or WATCH (retry storms under contention).

Data engineering interview question on persistence and patterns

A senior interviewer might ask: "You are running one Redis instance as both a cache and the durable buffer for a rate limiter and a stream ingestion pipeline. It has been OOM-killed twice and lost rate-limit state on the last restart. Harden it: choose the persistence mode, the memory bound and eviction policy that protects durable data while still evicting cache entries, make the write path atomic, and list the pitfalls you would audit for."

Solution Using AOF + noeviction on TTL-tagged caching + Lua-atomic writes

# redis.conf — mixed cache + durable-buffer instance

appendonly            yes
appendfsync           everysec         # ≤ 1s loss for durable buffer + limiter
aof-use-rdb-preamble  yes              # fast restart load

maxmemory             16gb
maxmemory-policy      volatile-lru     # evict ONLY keys that have a TTL (caches)
                                       # durable keys carry NO ttl -> never evicted

rename-command        KEYS   ""        # no accidental full-keyspace scans
rename-command        FLUSHALL ""      # no accidental wipe

import redis
r = redis.Redis(decode_responses=True)

# Cache entries ALWAYS carry a TTL -> eligible for volatile-lru eviction
def cache_put(key: str, value: str, ttl: int = 300):
    r.set(f"cache:{key}", value, ex=ttl)

# Durable buffer / limiter keys carry NO ttl (except sliding windows) ->
# volatile-lru will never evict them, so they survive memory pressure.
def buffer_append(stream: str, fields: dict):
    r.xadd(stream, fields, maxlen=5_000_000, approximate=True)

# Atomic write path via Lua (see rate limiter / reserve examples)

Step-by-step trace.

Concern	Setting	Result
data loss on crash	AOF everysec + preamble	≤ 1 s loss, fast reload
OOM kill	`maxmemory 16gb`	bounded; no OS OOM
protect durable keys	`volatile-lru`	only TTL'd cache keys evicted
cache keys	always `SET ... EX`	evictable under pressure
durable keys	no TTL	never eviction targets
atomic writes	Lua scripts	no read-modify-write races
operator safety	`rename-command KEYS/FLUSHALL`	can't block or wipe

After hardening, the instance survives restarts with at most one second of loss, never gets OOM-killed because maxmemory bounds it, and under memory pressure sheds only the disposable cache entries (which carry TTLs) while the TTL-less stream and limiter data are ineligible for eviction. The write paths that must be race-free run as Lua scripts, and the two most dangerous commands are disabled.

Output:

Metric	Before	After
Restart data loss	total (no persistence)	≤ 1 s
OOM kills	recurring	none (bounded)
Durable data under pressure	evicted	protected (no TTL)
Cache under pressure	—	evicted (volatile-lru)
Write-path races	possible	eliminated (Lua)

Why this works — concept by concept:

AOF everysec + RDB preamble — the append-only log caps crash loss at one second while the RDB preamble makes restart loads fast, giving the durable buffer and rate limiter the persistence a cache never needed.
volatile-lru with a TTL convention — tagging only cache entries with a TTL and leaving durable keys TTL-less makes eviction target exactly the expendable data; the policy physically cannot evict the stream or limiter state.
maxmemory bound — capping memory turns "OS OOM-kills the process" (total loss) into "Redis manages its own eviction" (controlled, policy-driven), which is the difference between a graceful and a catastrophic memory event.
Lua-atomic write paths — the limiter and inventory writes run as indivisible server-side scripts, so concurrency cannot corrupt counts even while the same instance serves cache traffic.
Disabling KEYS and FLUSHALL — removing the two commands most likely to block or wipe the server during an incident closes the biggest operational footguns on a shared instance.
Cost — AOF adds ~1 s fsync latency amortised and some disk I/O; volatile-lru adds negligible bookkeeping. In return you get restart durability, OOM safety, and eviction that respects the cache/durable boundary — O(1) on the hot path throughout.

Database
Topic — database
Persistence, eviction, and atomicity design problems

Practice →

Streaming
Topic — streaming
Durable buffer and backpressure problems

Practice →

Cheat sheet — Redis for data engineers recipes

Structure-selection matrix. Counter → string + INCR. Object / row you update field-by-field → hash + HSET/HINCRBY. Last-N buffer or simple queue → list + LPUSH/LTRIM/BRPOP. De-dup / membership / tags → set + SADD/SISMEMBER. Ranked or time-windowed anything → sorted set + ZADD/ZRANGE. Durable replayable log with groups → stream + XADD/XREADGROUP. Unique count at scale → HyperLogLog + PFADD/PFCOUNT. Membership pre-filter → Bloom (BF.ADD/BF.EXISTS).
Atomic counter template. INCR key (never GET+add+SET); bucket by time in the key name for windowed counts (views:{id}:{yyyymmddHH}); arm EXPIRE key ttl only on the first increment (if count == 1) for a fixed window, or on every access for a sliding one.
Hash-per-object rule. Store a session / profile / feature row as one hash, not a serialised JSON string — HGET/HSET touch single fields in O(1), HINCRBY gives race-free in-row counters, and the object expires and replicates as a unit. Small rows stay in compact listpack encoding.
Streams consumer-group loop. XGROUP CREATE stream grp $ MKSTREAM; workers XREADGROUP GROUP grp me COUNT n BLOCK ms STREAMS stream >; process; XACK stream grp id only after the sink commits; drain your own pending with cursor 0 first on restart; run XAUTOCLAIM ... min-idle 60000 on a timer to rescue crashed workers; always XADD ... MAXLEN ~ N to cap the stream.
Sliding-window rate limiter (sorted set). One Lua script: ZREMRANGEBYSCORE key -inf (now-window) → ZCARD key → if under limit ZADD key now uuid → PEXPIRE key window. Precise at edges, atomic under concurrency, self-cleaning for idle users. Fixed-window INCR is cheaper but allows 2× edge bursts.
Leaderboard recipe. ZINCRBY lb points member to score, ZREVRANGE lb 0 N-1 WITHSCORES for the top N, ZREVRANK lb member for a rank (O(log N)); cap memory with ZREMRANGEBYRANK lb 0 -(keep+1); break score ties by encoding a timestamp into the score's fractional part.
Unique-count rollup (HLL). One HyperLogLog per bucket (uv:{day}), PFADD per event (duplicates free, ≤ 12 KB, ~0.81% error), PFCOUNT for a day, PFMERGE+PFCOUNT to roll days into weeks/months without double-counting. Never sum daily counts; never bill from an HLL.
Bloom pre-filter contract. BF.RESERVE key error_rate capacity; BF.EXISTS "no" is exact (zero false negatives) → act on it; "yes" is probably-yes → confirm against the authoritative store. Use as a cheap gate in front of an expensive dedup lookup, never as the source of truth.
Persistence default. AOF appendfsync everysec + aof-use-rdb-preamble yes = ≤ 1 s crash loss and fast restart loads. RDB-only is for caches and backups. For a buffer you must not drop, set maxmemory-policy noeviction so a full instance rejects writes instead of silently evicting.
Eviction for mixed instances. Set maxmemory always. On an instance holding both cache and durable data, use volatile-lru/volatile-ttl and give only cache keys a TTL — durable keys stay TTL-less and are never eviction targets. Pure caches use allkeys-lru/allkeys-lfu.
Atomicity rule. Pipelining = throughput, not atomicity (commands can interleave). MULTI/EXEC = atomic but no branching. WATCH = optimistic CAS (retry storms under contention). Lua EVAL = atomic and branch-capable — the tool for conditional read-decide-write (rate limits, inventory reserve).
Pitfall audit. No big keys (bound every collection; UNLINK not DEL for huge keys); shard hot keys across sub-keys; SCAN never KEYS in prod (rename-command KEYS ""); default a TTL on every key unless truly permanent; guard hot cached keys with a single-flight SET NX rebuild lock to prevent cache stampedes.

Frequently asked questions

Is Redis just a cache?

No — Redis is an in-memory data-structure server that happens to be excellent at caching. Beyond GET/SET strings, it ships hashes (object rows), lists (queues, capped buffers), sets (membership, dedup), sorted sets (leaderboards, time windows, rate limiting), streams (durable replayable logs with consumer groups), and probabilistic sketches (HyperLogLog for cardinality, Bloom filters for membership). For data engineers, the same Redis you use as a cache is often the cheapest way to build a rate limiter, a unique-visitor counter, or a durable ingestion buffer — the caching use case is just the most visible one.

Redis Streams vs Kafka — when do I pick each?

Pick Redis Streams when you already run Redis and need a durable, ordered, replayable buffer with consumer groups and at-least-once delivery — it is dramatically less operational surface than a Kafka cluster, with XADD/XREADGROUP/XACK/XAUTOCLAIM covering produce, group fan-out, acknowledgement, and crash recovery. Pick Kafka when you need a distributed, partitioned, replicated log across many brokers and datacentres, multi-terabyte retention, a large ecosystem of connectors, or hundreds of independent consumer groups at very high throughput. A Redis Stream is a single-node log (bounded by one machine's memory and MAXLEN); Kafka is a horizontally-scaled commit log. For "buffer between a producer and a warehouse loader on infrastructure I already have", streams usually win.

When should I use a sorted set instead of a plain set or list?

Use a sorted set whenever you need order by a numeric key — because the score can be anything, one structure covers three big use cases: points for a leaderboard (ZREVRANGE, ZREVRANK), a timestamp for a time-series window (ZRANGEBYSCORE, ZREMRANGEBYSCORE), and a timestamp for a sliding-window rate limiter (drop old, count, admit — atomically in Lua). If you only need membership with no order, a plain set is cheaper; if you need insertion order with no ranking or windowing, a list is simpler. Reach for the sorted set the moment "ranked", "top N", "in this time range", or "how many in the last X" enters the requirements.

How much memory does HyperLogLog use, and how accurate is it?

A Redis HyperLogLog is capped at about 12 KB and estimates cardinalities up to ~2^64 with a standard error of roughly 0.81% — and crucially that memory is constant: it does not grow whether you add a thousand distinct items or a billion. That makes it the right tool for unique counts at scale (daily/monthly unique visitors over billions of events) where storing raw IDs is infeasible. PFMERGE unions daily sketches into weekly/monthly ones without double-counting returning visitors. The catch is the ~0.8% error: perfect for dashboards and analytics, unacceptable for billing, compliance, or any count with legal or financial consequences — those need an exact structure.

RDB vs AOF — which persistence should I run?

Run the hybrid: AOF with appendfsync everysec plus aof-use-rdb-preamble yes. That caps crash data loss at about one second (AOF fsyncs writes once per second) while keeping restarts fast (the RDB preamble loads a compact snapshot, then a short command tail replays). RDB-only snapshotting is compact and fast to load but loses everything since the last snapshot on a crash — fine for a pure cache or as a backup channel, not for a durable buffer. AOF always fsyncs every write for maximum durability at a large throughput cost — reserve it for data where even one second of loss is unacceptable. For anything you cannot rebuild on restart, also set maxmemory-policy noeviction so a full instance rejects writes instead of silently dropping your data.

How do I make a multi-step Redis operation atomic?

Use a Lua script (EVAL) when the operation must read, decide, and write in one indivisible step — the script runs server-side in Redis's single-threaded loop, so nothing interleaves, and unlike MULTI/EXEC it can branch on intermediate values (the sliding-window rate limiter and the "reserve stock only if available" pattern both need this). Use MULTI/EXEC for a fixed batch of commands that must all apply atomically but need no conditional logic. Use WATCH + MULTI for optimistic check-and-set ("update only if unchanged"), accepting that it retries under contention. Do not rely on pipelining for atomicity — a pipeline only batches commands into one round trip for throughput; other clients' commands can still interleave between them.

Practice on PipeCode

Drill the data-structures practice library → for the sorted-set ranking, sliding-window, HyperLogLog, and structure-selection problems senior interviewers love.
Rehearse on the database practice library → for key-value modelling, persistence and eviction trade-offs, atomicity, and rate-limiter design.
Sharpen the ingestion axis with the streaming practice library → for Redis Streams consumer groups, at-least-once delivery, and durable-buffer scenarios.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the structure-selection matrix against real graded inputs.

Lock in Redis-beyond-caching muscle memory

Docs list the commands. PipeCode drills make you pick the structure under pressure — when a sorted set beats a fixed-window counter, when a stream beats a list, when a HyperLogLog replaces a billion stored IDs, when AOF and noeviction are the difference between a durable buffer and silent data loss. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice data-structure problems →
Practice streaming problems →

MongoDB for Data Engineers: Aggregation Pipeline & $lookup

Gowtham Potureddi — Wed, 09 Sep 2026 11:33:21 +0000

The mongodb aggregation pipeline is the piece of MongoDB that turns a raw stream of JSON-shaped documents into the grouped, joined, reshaped result a report or a downstream warehouse actually needs — and for a data engineer arriving from the relational world, it is the single skill that decides whether MongoDB feels like a black box or like a query engine you can reason about. Instead of a declarative SELECT ... GROUP BY ... JOIN, MongoDB hands you an ordered array of stages: each stage receives the documents the previous stage emitted, transforms them, and passes them on. Before you write a single stage, though, you have to understand the thing being transformed — the document model, where related data lives inside one record rather than spread across normalized tables — because the shape of your documents decides which stages you need and how expensive they are.

This guide is the walkthrough you wished existed the first time someone handed you a MongoDB URI and said "get last month's revenue per customer into the warehouse." It opens the pipeline in layers: the document data model and the embed-versus-reference decision that precedes every query, the core $match / $group / $project stages that filter, fold, and reshape the stream, the $lookup join and its $unwind companion that flatten one collection into another, the indexes that separate a three-millisecond aggregation from a three-minute collection scan, and finally $merge / $out and sharding — the primitives that let you materialize results and move them off the cluster at scale. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works. All examples use the MongoDB aggregation syntax you type into mongosh or a driver, but the mental model carries straight over to Atlas, DocumentDB, and any other document store.

When you want hands-on reps immediately after reading, drill the database practice library →, sharpen your pipeline intuition on the aggregation practice library →, and rehearse document-shape queries on the JSON practice library →.

On this page

The document data model
Aggregation pipeline stages ($match/$group/$project)
$lookup joins & $unwind
Indexing & performance
$merge/$out to a warehouse + sharding
Cheat sheet — MongoDB aggregation recipes
Frequently asked questions
Practice on PipeCode

1. The document data model

One document per entity, nested not normalized — model for how you read, not for how you store

The one-sentence invariant: MongoDB stores data as BSON documents grouped into collections, and the central modelling decision — long before you write any aggregation — is what to embed inside a document versus what to reference in another collection, a choice driven by how the data is read together, how large it can grow, and whether it is shared across many parents. A relational modeller normalizes first and joins at query time; a document modeller embeds the data that is read together and pays the join tax only where it is unavoidable. Get this decision right and most of your queries never need a $lookup at all; get it wrong and every read fans out across collections or every write rewrites a megabyte-sized document.

The axes that decide embed vs reference.

Read-together. If a parent and its children are almost always fetched in the same request — an order and its line items, a blog post and its tags — embed the children as a sub-document or array. One read returns the whole aggregate; no join, no second round trip.
Cardinality and growth. Embedding is safe for bounded one-to-few relationships (an order has a handful of line items). It is dangerous for unbounded one-to-many relationships (a user with millions of events) because a document has a hard 16 MB cap and MongoDB must rewrite the whole document on every update.
Sharing. If the same child is referenced by many parents — a customer referenced by thousands of orders — reference it by _id rather than duplicating it into every parent, or a single edit means updating thousands of copies.
Update independence. Fields updated on very different cadences (a product's price versus its lifetime view count) often belong in separate documents so a hot-write counter does not contend with cold catalogue data.

The BSON building blocks every data engineer must know.

The _id field. Every document has a unique _id; if you do not supply one, MongoDB generates a 12-byte ObjectId that embeds a timestamp, so _id is roughly insertion-ordered and can double as a coarse "created at."
BSON types. BSON extends JSON with real types: Int32, Int64, Double, Decimal128, Date, ObjectId, Boolean, arrays, and nested objects. When you extract to a warehouse, Decimal128 and Date map cleanly to NUMERIC and TIMESTAMP; a field that is sometimes a string and sometimes a number does not.
Arrays are first-class. An array field can be indexed (a multikey index) and unwound in a pipeline — this is what makes embedding line items both natural and queryable.
Schema-on-read. Collections do not enforce a schema by default. Two documents in orders can have different fields. That flexibility is a feature for application teams and a hazard for pipelines, which is why aggregation defensively coerces types.

The 2026 reality for data engineers.

The document store owns the OLTP write path. Application teams pick MongoDB for developer velocity; the data engineer inherits it as a source, not a choice. Your job is to read it efficiently and land it in a warehouse, not to relitigate the schema.
Schema drift is the recurring tax. Because there is no enforced schema, fields appear, disappear, and change type across releases. Robust pipelines pin the fields they need with $project and coerce types with $convert / $toDecimal rather than trusting the raw shape.
Optional schema validation exists but is opt-in. MongoDB supports JSON-Schema validators on a collection; mature teams enable them, but you cannot assume they are on.

What interviewers listen for.

Do you say "model for the read pattern" rather than "normalize to third normal form"? — the core document-modelling signal.
Do you name the 16 MB document cap as the hard limit on embedding unbounded arrays? — required answer.
Do you distinguish embed (read-together, bounded) from reference (shared, unbounded) with a concrete rule? — senior signal.
Do you mention ObjectId carries a timestamp and can order documents roughly by creation? — nice-to-have that flags real experience.

Worked example — embedding line items vs referencing them

Detailed explanation. The canonical modelling decision: an orders collection whose orders each have a small number of line items. The question is whether to embed the line items as an array inside the order document or to store them in a separate line_items collection referenced by order_id. For a read pattern of "show the order and all its lines," embedding wins outright.

Read pattern. The order-detail screen always shows the order plus its lines together.
Cardinality. An order has 1–50 line items — bounded and small.
Growth. Line items are written once at checkout and rarely change — no hot-write contention.

Question. Model the orders collection with embedded line items, and contrast it with the normalized three-collection alternative.

Input.

Design	Collections	Read cost for order detail	Risk
Embedded	`orders` (lines inside)	1 document read	16 MB cap if lines unbounded
Referenced	`orders`, `line_items`, `customers`	1 read + `$lookup` per child	joins on every read

Code.

// Embedded model — one document holds the whole order aggregate
db.orders.insertOne({
  _id: ObjectId(),
  customer_id: ObjectId("6650a1f2c3d4e5f601020304"),
  status: "paid",
  currency: "USD",
  total_cents: 4200,
  created_at: ISODate("2026-08-14T10:22:00Z"),
  line_items: [                       // embedded array — bounded, read-together
    { sku: "TSHIRT-BLK-M", qty: 2, unit_cents: 1500 },
    { sku: "STICKER-PACK",  qty: 1, unit_cents: 1200 }
  ]
});

// Reading the whole order aggregate is a single indexed lookup — no join
db.orders.findOne({ _id: ObjectId("6650b0000000000000000001") });

Step-by-step explanation.

The order document embeds line_items as an array of sub-documents. Everything the order-detail screen needs is in one BSON document, so the read is a single index seek on _id — no join, no second collection, no round trip.
total_cents is stored denormalized (pre-summed) rather than computed at read time. In a document store you often store the aggregate you read most, because there is no cheap SUM(line_items.qty * unit) across a separate table.
Each line item is a full sub-document with its own fields. Because arrays are first-class, you can later index line_items.sku (a multikey index) and $unwind the array in a pipeline to compute per-SKU revenue.
The referenced alternative would put lines in their own collection keyed by order_id; every order-detail read then costs a $lookup. That is the right trade only when lines are shared or unbounded — neither is true here.
The 16 MB cap is the guardrail: embedding is correct precisely because an order has at most a few dozen lines. If "line items" were "user click events," embedding would eventually blow the cap and referencing would be mandatory.

Output.

Metric	Embedded	Referenced
Reads to render order detail	1	1 + N (or 1 `$lookup`)
Writes to add a line item	rewrite 1 doc	insert 1 doc
Max safe children	~thousands (16 MB)	unbounded
Best when	read-together, bounded	shared, unbounded

Rule of thumb. Embed the data you read together and that grows only to bounded size; reference the data that is shared across parents or grows without bound. When in doubt, ask "will this array ever exceed a few thousand elements?" — if yes, reference it.

Worked example — `_id`, `ObjectId`, and BSON types you will meet in the warehouse

Detailed explanation. Before you extract MongoDB to a relational warehouse, you must understand what the values actually are. An ObjectId is not a string; a MongoDB Date is milliseconds since epoch; money stored as a floating-point Double will not reconcile to the penny. Walk through the types and how the pipeline coerces them.

_id as ObjectId. 12 bytes: a 4-byte timestamp, a 5-byte random value, and a 3-byte counter. The leading timestamp means ObjectIds sort roughly by creation time.
Money. Store as Decimal128 (or integer cents), never Double — floating point loses pennies at scale.
Dates. BSON Date is a signed 64-bit millisecond offset; it maps to TIMESTAMP cleanly.

Question. Given a raw order document, project a warehouse-friendly row: a string id, a real timestamp derived from the ObjectId, and money as a decimal.

Input.

Field	Raw BSON type	Warehouse target
`_id`	ObjectId	STRING (hex)
created (from `_id`)	ObjectId timestamp	TIMESTAMP
`total_cents`	Int64	NUMERIC
`total`	Decimal128	NUMERIC(12,2)

Code.

db.orders.aggregate([
  { $project: {
      _id: 0,
      order_id:   { $toString: "$_id" },              // ObjectId -> hex string
      created_at: { $toDate:   "$_id" },              // extract the embedded timestamp
      total:      { $toDecimal: { $divide: ["$total_cents", 100] } },
      status:     1,
      customer_id:{ $toString: "$customer_id" }
  }}
]);

Step-by-step explanation.

$toString: "$_id" converts the 12-byte ObjectId into its 24-character hex form — the stable string key a relational warehouse expects as a primary key.
$toDate: "$_id" is a MongoDB idiom: converting an ObjectId to a date extracts the 4-byte creation timestamp baked into it, giving you a free "created at" even when the document never stored one explicitly.
$divide: ["$total_cents", 100] turns integer cents into a major-unit amount, and $toDecimal wraps it so the value lands as exact NUMERIC, not a lossy binary float.
Setting _id: 0 and listing only the fields you want makes the projection explicit — the pipeline no longer depends on whatever incidental fields the application happens to write, which is the defense against schema drift.
Every extraction pipeline should end with a $project like this: it pins the contract between MongoDB's flexible documents and the warehouse's fixed columns.

Output.

order_id	created_at	total	status
6650b0…0001	2026-08-14T10:22:00Z	42.00	paid
6650b0…0002	2026-08-14T10:24:11Z	18.50	paid

Rule of thumb. Never trust raw BSON types in a warehouse feed. End every extraction pipeline with an explicit $project that $toStrings ObjectIds, $toDates where you need timestamps, and $toDecimals money — coerce at the source, not in the warehouse.

Worked example — polymorphic and versioned documents

Detailed explanation. Because collections are schema-flexible, one events collection often holds several shapes distinguished by a type field, and documents accumulate a schema_version as the application evolves. A pipeline must branch on shape rather than assume one. Walk through handling a polymorphic collection defensively.

Polymorphism. events holds click, purchase, and refund documents with overlapping but not identical fields.
Versioning. A schema_version field lets the pipeline apply the right field mapping per generation.
Defense. $ifNull and $switch normalize missing or renamed fields into one clean shape.

Question. Normalize a polymorphic events collection into a uniform (event_id, type, amount, occurred_at) shape regardless of version.

Input.

type	v1 amount field	v2 amount field	has occurred_at?
purchase	`value_cents`	`amount_cents`	v2 only
refund	`value_cents`	`amount_cents`	v2 only
click	(none)	(none)	v2 only

Code.

db.events.aggregate([
  { $addFields: {
      // Coalesce the renamed field across schema versions
      amount_cents: { $ifNull: ["$amount_cents", "$value_cents"] },
      // Backfill occurred_at from the _id timestamp for v1 docs
      occurred_at:  { $ifNull: ["$occurred_at", { $toDate: "$_id" }] }
  }},
  { $project: {
      _id: 0,
      event_id: { $toString: "$_id" },
      type: 1,
      amount: {
        $switch: {
          branches: [
            { case: { $in: ["$type", ["purchase"]] }, then: { $divide: ["$amount_cents", 100] } },
            { case: { $in: ["$type", ["refund"]]   }, then: { $multiply: [{ $divide: ["$amount_cents", 100] }, -1] } }
          ],
          default: 0          // clicks carry no amount
        }
      },
      occurred_at: 1
  }}
]);

Step-by-step explanation.

$ifNull: ["$amount_cents", "$value_cents"] coalesces the v2 field name over the v1 name, so both generations produce one amount_cents regardless of when the document was written.
$ifNull: ["$occurred_at", { $toDate: "$_id" }] backfills a timestamp for old documents that predate the explicit field — again leaning on the ObjectId's embedded creation time.
$switch branches on type: purchases stay positive, refunds are negated so a later $group can sum net revenue directly, and clicks default to 0.
The final $project collapses three document shapes into one uniform row shape. Downstream stages (and the warehouse) now see a single stable schema, decoupled from the source's polymorphism.
This defensive pattern — coalesce, backfill, branch, project — is the everyday reality of aggregating a schema-flexible collection; skipping it is how "the numbers don't match" bugs enter a pipeline.

Output.

event_id	type	amount	occurred_at
66…aa01	purchase	42.00	2026-08-14T10:22:00Z
66…aa02	refund	-18.50	2026-08-14T10:31:00Z
66…aa03	click	0	2026-08-14T10:33:00Z

Rule of thumb. Treat a schema-flexible collection as polymorphic until proven otherwise. Normalize with $ifNull (renamed fields), $toDate: "$_id" (missing timestamps), and $switch (per-type logic) before any $group, so every downstream stage sees one clean shape.

Data engineering interview question on the document model

A senior interviewer might open with: "You're modelling a blogging platform in MongoDB — posts, the comments on each post, and the authors who write both. Comments can number in the thousands on a popular post; authors write many posts. Walk me through what you'd embed, what you'd reference, why, and how you'd read a post with its recent comments and the author's display name."

Solution Using an embed-plus-reference hybrid keyed by ObjectId

// authors — referenced by both posts and comments (shared entity)
db.authors.insertOne({
  _id: ObjectId("6651000000000000000000a1"),
  display_name: "Priya N.",
  joined_at: ISODate("2025-02-01T00:00:00Z")
});

// posts — embed a *bounded* preview of recent comments; reference the author
db.posts.insertOne({
  _id: ObjectId("6651000000000000000000b1"),
  author_id: ObjectId("6651000000000000000000a1"),   // reference (shared)
  title: "Modelling in MongoDB",
  body_md: "…",
  created_at: ISODate("2026-08-20T09:00:00Z"),
  comment_count: 3120,                                 // denormalized counter
  recent_comments: [                                   // embedded, capped at ~20
    { author_id: ObjectId("…c1"), text: "Great post", at: ISODate("2026-08-20T09:05:00Z") }
  ]
});

// comments — the full, unbounded set lives in its own collection
db.comments.insertOne({
  _id: ObjectId(),
  post_id: ObjectId("6651000000000000000000b1"),      // reference to parent
  author_id: ObjectId("6651000000000000000000c1"),
  text: "Great post",
  created_at: ISODate("2026-08-20T09:05:00Z")
});

// Read a post with its author display name and most-recent comments
db.posts.aggregate([
  { $match: { _id: ObjectId("6651000000000000000000b1") } },
  { $lookup: {
      from: "authors", localField: "author_id",
      foreignField: "_id", as: "author"
  }},
  { $unwind: "$author" },
  { $project: {
      title: 1, created_at: 1, comment_count: 1,
      author_name: "$author.display_name",
      recent_comments: { $slice: ["$recent_comments", 5] }
  }}
]);

Step-by-step trace.

Step	Input	Action	Output
Model authors	shared entity	one `authors` doc per person	referenced by `_id`
Model posts	bounded preview	embed `recent_comments` (~20) + `comment_count`	fast post render
Model comments	unbounded set	own collection keyed by `post_id`	no 16 MB risk
`$match`	post `_id`	index seek to one post	1 document
`$lookup` + `$unwind`	`author_id`	join the single author, flatten	post + author_name
`$project` + `$slice`	full doc	keep title, count, 5 recent comments	render payload

After modelling, rendering a post is one indexed $match, a single-document $lookup on the author, and a $slice of the embedded preview — no scan over the thousands of full comments. The unbounded comment set lives safely in its own collection; the post document stays small and read-cheap.

Output:

title	author_name	comment_count	recent_comments (shown)
Modelling in MongoDB	Priya N.	3120	5 most recent

Why this works — concept by concept:

Embed the bounded, reference the unbounded — recent_comments is capped at ~20 and embedded so the post renders in one read; the full comment set is unbounded and lives in its own collection, keeping every post document far below the 16 MB cap.
Reference shared entities by ObjectId — an author writes many posts and many comments, so the author is stored once and referenced by _id; a display-name change updates exactly one document instead of thousands of copies.
Denormalized counters — comment_count is stored on the post so the common "3,120 comments" render never scans the comments collection; a change stream or write-time increment keeps it fresh.
$lookup only where it's cheap — the author join touches exactly one document because the post already resolved to one; the expensive many-row join over comments is avoided entirely by the embedded preview.
Cost — read of a post is O(1) index seek + O(1) author lookup + O(k) slice of a k≈5 preview, versus O(comments) for a naive "join all comments" design. Writes to add a comment are O(1) insert plus an O(1) counter/preview update.

Database
Topic — database
Document-model and data-modelling problems

Practice →

JSON Topic — json JSON and nested-document query problems

Practice →

2. Aggregation pipeline stages ($match/$group/$project)

The pipeline is an ordered conveyor — `$match` filters, `$group` folds, `$project` reshapes, and order is the program

The mental model in one line: the mongodb aggregation pipeline is an ordered array of stages passed to db.collection.aggregate([...]), where each stage consumes the document stream from the stage before it and emits a transformed stream — $match filters documents (and can use an index when it runs first), $group folds many documents into one per key using accumulators, and $project / $addFields reshape each document — and because the stages run in the order you write them, stage ordering is not cosmetic: it decides both correctness and cost. If you know SQL, $match is WHERE, $group is GROUP BY with aggregate functions, $project is the SELECT list, $sort is ORDER BY, and $limit is LIMIT — but written as a sequence you control explicitly.

The core stages every pipeline uses.

$match. Filters the stream with the same query syntax as find(). Placed first, it can use an index and shrink the stream before any expensive stage runs. Placed after a $group, it filters the grouped output (the equivalent of SQL HAVING).
$group. Folds documents sharing the same _id expression into one output document, computing accumulators — $sum, $avg, $min, $max, $push, $addToSet, $first, $last. The _id of a $group is the grouping key; _id: null groups everything into one total.
$project and $addFields. $project replaces the document with exactly the fields you list (inclusion/exclusion + computed fields); $addFields (alias $set) adds computed fields while keeping everything else. Use $project to shape the final contract, $addFields to enrich mid-pipeline.
$sort, $limit, $skip, $count. Ordering, top-N, pagination, and cardinality. $sort + $limit together are special-cased by the engine into an efficient top-N that avoids sorting the whole stream.

The optimization rules the engine applies.

$match pushdown. The optimizer moves a $match as early as possible — ideally before a $project or $lookup — so filtering happens on the smallest possible stream and an index can serve it.
Index eligibility ends at the first transforming stage. Only a $match (or $sort) at the front of the pipeline can use a collection index. Once a $group or $project has transformed the documents, later $match stages filter in memory.
$sort + $limit coalescing. When a $limit follows a $sort, the engine keeps only the top-N in a bounded heap instead of sorting everything — turning an O(n log n) full sort into O(n log k).
The 100 MB memory limit. A blocking stage ($group, $sort) that exceeds 100 MB errors unless you set allowDiskUse: true, which spills to disk. Index-served sorts avoid the limit entirely.

What interviewers listen for.

Do you put $match first and explain that it enables an index and shrinks the stream? — required answer.
Do you name $group accumulators ($sum, $avg, $push) rather than hand-waving "it groups"? — senior signal.
Do you distinguish $project (replace) from $addFields (augment)? — a real-experience tell.
Do you know $match after $group is HAVING and that only a leading $match uses an index? — senior signal.

Worked example — revenue per status with `$match` + `$group`

Detailed explanation. The everyday pipeline: filter orders to a date window with $match, then fold them into one row per status with $group, summing revenue and counting orders. This is SELECT status, SUM(total), COUNT(*) FROM orders WHERE created_at >= … GROUP BY status written as stages.

Filter. $match on created_at and status != cancelled.
Fold. $group by status, $sum revenue, $sum: 1 for count.
Shape. $project to rename _id to status and round money.

Question. Compute total revenue and order count per status for orders created on or after 2026-08-01.

Input.

_id	status	total_cents	created_at
o1	paid	4200	2026-08-14
o2	paid	1850	2026-08-15
o3	refunded	900	2026-08-16
o4	paid	3000	2026-07-30

Code.

db.orders.aggregate([
  { $match: {
      created_at: { $gte: ISODate("2026-08-01T00:00:00Z") },
      status: { $ne: "cancelled" }
  }},
  { $group: {
      _id: "$status",                       // grouping key
      revenue_cents: { $sum: "$total_cents" },
      order_count:   { $sum: 1 }
  }},
  { $project: {
      _id: 0,
      status: "$_id",
      revenue: { $round: [{ $divide: ["$revenue_cents", 100] }, 2] },
      order_count: 1
  }},
  { $sort: { revenue: -1 } }
]);

Step-by-step explanation.

$match runs first so it can use an index on created_at and drop o4 (July) and any cancelled orders before anything expensive happens. Filtering early is the single biggest performance lever in a pipeline.
$group with _id: "$status" creates one output document per distinct status. $sum: "$total_cents" accumulates revenue; $sum: 1 counts documents — the idiomatic MongoDB COUNT(*).
After $group, the stream is tiny (one doc per status), so the following stages are cheap regardless of the source size.
$project renames the group key _id to a friendly status, divides cents to dollars, and $rounds to two places — coercing the money into a warehouse-ready shape.
$sort: { revenue: -1 } orders the small grouped result; because it follows a $group it sorts in memory, but the input is only a handful of rows so the 100 MB limit is irrelevant.

Output.

status	revenue	order_count
paid	60.50	2
refunded	9.00	1

Rule of thumb. Always lead with $match so an index shrinks the stream, and count with $sum: 1. Put the $project that shapes money and renames _id after the $group, when the stream is already small.

Worked example — `$project` vs `$addFields` and computed fields

Detailed explanation. The two reshaping stages look similar but differ in one way that trips up newcomers: $project outputs only the fields you name (plus _id unless you exclude it), while $addFields keeps the whole document and layers new fields on top. Use $addFields to compute intermediate values you still need downstream; use $project to finalize the output contract.

$addFields. Adds net_cents mid-pipeline while retaining total_cents, discount_cents, etc.
$project. Emits the final (order_id, net) shape and drops everything else.
Computed expressions. $subtract, $multiply, $cond, $dateToString build derived fields.

Question. Add a net_cents field (total minus discount) mid-pipeline, then project a clean final row with a formatted date.

Input.

_id	total_cents	discount_cents	created_at
o1	4200	200	2026-08-14T10:22Z
o2	1850	0	2026-08-15T14:05Z

Code.

db.orders.aggregate([
  { $addFields: {                                  // augment, keep everything
      net_cents: { $subtract: ["$total_cents", { $ifNull: ["$discount_cents", 0] }] },
      day: { $dateToString: { format: "%Y-%m-%d", date: "$created_at" } }
  }},
  { $project: {                                    // finalize the contract
      _id: 0,
      order_id: { $toString: "$_id" },
      day: 1,
      net: { $round: [{ $divide: ["$net_cents", 100] }, 2] },
      is_free: { $cond: [{ $eq: ["$net_cents", 0] }, true, false] }
  }}
]);

Step-by-step explanation.

$addFields computes net_cents with $subtract, guarding a possibly-missing discount_cents via $ifNull. Because $addFields keeps the original fields, later stages could still reference total_cents if needed.
$dateToString derives a day bucket string — the document-store equivalent of DATE_TRUNC('day', ...), useful for daily grouping.
$project then discards everything except the fields the consumer wants, converting _id to a string and rounding money. This is where the flexible document becomes a fixed row.
$cond: [{ $eq: [...] }, true, false] shows an inline conditional — the pipeline's ternary — producing a boolean is_free flag without a separate stage.
The division of labour is the lesson: $addFields for intermediate enrichment you may reuse, $project for the final, minimal, typed output shape.

Output.

order_id	day	net	is_free
o1	2026-08-14	40.00	false
o2	2026-08-15	18.50	false

Rule of thumb. Reach for $addFields when you need a computed value and still want the rest of the document; reach for $project when you want to lock the output down to a specific, minimal shape. Never use $project mid-pipeline to add one field if it silently drops fields a later stage needs.

Worked example — multi-stage top-N per customer

Detailed explanation. Combining stages produces real reports: filter to a window, group per customer, sort by spend, and keep the top N. The $sort + $limit pair is special-cased into an efficient top-N, so this scales even over large collections when $match is index-served.

Window. $match last 30 days.
Fold. $group per customer_id, sum revenue, count orders.
Rank. $sort by revenue descending, $limit 3.

Question. Find the top 3 customers by revenue over the last 30 days, with their order counts.

Input.

_id	customer_id	total_cents	created_at
o1	c1	4200	2026-08-30
o2	c2	9900	2026-08-29
o3	c1	1800	2026-08-28
o4	c3	500	2026-08-27

Code.

db.orders.aggregate([
  { $match: { created_at: { $gte: ISODate("2026-08-06T00:00:00Z") } } },
  { $group: {
      _id: "$customer_id",
      revenue_cents: { $sum: "$total_cents" },
      order_count:   { $sum: 1 }
  }},
  { $sort:  { revenue_cents: -1 } },       // coalesced with the $limit below
  { $limit: 3 },
  { $project: {
      _id: 0,
      customer_id: { $toString: "$_id" },
      revenue: { $divide: ["$revenue_cents", 100] },
      order_count: 1
  }}
]);

Step-by-step explanation.

$match on created_at is index-served and drops everything outside the 30-day window before grouping.
$group per customer_id produces one row per customer with summed revenue and an order count.
The engine coalesces the adjacent $sort + $limit into a bounded top-N: it maintains a heap of the 3 highest-revenue customers rather than sorting the entire grouped set, which matters when there are millions of customers.
$limit: 3 keeps only the leaders; because it is fused with the sort, memory stays O(3), not O(customers).
The trailing $project formats the survivors — money to dollars, _id to a string — leaving a clean, ranked, warehouse-ready result.

Output.

customer_id	revenue	order_count
c2	99.00	1
c1	60.00	2
c3	5.00	1

Rule of thumb. Put $sort immediately before $limit so the engine coalesces them into a memory-bounded top-N. Keep the shaping $project last, after the stream is already reduced to the rows you will actually return.

Data engineering interview question on the aggregation pipeline

A senior interviewer might ask: "Off a 40-million-document orders collection, produce a daily report for the last 30 days: for each (day, customer) give total revenue and order count, then return only the top 10 customer-days by revenue. Write the pipeline, explain the stage order, and tell me which stage should use an index."

Solution Using $match → $group → $sort → $limit with a date-bucket key

db.orders.aggregate([
  // 1. Index-served filter — shrink 40M docs to the 30-day window first
  { $match: {
      created_at: { $gte: ISODate("2026-08-06T00:00:00Z"),
                    $lt:  ISODate("2026-09-05T00:00:00Z") },
      status: { $in: ["paid", "refunded"] }
  }},
  // 2. Bucket each doc into a (day, customer) key
  { $addFields: {
      day: { $dateToString: { format: "%Y-%m-%d", date: "$created_at" } }
  }},
  // 3. Fold to one row per (day, customer)
  { $group: {
      _id: { day: "$day", customer_id: "$customer_id" },
      revenue_cents: { $sum: "$total_cents" },
      order_count:   { $sum: 1 }
  }},
  // 4. Rank and keep the top 10 (coalesced top-N)
  { $sort:  { revenue_cents: -1 } },
  { $limit: 10 },
  // 5. Finalize the output contract
  { $project: {
      _id: 0,
      day: "$_id.day",
      customer_id: { $toString: "$_id.customer_id" },
      revenue: { $round: [{ $divide: ["$revenue_cents", 100] }, 2] },
      order_count: 1
  }}
], { allowDiskUse: true });

Step-by-step trace.

Step	Stage	Effect on the stream	Cost note
1	`$match`	40M → ~1.2M (30-day window)	uses index on `{status, created_at}`
2	`$addFields`	add `day` bucket	streaming, cheap
3	`$group`	~1.2M → N (day×customer)	blocking; `allowDiskUse` if >100 MB
4	`$sort`+`$limit`	N → 10	coalesced top-N heap
5	`$project`	shape 10 rows	trivial

The $match is the load-bearing stage: served by a compound index on {status: 1, created_at: 1}, it turns a 40-million-document scan into a bounded index range, so the $group folds ~1.2M documents rather than 40M. The $sort + $limit never materializes the full grouped set — it keeps a heap of 10.

Output:

day	customer_id	revenue	order_count
2026-08-29	c2	990.00	6
2026-08-14	c1	812.50	9
2026-08-22	c7	640.00	4

Why this works — concept by concept:

$match first (index-served) — leading the pipeline with $match lets the compound index on {status, created_at} restrict the scan to the 30-day window, so every later stage processes ~1.2M documents instead of 40M. This is the single decision that makes the report tractable.
$group by a composite key — _id: { day, customer_id } folds the stream to one row per customer-day; accumulators $sum revenue and count in a single pass over the filtered stream.
$sort + $limit coalescing — placing $limit right after $sort triggers the top-N optimization: the engine keeps a bounded heap of 10 rather than sorting the whole grouped set, holding memory flat.
allowDiskUse safety valve — a $group over 1.2M documents can exceed the 100 MB in-memory limit; allowDiskUse: true lets the blocking stage spill to disk instead of erroring, trading a little speed for reliability.
Cost — O(range) index seek for the filter, one streaming pass for the $group (O(m) over the m filtered docs), and O(N log 10) for the top-N, versus O(40M) for a naive full scan. The index turns the dominant term from collection size into result-window size.

Aggregation
Topic — aggregation
Aggregation pipeline and group-by problems

Practice →

Database Topic — database Database query and reporting problems

Practice →

3. $lookup joins & $unwind

`$lookup` is a left outer join into an array; `$unwind` flattens that array into one document per element

The mental model in one line: $lookup performs a left outer join from the current collection into another, attaching every matched foreign document as an array on a new field, and $unwind then expands that array so the stream carries one document per matched element — together they let a document store answer the relational questions embedding cannot, at the cost of a nested-loop join that you must back with an index on the joined-to collection's foreignField. Where the relational world joins by default and denormalizes reluctantly, MongoDB embeds by default and joins reluctantly — but when a query spans two collections (orders and the customers who placed them), $lookup + $unwind is the tool, and understanding its cost is what separates a fast pipeline from a cluster-melting one.

The two forms of $lookup.

Equality form. { from, localField, foreignField, as } joins where localField == foreignField. Simple, and the engine can use an index on foreignField. This is the everyday form for a foreign-key join.
Sub-pipeline (correlated) form. { from, let: { … }, pipeline: [ … ], as } runs a whole aggregation against the foreign collection, with let binding local values into $expr conditions. Use it to filter, project, or aggregate the joined side — e.g. "only the customer's active address," or "the sum of this order's refunds."
The as field is always an array. Even a one-to-one join lands the match in a single-element array; $unwind (or $arrayElemAt) turns it into a scalar/object.

The mechanics of $unwind.

Array to rows. $unwind: "$items" emits one output document per element of items, copying the parent fields onto each — exactly SQL's CROSS JOIN LATERAL / "explode."
Empty and missing arrays. By default $unwind drops documents whose array is empty or missing. { path: "$items", preserveNullAndEmptyArrays: true } keeps them (the array field becomes null) — the difference between an inner and a left join semantics after a $lookup.
includeArrayIndex. Optionally emits the element's position, useful when order matters (line 1, line 2, …).

Performance caveats you must state.

$lookup is a nested loop. For each input document it probes the foreign collection. Without an index on foreignField, each probe is a collection scan — an O(n·m) disaster. Index the foreignField.
$unwind multiplies the stream. Unwinding a 50-element array turns 1 document into 50; do it after you have filtered down, and group back afterward if you only need aggregates.
Filter before you join. A $match before $lookup shrinks the number of probes; a $match inside a sub-pipeline shrinks each probe's result.

What interviewers listen for.

Do you call $lookup a left outer join and note the result is an array? — required answer.
Do you say "index the foreignField" because $lookup is a nested loop? — senior signal.
Do you know $unwind drops empty arrays unless preserveNullAndEmptyArrays is set? — a real-experience tell.
Do you reach for the sub-pipeline form to filter/aggregate the joined side instead of joining everything then filtering? — senior signal.

Worked example — basic `$lookup` from orders to customers

Detailed explanation. The foreign-key join: each order carries a customer_id; join to customers._id to attach the customer, then flatten the single-element array to an object. This is orders LEFT JOIN customers ON orders.customer_id = customers._id.

Join. Equality form on customer_id → _id.
Flatten. $unwind the single-element customer array.
Shape. $project the customer's name onto the order.

Question. Attach each order's customer name and country using a $lookup.

Input.

orders:

_id	customer_id	total_cents
o1	c1	4200
o2	c9	1850

customers:

_id	name	country
c1	Ana	US
c9	Bo	CA

Code.

db.orders.aggregate([
  { $lookup: {
      from: "customers",
      localField: "customer_id",
      foreignField: "_id",          // index this field on customers
      as: "customer"
  }},
  { $unwind: "$customer" },          // single-element array -> object
  { $project: {
      _id: 0,
      order_id: { $toString: "$_id" },
      total: { $divide: ["$total_cents", 100] },
      customer_name: "$customer.name",
      customer_country: "$customer.country"
  }}
]);

Step-by-step explanation.

$lookup matches each order's customer_id against customers._id, attaching the matched customer as a one-element array on customer. With an index on customers._id (the default _id index), each probe is an O(log n) seek, not a scan.
$unwind: "$customer" flattens that single-element array into an object. Because every order here has a matching customer, no documents are dropped; if an order referenced a deleted customer, the default $unwind would drop it (use preserveNullAndEmptyArrays to keep it as a left join).
$project lifts customer.name and customer.country up to top-level fields and formats money — the flattened, joined row is now a clean record.
The order of stages matters: joining before projecting keeps the pipeline readable, and because orders is the driving side, the number of nested-loop probes equals the number of orders.
This is the bread-and-butter enrichment pattern: a fact collection (orders) enriched with a dimension collection (customers) via an indexed $lookup.

Output.

order_id	total	customer_name	customer_country
o1	42.00	Ana	US
o2	18.50	Bo	CA

Rule of thumb. For a foreign-key join, use the equality $lookup and make sure the foreignField is indexed (joining to _id gets this for free). Follow with $unwind to flatten one-to-one matches into objects; add preserveNullAndEmptyArrays: true when you need left-join semantics.

Worked example — `$unwind` an embedded array then group back

Detailed explanation. When line items are embedded (section 1), you often need per-SKU numbers across all orders. $unwind explodes the embedded line_items array into one document per line, and a following $group folds those lines by SKU. No $lookup needed — the join is already "done" by embedding.

Explode. $unwind: "$line_items".
Fold. $group by line_items.sku, sum quantity and revenue.
Rank. $sort by revenue.

Question. Compute total quantity and revenue per SKU across all embedded line items.

Input.

_id	line_items
o1	[{sku:A, qty:2, unit_cents:1500}, {sku:B, qty:1, unit_cents:1200}]
o2	[{sku:A, qty:1, unit_cents:1500}]

Code.

db.orders.aggregate([
  { $unwind: "$line_items" },                 // one doc per line item
  { $group: {
      _id: "$line_items.sku",
      qty:          { $sum: "$line_items.qty" },
      revenue_cents:{ $sum: { $multiply: ["$line_items.qty", "$line_items.unit_cents"] } }
  }},
  { $project: {
      _id: 0,
      sku: "$_id",
      qty: 1,
      revenue: { $divide: ["$revenue_cents", 100] }
  }},
  { $sort: { revenue: -1 } }
]);

Step-by-step explanation.

$unwind: "$line_items" turns each order into as many documents as it has line items, copying the order's other fields onto each. Order o1 (2 lines) becomes 2 documents; o2 becomes 1.
$group by line_items.sku folds those exploded lines by product. $sum: "$line_items.qty" totals units; the $multiply inside $sum computes per-line revenue (qty × unit price) before summing.
Because the join was pre-materialized by embedding, there is no $lookup and no nested loop — $unwind + $group over an indexed/scanned collection is the whole cost.
$project reshapes to (sku, qty, revenue) and converts cents; $sort ranks SKUs by revenue.
The pattern generalizes: explode the array, group by the element key, aggregate. When you only need aggregates, always group back down after unwinding so the multiplied stream does not flow downstream.

Output.

sku	qty	revenue
A	3	45.00
B	1	12.00

Rule of thumb. $unwind the array, $group by the element key, then aggregate — and group back down immediately, because an unwound stream is N× larger than the source. If you later need the parent shape, $group with $push rebuilds arrays.

Worked example — correlated sub-pipeline `$lookup`

Detailed explanation. The sub-pipeline form joins with a filter or aggregation applied to the foreign side, using let to bind local fields and $expr to compare them. Here, for each customer, join only their paid orders and compute the customer's lifetime paid revenue — filtering the joined side instead of joining everything and filtering after.

Bind. let: { cid: "$_id" } exposes the customer id to the sub-pipeline.
Filter foreign. Sub-pipeline $match on status: "paid" and $expr customer_id == cid.
Aggregate foreign. Sub-pipeline $group sums paid revenue.

Question. For each customer, attach their lifetime paid revenue using a correlated sub-pipeline $lookup.

Input.

customers: c1, c9. orders: o1(c1, paid, 4200), o2(c1, cancelled, 999), o3(c9, paid, 1850).

Code.

db.customers.aggregate([
  { $lookup: {
      from: "orders",
      let: { cid: "$_id" },
      pipeline: [
        { $match: { $expr: { $and: [
            { $eq: ["$customer_id", "$$cid"] },   // correlate to the local customer
            { $eq: ["$status", "paid"] }          // filter the joined side
        ]}}},
        { $group: { _id: null, paid_cents: { $sum: "$total_cents" } } }
      ],
      as: "paid"
  }},
  { $addFields: {
      lifetime_paid: {
        $round: [{ $divide: [{ $ifNull: [{ $arrayElemAt: ["$paid.paid_cents", 0] }, 0] }, 100] }, 2]
      }
  }},
  { $project: { _id: 0, customer_id: { $toString: "$_id" }, lifetime_paid: 1 } }
]);

Step-by-step explanation.

let: { cid: "$_id" } binds each customer's _id into a variable $$cid that the sub-pipeline can reference — this is what makes the join correlated (per-customer), not a cartesian product.
The sub-pipeline's $match with $expr filters orders to that customer's paid rows only. Filtering inside the join means only relevant orders are ever materialized, far cheaper than joining all orders then filtering.
The sub-pipeline $group with _id: null collapses those paid orders into a single total per customer, so the as: "paid" array holds at most one document.
$arrayElemAt: ["$paid.paid_cents", 0] pulls that single total out of the array, $ifNull defaults customers with no paid orders to 0, and $round/$divide format the money.
The result: one clean row per customer with lifetime paid revenue, computed by pushing the filter and aggregation into the join — the correlated sub-pipeline is the tool for "join, but only the matching, filtered, aggregated foreign rows."

Output.

customer_id	lifetime_paid
c1	42.00
c9	18.50

Rule of thumb. Use the sub-pipeline $lookup (let + $expr) whenever you need to filter, project, or aggregate the joined collection — it shrinks each probe's result at the source. For a plain foreign-key attach with no foreign-side filtering, the simpler equality form is faster and index-friendlier.

Data engineering interview question on $lookup and $unwind

A senior interviewer might ask: "You have orders with embedded line_items and a separate customers collection. Produce one row per line item that includes the order id, the SKU, the line revenue, and the customer's name and country. Explain where the join happens, where the explosion happens, and how you keep it from being O(n·m)."

Solution Using $lookup on customers + $unwind on line_items with an index

db.orders.aggregate([
  // 1. Narrow first so we probe customers fewer times
  { $match: { status: "paid",
              created_at: { $gte: ISODate("2026-08-01T00:00:00Z") } } },

  // 2. Join the customer dimension (indexed foreignField = _id)
  { $lookup: {
      from: "customers",
      localField: "customer_id",
      foreignField: "_id",
      as: "customer"
  }},
  { $unwind: { path: "$customer", preserveNullAndEmptyArrays: true } }, // left join

  // 3. Explode the embedded line items into one doc per line
  { $unwind: "$line_items" },

  // 4. Shape one row per line item
  { $project: {
      _id: 0,
      order_id: { $toString: "$_id" },
      sku: "$line_items.sku",
      line_revenue: { $round: [
        { $divide: [{ $multiply: ["$line_items.qty", "$line_items.unit_cents"] }, 100] }, 2 ] },
      customer_name:    "$customer.name",
      customer_country: "$customer.country"
  }}
]);

Step-by-step trace.

Step	Stage	Stream effect	Cost control
1	`$match`	drop unpaid / old orders	index on `{status, created_at}`
2	`$lookup`	attach `customer` array	index on `customers._id` → O(log n) probe
2b	`$unwind` customer	flatten to object (left join)	`preserveNullAndEmptyArrays`
3	`$unwind` line_items	1 order → k lines	done after narrowing
4	`$project`	one row per line item	streaming

The join to customers is a nested loop, but each probe is an indexed _id seek, so it is O(orders · log customers), not O(orders · customers). The line-item explosion happens last, after $match has already reduced the driving set, so the N× multiplication applies to a small stream. Filtering first, joining on an index, exploding last is the recipe.

Output:

order_id	sku	line_revenue	customer_name	customer_country
o1	TSHIRT-BLK-M	30.00	Ana	US
o1	STICKER-PACK	12.00	Ana	US
o2	TSHIRT-BLK-M	15.00	Bo	CA

Why this works — concept by concept:

$lookup is a left outer join — it attaches matched customers as an array; with preserveNullAndEmptyArrays on the $unwind, orders whose customer was deleted still survive with null customer fields, exactly matching SQL left-join semantics.
Index the foreignField — joining on customers._id uses the built-in _id index, so each of the nested-loop probes is an O(log n) seek instead of a full collection scan; this is the difference between a fast join and an O(n·m) meltdown.
$unwind after narrowing — exploding line_items multiplies the stream by k; doing it after $match (and after the single-row customer join) means the multiplication hits a small, already-filtered set.
Filter → join → explode → shape — the stage order is the optimization: shrink the driving set, join on an index, explode late, project last. Reordering these degrades cost, not correctness.
Cost — O(range) for the indexed filter, O(m · log c) for the customer join over m filtered orders and c customers, and O(m · k) for the line explosion — versus O(orders · customers) for an unindexed join or an early explosion. Indexing and ordering keep every term bounded.

Aggregation
Topic — aggregation
$lookup, $unwind, and join problems

Practice →

JSON Topic — json JSON array and nested-join problems

Practice →

4. Indexing & performance

Indexes turn a COLLSCAN into an IXSCAN — order compound keys Equality → Sort → Range

The mental model in one line: MongoDB indexes are B-trees over one or more fields, and the difference between a query that reads three documents and one that reads fifty million is whether the planner can serve it from an index (an IXSCAN) or must examine every document (a COLLSCAN) — for a compound index the field order follows the ESR rule (Equality fields first, then the Sort field, then Range fields), and explain() is the instrument that tells you which path the planner chose and how many documents it had to examine to return your results. Every performance conversation about an aggregation reduces to two numbers from explain(): totalDocsExamined and nReturned. When they are far apart, you are scanning; when they are close, your index is doing its job.

The index types a data engineer reaches for.

Single-field. { created_at: 1 } — the default building block. _id is always indexed.
Compound. { status: 1, created_at: -1 } — one B-tree over several fields, ordered left to right. Serves queries that filter on a prefix of the fields.
Multikey. An index on an array field ({ "line_items.sku": 1 }) automatically indexes every element — this is what makes embedded arrays queryable.
Partial and TTL. A partial index covers only documents matching a filter ({ status: "paid" }), saving space; a TTL index ({ created_at: 1 }, expireAfterSeconds) auto-deletes old documents — handy for event collections.

The ESR rule for compound-index order.

Equality first. Fields matched with $eq / $in go leftmost, so the B-tree seeks directly to the matching range.
Sort next. The field(s) the query sorts by come after equality, so the index returns rows already in order — no in-memory sort, no 100 MB limit.
Range last. Inequality fields ($gte, $lt) go last, because a range scan on an earlier field would break the ordering the sort relies on.
Why order matters. A compound index only serves a query that uses a left prefix of its fields; { status, created_at } serves a filter on status alone or status + created_at, but not created_at alone.

Reading explain().

winningPlan.stage. IXSCAN (good — index used) vs COLLSCAN (every document examined). For aggregations, run db.coll.explain("executionStats").aggregate([...]).
totalDocsExamined vs nReturned. The ratio is your efficiency: examining 50M to return 10 is a scan; examining 12 to return 10 is a healthy index.
totalKeysExamined. Index entries read; close to nReturned means the index is selective.
Covered queries. When the index contains every field the query needs (filter + projection) and you exclude _id, the planner never fetches the document — totalDocsExamined is 0.

What interviewers listen for.

Do you name the ESR rule for ordering compound-index fields? — senior signal.
Do you read totalDocsExamined vs nReturned from explain() rather than guessing? — required answer.
Do you know a compound index serves only a left prefix of its fields? — a real-experience tell.
Do you mention covered queries (index answers the whole query, no document fetch)? — senior signal.

Worked example — COLLSCAN to IXSCAN with `explain()`

Detailed explanation. The diagnostic workflow: run explain("executionStats") on a slow aggregation, see a COLLSCAN with totalDocsExamined equal to the whole collection, add the right index, and confirm the plan flips to IXSCAN with totalDocsExamined near nReturned.

Before. No index on status; $match scans everything.
After. Index on { status: 1 }; $match seeks the matching range.
Instrument. explain("executionStats") reports the two counts.

Question. Diagnose and fix a $match: { status: "paid" } aggregation that scans all 50M documents.

Input.

Metric	Before index	After index
winningPlan	COLLSCAN	IXSCAN
totalDocsExamined	50,000,000	1,240,000
nReturned	1,240,000	1,240,000
elapsed	~40 s	~0.9 s

Code.

// 1. Diagnose — what does the planner do today?
db.orders.explain("executionStats").aggregate([
  { $match: { status: "paid" } },
  { $group: { _id: "$customer_id", n: { $sum: 1 } } }
]);
// winningPlan.stage: "COLLSCAN"  ->  totalDocsExamined: 50000000

// 2. Fix — index the filtered field
db.orders.createIndex({ status: 1 });

// 3. Confirm — plan flips to IXSCAN
db.orders.explain("executionStats").aggregate([
  { $match: { status: "paid" } },
  { $group: { _id: "$customer_id", n: { $sum: 1 } } }
]);
// winningPlan.inputStage.stage: "IXSCAN"  ->  totalDocsExamined ≈ nReturned

Step-by-step explanation.

explain("executionStats") runs the pipeline and reports actual execution numbers. A COLLSCAN with totalDocsExamined: 50000000 is the smoking gun: the planner had no index for status, so it read every document.
createIndex({ status: 1 }) builds a B-tree on status. Now the planner can seek directly to the "paid" entries instead of scanning.
Re-running explain shows the $match served by an IXSCAN feeding the $group; totalDocsExamined drops from 50M to the ~1.24M paid orders — it now examines only the documents it returns.
The $group still processes 1.24M documents, but the collection is no longer scanned to find them — the index provides them directly, cutting wall time from ~40 s to ~0.9 s.
The workflow generalizes: never guess. Run explain, look at COLLSCAN/IXSCAN and the examined-vs-returned ratio, add the index that makes the leading $match seekable, and confirm.

Output.

Query	Plan	Docs examined	Time
status = paid (no index)	COLLSCAN	50,000,000	~40 s
status = paid (indexed)	IXSCAN	1,240,000	~0.9 s

Rule of thumb. When an aggregation is slow, run explain("executionStats") first and read totalDocsExamined vs nReturned. A COLLSCAN or a huge examined-to-returned ratio means the leading $match has no usable index — add one and confirm the plan flips to IXSCAN.

Worked example — compound index following the ESR rule

Detailed explanation. A query that filters on status (equality), sorts by created_at (sort), and ranges on total_cents (range) needs a compound index ordered E-S-R: { status: 1, created_at: -1, total_cents: 1 }. Ordered this way, the index serves the filter, returns rows already sorted (no in-memory sort), and range-scans last.

Equality. status = "paid" → leftmost.
Sort. created_at desc → middle, matching the index direction.
Range. total_cents >= 1000 → rightmost.

Question. Design one compound index that serves a filter + sort + range query without an in-memory sort.

Input.

Clause	Field	Role
`$match` eq	status	Equality
`$sort`	created_at desc	Sort
`$match` range	total_cents ≥ 1000	Range

Code.

// ESR: Equality (status) -> Sort (created_at) -> Range (total_cents)
db.orders.createIndex({ status: 1, created_at: -1, total_cents: 1 });

db.orders.explain("executionStats").aggregate([
  { $match: { status: "paid", total_cents: { $gte: 1000 } } },
  { $sort:  { created_at: -1 } },
  { $limit: 20 }
]);
// IXSCAN serves status=, walks created_at in -1 order (no SORT stage),
// applies total_cents range on the trailing key.

Step-by-step explanation.

status is an equality predicate, so it goes first: the index seeks straight to the "paid" sub-tree, discarding all other statuses without examining them.
created_at: -1 is the sort field and comes second, matching the query's $sort direction. Because the index already stores entries in that order within the "paid" sub-tree, the planner streams them out sorted — there is no separate SORT stage and thus no 100 MB memory risk.
total_cents is a range predicate and goes last. A range on an earlier key would scatter the sort order; keeping it last means the range is applied along the already-ordered scan.
With $limit: 20, the index-ordered scan can stop after 20 qualifying entries — a true top-N served entirely by the index.
Violating ESR (e.g. putting total_cents before created_at) forces the planner to either scan more of the index or add an in-memory SORT; explain would then show a SORT stage, the tell that the index order is wrong.

Output.

Aspect	With ESR index	Wrong order
Filter (status)	index seek	index seek
Sort (created_at)	index-provided	in-memory SORT stage
Range (total_cents)	trailing scan	scan then filter
Memory	O(limit)	up to 100 MB

Rule of thumb. Order a compound index Equality → Sort → Range. If explain still shows a SORT stage, your sort field is in the wrong position — move all equality fields ahead of it and the range fields behind it.

Worked example — covered query and multikey index

Detailed explanation. Two more levers. A covered query is answered entirely from the index — when the index holds every field the query filters and returns (and you exclude _id), MongoDB never touches the documents (totalDocsExamined: 0). A multikey index on an array field indexes each element, making $match/$unwind on embedded arrays fast.

Covered. Index { status: 1, customer_id: 1 }; project only those fields, exclude _id.
Multikey. Index { "line_items.sku": 1 } to seek line items by SKU.

Question. Build a covered query for (status, customer_id) and a multikey index for SKU lookups.

Input.

Goal	Index	Query shape
Covered	`{status:1, customer_id:1}`	match status, return customer_id, no `_id`
Multikey	`{"line_items.sku":1}`	match a SKU inside the array

Code.

// Covered query — index holds every field the query needs
db.orders.createIndex({ status: 1, customer_id: 1 });
db.orders.explain("executionStats").aggregate([
  { $match: { status: "paid" } },
  { $project: { _id: 0, customer_id: 1 } }   // only indexed fields, _id excluded
]);
// -> totalDocsExamined: 0   (answered from the index alone)

// Multikey index — each array element is indexed
db.orders.createIndex({ "line_items.sku": 1 });
db.orders.find({ "line_items.sku": "TSHIRT-BLK-M" });  // IXSCAN, not COLLSCAN

Step-by-step explanation.

The covered-query index { status: 1, customer_id: 1 } contains both fields the pipeline touches. Because the $project returns only customer_id and excludes _id, the planner can satisfy the whole query from index keys — totalDocsExamined is 0, the fastest possible read.
Excluding _id is mandatory for coverage: if _id is returned, MongoDB must fetch the document to get it (unless _id is in the index), breaking coverage.
The multikey index on line_items.sku indexes every element of every array. A find for a SKU seeks the B-tree instead of scanning and unwinding the whole collection — this is why embedded arrays remain queryable at scale.
Multikey has one restriction worth stating: an index cannot be multikey on two array fields at once (that would be a combinatorial explosion), so you index one array path per compound index.
Together, covered queries (zero document fetches) and multikey indexes (arrays stay seekable) are the two advanced levers that keep document-store analytics fast.

Output.

Query	Plan	totalDocsExamined
covered (status→customer_id)	IXSCAN, PROJECTION_COVERED	0
multikey (line_items.sku)	IXSCAN	≈ nReturned

Rule of thumb. For a hot read that returns only a few fields, build a covering compound index and exclude _id so the query is answered from the index alone. For queries into embedded arrays, add a multikey index on the array path — but only one array path per index.

Data engineering interview question on indexing

A senior interviewer might ask: "An aggregation over a 50-million-document orders collection filters status = 'paid', restricts created_at to the last 7 days, sorts newest-first, and returns the top 100. It takes 30 seconds. Walk me through how you'd diagnose it with explain() and the exact index you'd build, ordered correctly."

Solution Using an ESR compound index verified with explain()

// 1. Diagnose the current plan
db.orders.explain("executionStats").aggregate([
  { $match: { status: "paid",
              created_at: { $gte: ISODate("2026-08-29T00:00:00Z") } } },
  { $sort:  { created_at: -1 } },
  { $limit: 100 }
]);
// Symptom: COLLSCAN + in-memory SORT, totalDocsExamined ≈ 50,000,000

// 2. Build the ESR index:
//    Equality = status, Sort = created_at (desc to match), Range = (none extra here)
db.orders.createIndex({ status: 1, created_at: -1 });

// 3. Re-check — plan should be IXSCAN with no SORT stage
db.orders.explain("executionStats").aggregate([
  { $match: { status: "paid",
              created_at: { $gte: ISODate("2026-08-29T00:00:00Z") } } },
  { $sort:  { created_at: -1 } },
  { $limit: 100 }
]);
// winningPlan: IXSCAN {status:1, created_at:-1}; no SORT; stops after 100 keys

Step-by-step trace.

Step	Observation	Action
Diagnose	COLLSCAN, examined ≈ 50M, SORT stage present	read `explain`
Root cause	no index serves `status` + `created_at` order	ESR analysis
Build	`{status:1, created_at:-1}` (E then S)	createIndex
`status`	equality → index prefix	seek to "paid"
`created_at`	sort desc, matches index dir	no in-memory SORT
`$limit 100`	index-ordered scan stops early	O(100) work

The index puts the equality field (status) first so the scan seeks straight to paid orders, then stores created_at descending so the requested newest-first order is provided by the index — the SORT stage disappears. The 7-day created_at filter is applied along that ordered scan, and $limit: 100 lets the scan stop after 100 qualifying keys. totalDocsExamined collapses from ~50M to ~100.

Output:

Metric	Before	After
winningPlan	COLLSCAN + SORT	IXSCAN, no SORT
totalDocsExamined	~50,000,000	~100
totalKeysExamined	n/a	~a few hundred
elapsed	~30 s	~5 ms

Why this works — concept by concept:

COLLSCAN vs IXSCAN — the whole fix is moving the leading $match from a full collection scan to an index seek; explain's winningPlan.stage names which one you got, and totalDocsExamined quantifies the waste.
ESR ordering — status (Equality) first lets the B-tree jump to the paid sub-tree; created_at: -1 (Sort) second means the index already yields rows newest-first, eliminating the blocking in-memory sort and its 100 MB ceiling.
Index-provided sort + $limit — because the scan emerges pre-sorted, $limit: 100 stops the scan after 100 matching keys instead of sorting millions then slicing — a true index-served top-N.
Read explain, don't guess — the before/after explain("executionStats") is the proof: examined-vs-returned went from 50M:100 to ~100:100, and the SORT stage vanished, confirming the index order is right.
Cost — O(log n) seek + O(limit) ordered scan, versus O(n) scan + O(n log n) sort. The index converts the dominant cost from collection size n to the requested limit — the defining win of correct indexing.

Database
Topic — database
Indexing, explain-plan, and performance problems

Practice →

Aggregation Topic — aggregation Aggregation performance and index-tuning problems

Practice →

5. $merge/$out to a warehouse + sharding

`$merge` upserts pipeline output incrementally; `$out` replaces a collection; sharding partitions data by a shard key

The mental model in one line: $out and $merge are the pipeline stages that write results back into a collection instead of returning them to the client — $out fully replaces a target collection with the pipeline's output, while $merge upserts the output into an existing target on a match key (whenMatched / whenNotMatched), making it the primitive for incremental materialized rollups — and when one server can no longer hold the data, sharding partitions each collection across shards by a shard key, which every aggregation and every write must be designed around. For a data engineer, $merge is on-cluster ELT (build the rollup in MongoDB, then export the small rollup rather than the huge raw collection), and the shard key is the single decision that determines whether queries hit one shard or fan out to all of them.

$out vs $merge — full replace vs incremental upsert.

$out. Writes the pipeline result to a collection, replacing it entirely (atomically swaps in the new collection). Perfect for a full nightly snapshot; wrong for incremental updates because it discards whatever was there.
$merge. Writes into a target with per-document semantics: whenMatched (replace, merge, keepExisting, fail, or a custom pipeline) and whenNotMatched (insert, discard, fail). Matched on the target's unique key (often _id or a unique index). This is how you maintain an incremental materialized view.
Materialized views. Run a pipeline that computes today's rollup and $merge it into a daily_rollup collection keyed by (day, customer); re-running only updates the changed days. This is on-cluster incremental ELT.

Getting data to the warehouse.

Rollup then export. $merge the aggregate into a small collection, then let a connector (the MongoDB Kafka/Spark connector, Airbyte, or a mongoexport) ship that to Snowflake/BigQuery — you move megabytes of rollup, not terabytes of raw documents.
Change streams for CDC. For continuous sync, a change stream (db.coll.watch()) tails the oplog and emits insert/update/delete events with resume tokens — the document-store equivalent of log-based CDC.
Incremental watermark. Combine a $match on updated_at > last_run with a $merge upsert so each export processes only changed documents.

Sharding — scaling writes and reads horizontally.

The shard key. The field(s) MongoDB uses to partition documents into chunks across shards. Chosen once, painful to change. Must have high cardinality, low update frequency, and even access distribution.
Hashed vs ranged. A hashed shard key spreads writes evenly (great for monotonically increasing keys like timestamps or ObjectIds that would otherwise hot-spot one shard); a ranged shard key keeps nearby values together (great for range queries, risky for monotonic inserts).
Targeted vs scatter-gather. A query that includes the shard key is targeted to the one shard holding it; a query without it is scatter-gather, broadcast to every shard and merged — much more expensive. Aggregations should carry the shard key in an early $match whenever possible.

What interviewers listen for.

Do you use $merge for incremental rollups and $out for full snapshots, not interchangeably? — senior signal.
Do you warn that a monotonic shard key hot-spots one shard and prefer hashed for it? — required answer.
Do you distinguish targeted (shard key present) from scatter-gather queries? — a real-experience tell.
Do you mention rollup-then-export (move the small aggregate, not the raw collection) and change streams for CDC? — senior signal.

Worked example — incremental daily rollup with `$merge`

Detailed explanation. The materialized-view pattern: a pipeline computes per-day, per-customer revenue and $merges it into a daily_rollup collection keyed by (day, customer_id). Re-running for a day replaces just that day's rows, so the rollup is idempotent and incremental.

Compute. $match a day window, $group by (day, customer).
Key. on: ["day", "customer_id"] (a unique index on the target).
Semantics. whenMatched: "replace", whenNotMatched: "insert".

Question. Maintain an incremental daily_rollup of revenue per customer per day using $merge.

Input.

source day	customer	revenue_cents
2026-09-04	c1	4200
2026-09-04	c2	1850

Code.

// Target needs a unique index on the merge key
db.daily_rollup.createIndex({ day: 1, customer_id: 1 }, { unique: true });

db.orders.aggregate([
  { $match: { created_at: { $gte: ISODate("2026-09-04T00:00:00Z"),
                            $lt:  ISODate("2026-09-05T00:00:00Z") } } },
  { $group: {
      _id: { day: "2026-09-04", customer_id: "$customer_id" },
      revenue_cents: { $sum: "$total_cents" },
      order_count:   { $sum: 1 }
  }},
  { $project: {
      _id: 0,
      day: "$_id.day",
      customer_id: "$_id.customer_id",
      revenue_cents: 1,
      order_count: 1
  }},
  { $merge: {
      into: "daily_rollup",
      on: ["day", "customer_id"],       // unique key to match on
      whenMatched: "replace",           // idempotent re-run of a day
      whenNotMatched: "insert"
  }}
]);

Step-by-step explanation.

The target daily_rollup has a unique index on (day, customer_id) — $merge's on clause requires a unique index over the match fields so it can decide matched vs not-matched deterministically.
The pipeline computes one row per (day, customer_id) for the target day only, thanks to the $match window — so a re-run touches just that day.
$merge with whenMatched: "replace" overwrites any existing row for a (day, customer_id) with the freshly computed one; whenNotMatched: "insert" adds new customer-days. The operation is idempotent: running it twice yields the same rollup.
Because only the target day is recomputed and merged, this is incremental — yesterday's and last week's rollup rows are untouched, unlike $out which would wipe the whole collection.
The pattern is on-cluster ELT: heavy aggregation runs where the data lives, and only the compact rollup is produced — ready to export to a warehouse cheaply.

Output (daily_rollup after the run):

day	customer_id	revenue_cents	order_count
2026-09-04	c1	4200	1
2026-09-04	c2	1850	1

Rule of thumb. Use $merge with a unique index on the match key and whenMatched: "replace" for idempotent, incremental rollups — re-running a day corrects it without touching other days. Reserve $out for when you genuinely want to rebuild the whole target from scratch.

Worked example — `$out` full snapshot for export

Detailed explanation. When a consumer wants a clean, complete snapshot (a nightly full extract with no incremental complexity), $out is the right tool: it atomically replaces the target with the pipeline output. The trade-off is that it rebuilds everything each run.

Compute. Flatten and coerce the full collection.
Write. $out to a orders_export collection.
Export. A connector ships orders_export to the warehouse.

Question. Produce a full, warehouse-typed snapshot of orders into orders_export with $out.

Input.

Field	From	To
order_id	`_id` ObjectId	string
revenue	total_cents	decimal
created_at	`_id` timestamp	date

Code.

db.orders.aggregate([
  { $project: {
      _id: 0,
      order_id:   { $toString: "$_id" },
      customer_id:{ $toString: "$customer_id" },
      status: 1,
      revenue: { $toDecimal: { $divide: ["$total_cents", 100] } },
      created_at: { $toDate: "$_id" }
  }},
  { $out: "orders_export" }        // replaces orders_export entirely
]);

Step-by-step explanation.

The $project coerces BSON types into warehouse-friendly ones (ObjectId → string, cents → decimal, ObjectId → date) — the same source-side contract discipline from section 1.
$out: "orders_export" writes the entire result into orders_export, atomically swapping the new collection in for the old one. Readers see either the complete old snapshot or the complete new one, never a half-written state.
Unlike $merge, $out does not upsert or key-match — it replaces. There is no incremental behavior; every run reprocesses the whole collection.
This is the right choice when the downstream export is a full-refresh (truncate-and-load into the warehouse) and the collection is small enough that a full rebuild each night is acceptable.
For large collections where a full nightly rebuild is too expensive, prefer the incremental $merge pattern with an updated_at watermark instead.

Output (orders_export, sample):

order_id	customer_id	status	revenue	created_at
6650…01	6651…a1	paid	42.00	2026-08-14T10:22:00Z

Rule of thumb. Use $out for a full, atomic snapshot when a truncate-and-load export is acceptable and the collection is modest; switch to $merge with a watermark once a nightly full rebuild costs more than processing only the changed documents.

Worked example — shard key choice and query routing

Detailed explanation. On a sharded cluster, the shard key decides both write distribution and query routing. A monotonically increasing key (like created_at or a raw ObjectId) sends every new write to the same "highest" chunk — a hot shard. Hashing the key spreads writes; including the shard key in queries keeps them targeted.

Bad. Ranged shard key on created_at → all inserts hit one shard.
Good. Hashed shard key on customer_id → even writes, targeted per-customer queries.
Routing. Queries with customer_id are targeted; queries without it scatter-gather.

Question. Choose a shard key for orders that spreads writes and keeps per-customer analytics targeted, and show a targeted vs scatter-gather query.

Input.

Candidate key	Write spread	Per-customer query	Range query
ranged `created_at`	hot shard (bad)	scatter	good
hashed `customer_id`	even (good)	targeted	scatter
hashed `_id`	even	scatter	scatter

Code.

// Enable sharding and choose a hashed customer_id shard key
sh.enableSharding("shop");
sh.shardCollection("shop.orders", { customer_id: "hashed" });

// TARGETED — includes the shard key, routed to one shard
db.orders.aggregate([
  { $match: { customer_id: ObjectId("6651…a1") } },   // shard key present
  { $group: { _id: "$status", revenue: { $sum: "$total_cents" } } }
]);

// SCATTER-GATHER — no shard key, broadcast to every shard then merged
db.orders.aggregate([
  { $match: { status: "paid" } },                      // shard key absent
  { $group: { _id: "$customer_id", revenue: { $sum: "$total_cents" } } }
]);

Step-by-step explanation.

sh.shardCollection("shop.orders", { customer_id: "hashed" }) partitions orders by a hash of customer_id. Hashing spreads inserts evenly across shards even though customer ids are not sequential — no single hot shard.
Choosing customer_id (not created_at) as the key means the common analytical access — "everything for this customer" — carries the shard key, so mongos routes the query to the single shard holding that customer's chunk. That is a targeted query.
The second aggregation filters on status only, which is not the shard key. mongos must broadcast it to every shard, run it everywhere, and merge — a scatter-gather query that costs proportionally to the number of shards.
The shard-key lesson: pick the key that (a) spreads writes (hash a monotonic or low-cardinality candidate) and (b) appears in your hottest queries so they stay targeted. These two goals sometimes conflict, and resolving that trade-off is the senior judgment call.
On a sharded cluster, always try to put the shard key in an early $match in an aggregation; without it, every stage runs on every shard.

Output.

Query	Shard key present?	Routing	Shards touched
per-customer	yes (`customer_id`)	targeted	1
by status	no	scatter-gather	all

Rule of thumb. Pick a shard key with high cardinality that appears in your hottest queries, and hash it when the natural key is monotonic (timestamps, ObjectIds) to avoid a hot shard. Design aggregations to carry the shard key in an early $match so they stay targeted instead of scatter-gather.

Data engineering interview question on $merge and sharding

A senior interviewer might ask: "You run a sharded orders cluster. You need a nightly incremental revenue rollup per customer per day, exported to Snowflake, with re-runs that safely correct a day without rewriting history. Walk me through the shard key, the incremental pipeline, the $merge semantics, and how you keep the aggregation from scatter-gathering across all shards."

Solution Using $merge whenMatched upsert with a watermark on a hashed shard key

// 0. Cluster: hashed shard key spreads writes; customer_id keeps per-customer targeted
sh.shardCollection("shop.orders", { customer_id: "hashed" });

// Target rollup with a unique merge key
db.daily_rollup.createIndex({ day: 1, customer_id: 1 }, { unique: true });

// 1. Incremental pipeline — only changed docs since the last watermark
const lastRun = ISODate("2026-09-04T00:00:00Z");
db.orders.aggregate([
  { $match: {
      updated_at: { $gte: lastRun },                  // watermark: only new/changed
      created_at: { $gte: ISODate("2026-09-04T00:00:00Z"),
                    $lt:  ISODate("2026-09-05T00:00:00Z") }
  }},
  { $addFields: { day: { $dateToString: { format: "%Y-%m-%d", date: "$created_at" } } } },
  { $group: {
      _id: { day: "$day", customer_id: "$customer_id" },
      revenue_cents: { $sum: "$total_cents" },
      order_count:   { $sum: 1 }
  }},
  { $project: { _id: 0, day: "$_id.day", customer_id: "$_id.customer_id",
                revenue_cents: 1, order_count: 1 } },
  { $merge: {
      into: "daily_rollup",
      on: ["day", "customer_id"],
      whenMatched: "replace",        // idempotent re-run of a day
      whenNotMatched: "insert"
  }}
], { allowDiskUse: true });

// 2. Export just the small rollup to Snowflake (connector / mongoexport)
//    -> ships megabytes of rollup, not terabytes of raw orders

Step-by-step trace.

Layer	Choice	Why
Shard key	`customer_id` hashed	even writes; per-customer queries targeted
Watermark	`updated_at >= lastRun`	process only changed documents
Grouping	`(day, customer_id)`	one rollup row per customer-day
Merge key	unique `{day, customer_id}`	deterministic matched/not-matched
`whenMatched`	`replace`	idempotent day re-runs
Export	rollup only	move the aggregate, not the raw data

The hashed customer_id shard key spreads inserts so no shard hot-spots, and because customer-scoped reads carry that key they stay targeted. The updated_at watermark limits the pipeline to changed documents; the (day, customer_id) group + $merge whenMatched: "replace" makes a re-run of any day idempotent, correcting it without touching other days. Only the compact daily_rollup crosses the wire to Snowflake.

Output:

day	customer_id	revenue_cents	order_count
2026-09-04	c1	4200	1
2026-09-04	c2	1850	1

Why this works — concept by concept:

$merge whenMatched: replace — keyed on a unique (day, customer_id) index, the merge overwrites exactly the recomputed rows and inserts new ones, so re-running a day is idempotent and never corrupts history — the defining property of an incremental materialized view.
Watermark on updated_at — filtering to updated_at >= lastRun means each run processes only changed documents, turning a full recompute into an incremental one; the watermark advances each run like log-based/timestamp CDC.
Hashed shard key on customer_id — hashing spreads inserts evenly (no monotonic hot shard), while choosing customer_id keeps the common per-customer analytics targeted to one shard instead of scatter-gathering across all of them.
Rollup-then-export — the heavy $group runs on-cluster where the data lives, and only the small daily_rollup is exported, so the warehouse feed moves megabytes, not terabytes.
Cost — O(changed docs) per run for the watermark-limited scan, O(groups) for the merge, and O(rollup size) for the export, versus O(all orders) for a full nightly rebuild and O(shards) for a scatter-gather. Incrementality and shard-key targeting keep every term proportional to change, not to total data.

Database
Topic — database
Sharding, $merge, and warehouse-export problems

Practice →

Aggregation
Topic — aggregation
Materialized-rollup and incremental-aggregation problems

Practice →

Cheat sheet — MongoDB aggregation recipes

Embed vs reference. Embed data that is read together and bounded (order + line items); reference data that is shared across parents or unbounded (a customer referenced by many orders, a user's millions of events). The hard limit is the 16 MB document cap — any array that can grow without bound must be referenced, not embedded.
Extraction projection. End every warehouse-feed pipeline with an explicit $project that coerces types: { $toString: "$_id" } for ids, { $toDate: "$_id" } to recover a creation timestamp from an ObjectId, { $toDecimal: { $divide: ["$cents", 100] } } for money. Coerce at the source, never in the warehouse.
Stage skeleton. [$match → $group → $sort → $limit → $project]. Lead with $match (index-eligible, shrinks the stream), fold with $group ($sum, $avg, $push, $sum: 1 for count), rank with adjacent $sort + $limit (coalesced top-N), shape last with $project. Set allowDiskUse: true when a $group/$sort can exceed 100 MB.
$project vs $addFields. $project replaces the document with only the listed fields; $addFields (alias $set) augments and keeps the rest. Use $addFields for intermediate computed values, $project for the final minimal contract. $match after $group is SQL HAVING.
$lookup join. Equality form { from, localField, foreignField, as } for a foreign-key attach — always index the foreignField (joining to _id is free). Sub-pipeline form { from, let, pipeline, as } with $expr to filter/aggregate the joined side. The as field is always an array; $unwind it to get an object/scalar.
$unwind semantics. $unwind: "$arr" emits one document per element and drops empty/missing arrays; add { path: "$arr", preserveNullAndEmptyArrays: true } for left-join semantics. Explode late (after $match) and $group back down immediately, because the stream grows N×.
ESR index rule. Order compound-index fields Equality → Sort → Range: equality predicates first (seek), the sort field next (index-provided order, no in-memory SORT), range predicates last. A compound index serves only a left prefix of its fields — {status, created_at} serves status or status+created_at, never created_at alone.
explain() triage. Run db.coll.explain("executionStats").aggregate([...]). Read winningPlan.stage (IXSCAN good, COLLSCAN bad), and totalDocsExamined vs nReturned — a large ratio means you are scanning. A lingering SORT stage means your index order violates ESR.
Covered query. When the index holds every field the query filters and returns, and you exclude _id in the projection, the query is answered from the index alone (totalDocsExamined: 0). Build a covering compound index for hot, few-field reads.
Multikey index. Index an array path ({ "line_items.sku": 1 }) to keep embedded arrays seekable; each element is indexed. Restriction: an index cannot be multikey on two array fields at once — one array path per index.
$out vs $merge. $out fully replaces the target collection (atomic full snapshot) — use for truncate-and-load exports of modest collections. $merge upserts with on: [keys], whenMatched (replace/merge/keepExisting/pipeline), whenNotMatched (insert/discard) — use for incremental materialized rollups keyed by a unique index; whenMatched: "replace" makes day re-runs idempotent.
Sharding. Pick a shard key with high cardinality that appears in your hottest queries; hash it when the natural key is monotonic (timestamps, ObjectIds) to avoid a hot shard. Queries carrying the shard key are targeted to one shard; queries without it scatter-gather across all shards — put the shard key in an early $match. For warehouse feeds, $merge a rollup then export the rollup, and use change streams (watch()) for continuous CDC.

Frequently asked questions

What is the MongoDB aggregation pipeline in one sentence?

The mongodb aggregation pipeline is an ordered array of stages passed to db.collection.aggregate([...]), where each stage consumes the stream of documents produced by the previous stage and emits a transformed stream — filtering with $match, folding with $group, joining with $lookup, flattening with $unwind, and reshaping with $project — until the final stage returns (or, with $out/$merge, writes) the result. It is MongoDB's answer to SQL's SELECT … WHERE … GROUP BY … JOIN, but written as an explicit, ordered sequence you control stage by stage. Because the stages run in the order you write them, stage ordering is part of the program: a $match placed first can use an index and shrink the stream before any expensive stage runs.

Embed or reference — how do I decide?

Embed data that is read together and bounded; reference data that is shared or unbounded. An order and its handful of line items are read together and few in number, so embed the line items as an array inside the order — one read returns the whole aggregate with no join. A customer referenced by thousands of orders is shared, so store the customer once and reference it by _id — otherwise a name change means rewriting thousands of copies. The hard constraint is the 16 MB document cap: any array that can grow without bound (a user's events, a post's comments) must be referenced into its own collection, often with a small embedded preview on the parent for the common read. Model for the read pattern first; normalize only where sharing or unbounded growth forces your hand.

Is $lookup a real join, and how slow is it?

$lookup is a genuine left outer join, but it is implemented as a nested loop: for each input document it probes the foreign collection, attaching matches as an array on the as field. That means its cost depends entirely on whether the foreignField is indexed. Joining orders.customer_id to customers._id uses the built-in _id index, so each probe is an O(log n) seek and the whole join is O(orders · log customers) — fast. Without an index on the foreignField, each probe becomes a full collection scan and the join degrades to O(n · m), which is catastrophic at scale. Two rules follow: always index the field you join to, and filter the driving collection with a $match before the $lookup so you perform fewer probes. For filtering or aggregating the joined side, use the sub-pipeline form (let + $expr) so each probe returns only what you need.

$match vs $group vs $project — what does each stage do?

$match filters the document stream using the same query syntax as find(); placed first, it can use an index, and placed after a $group it filters grouped output (SQL's HAVING). $group folds all documents sharing the same _id expression into one output document, computing accumulators like $sum, $avg, $min, $max, and $push; _id: null collapses everything into a single total, and $sum: 1 is the idiomatic row count. $project reshapes each document, keeping only the fields you list and adding computed ones — use it to lock down the final output contract; its cousin $addFields (alias $set) instead augments the document with new fields while keeping the rest, which is what you want for intermediate values a later stage still needs.

How do I make an aggregation use an index?

Only a $match (or $sort) at the front of the pipeline can use a collection index — once a $group or transforming $project has run, later stages filter in memory. So lead with $match, and build a compound index ordered by the ESR rule: Equality fields first, then the Sort field, then Range fields. Verify with db.coll.explain("executionStats").aggregate([...]) and read two numbers: winningPlan.stage should be IXSCAN (not COLLSCAN), and totalDocsExamined should be close to nReturned — a large gap means you are scanning. A lingering SORT stage in the plan means your sort field is in the wrong index position; move all equality fields ahead of it. For hot reads that return only a couple of fields, build a covering index and exclude _id so the query is answered from the index alone.

$out vs $merge — which do I use for a warehouse feed?

Use $out when you want a full snapshot: it atomically replaces the entire target collection with the pipeline's output, ideal for a modest truncate-and-load export where rebuilding everything each run is acceptable. Use $merge when you want an incremental materialized rollup: it upserts the output into an existing target keyed by a unique index, with whenMatched (replace, merge, keepExisting, or a custom pipeline) and whenNotMatched (insert, discard) controlling per-document behavior. The standard warehouse pattern is $merge with whenMatched: "replace" on a (day, key) unique index plus an updated_at watermark: each run recomputes only changed days and corrects them idempotently, and you export the small rollup rather than the raw collection. On a sharded cluster, keep the aggregation targeted by carrying the shard key in an early $match, and consider change streams (watch()) for continuous CDC instead of batch re-runs.

Practice on PipeCode

Drill the database practice library → for the document-modelling, indexing, explain(), sharding, and $merge problems senior interviewers love.
Rehearse on the aggregation practice library → for the $match/$group/$project stage-ordering, $lookup + $unwind join, and incremental-rollup patterns.
Sharpen document-shape fluency on the JSON practice library → for nested documents, embedded arrays, and polymorphic-schema queries.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the embed-vs-reference and ESR-index decisions against real graded inputs.

Lock in MongoDB aggregation muscle memory

Docs explain the stages. PipeCode drills explain the decision — when to embed versus reference, why `$match` goes first, how `$lookup` becomes a nested-loop trap without an index, how the ESR rule turns a COLLSCAN into an IXSCAN, and when `$merge` beats `$out` for a warehouse feed. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs data engineers actually face.

Practice aggregation problems →
Practice database problems →

Cassandra & ScyllaDB: Wide-Column Data Modeling Done Right

Gowtham Potureddi — Tue, 08 Sep 2026 13:54:28 +0000

cassandra data modeling is the discipline that decides whether a query returns in two milliseconds or times out under load — and it is the single skill relational engineers get wrong most often, because every instinct they carry from Postgres and MySQL is actively harmful here. In a wide-column store there are no joins, no ad-hoc WHERE clauses, no foreign keys, and no query planner rescuing a badly-shaped schema at runtime. The data model is the query plan: you decide, at design time, exactly which reads the cluster will serve cheaply, and any read you did not plan for is either impossible or catastrophically slow. Normalizing a schema — the thing you were trained to do for two decades — produces a design that cannot be queried, because the row you want lives on a node the coordinator has no way to find without scanning every partition on every machine.

This guide is the walkthrough you wished existed the first time an interviewer asked "model a messaging inbox for Cassandra" and your carefully-normalized users/messages/threads diagram earned a slow shake of the head. It rebuilds the mental model from the physical layout up: what a wide-column store actually stores on disk (a partitioned, sorted map of maps written to an LSM tree), how the two halves of the primary key — the partition key that routes a row to a node and the clustering key that sorts rows within a partition — determine every access path, why query-first denormalization means duplicating the same fact into one purpose-built table per read, and the three failure modes that end careers in production: tombstones that turn deletes into read-time landmines, hot partitions that funnel a whole cluster's traffic onto one overloaded node, and oversized partitions that blow past the size budget and stall compaction. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the database practice library →, rehearse pipeline shaping on the data-processing practice library →, and sharpen your schema instincts on the dimensional-modeling practice library →.

On this page

The wide-column data model
Partition keys and clustering keys
Query-first denormalization
Pitfalls — tombstones, hot and large partitions
Cassandra vs ScyllaDB and tuning
Cheat sheet — wide-column modeling recipes
Frequently asked questions
Practice on PipeCode

1. The wide-column data model

A wide-column store is a partitioned, sorted map of maps — not a table of rows, and every relational instinct you have is a liability

The one-sentence invariant: a wide-column store like Cassandra or ScyllaDB persists a two-level nested map — an outer map from partition key to partition, and an inner map from clustering key to row — physically sorted on disk within each partition, replicated across a ring of nodes, and written through a log-structured merge tree that never updates data in place, which means the schema you design is a direct encoding of the exact reads you will serve and nothing else. There is no query optimizer that will rescue a normalized design at runtime; the access path is frozen the moment you choose the primary key. If you can name the partition and you can name (or range-scan) the clustering values inside it, the read is O(1) coordinator routing plus a sorted-range disk read; if you cannot, the read is either rejected outright or degenerates into a cluster-wide scatter-gather that pages on-call at 3 AM.

The axes that matter — what "wide-column" actually means.

Partitioned. Every row belongs to exactly one partition, chosen by hashing the partition key to a token. The token maps to a position on a ring of nodes; the partition (and all its rows) lives on the node(s) that own that token range. Cross-partition queries have no efficient path — the coordinator would have to ask every node.
Sorted. Within a partition, rows are stored physically sorted by the clustering key. This makes range reads inside a partition (WHERE partition_key = ? AND clustering_col > ?) a contiguous disk scan — the single most important performance property of the model.
Map of maps. Conceptually partition_key -> (clustering_key -> row). A "wide" partition can hold millions of rows/cells; a "narrow" one holds a handful. The width is a design choice with a hard budget, covered in section 4.
LSM-tree storage. Writes go to an in-memory memtable plus a commit log, then flush to immutable SSTables. Updates and deletes are appended as new versions (or tombstones), and background compaction merges SSTables. Nothing is mutated in place — a fact that explains tombstones, read amplification, and compaction strategy all at once.

The 2026 reality — why teams still reach for wide-column.

Linear write scalability. Cassandra and ScyllaDB scale writes linearly by adding nodes, with no single write master. Append-only LSM storage makes writes cheap and predictable — the workload profile behind time-series, event logging, IoT telemetry, messaging, and feature stores.
No single point of failure. A masterless ring with tunable replication (RF) survives node and even datacenter loss with the right consistency level. This availability story is the reason the pattern outlives every "just use Postgres" objection at scale.
Predictable latency at volume. Because the access path is fixed by the schema and there is no planner variance, a well-modeled table delivers flat p99 latency into the billions of rows — provided you never issue a query the model was not built for.
The catch. All of that is conditional on modeling for the query. The same engine that serves 2 ms reads at a million writes/second will fall over on a single ALLOW FILTERING scan or one unbounded partition. The engine gives you a scalpel with no safety guard.

What interviewers listen for.

Do you say "model for the query, not the entity" in the first sentence? — required answer.
Do you name the storage engine as an LSM tree and connect it to tombstones and compaction? — senior signal.
Do you state "there are no joins and no ad-hoc WHERE" and design around it instead of wishing it away? — required answer.
Do you describe a table as a "partition of sorted rows" rather than as "a table like in SQL"? — senior signal.
Do you refuse to reach for ALLOW FILTERING and instead add a purpose-built table? — senior signal.

Worked example — reading a wide-column table as a map of maps

Detailed explanation. The fastest way to internalize the model is to stop picturing a spreadsheet grid and start picturing nested dictionaries. A CQL table with a compound primary key is a dictionary keyed by the partition key, whose values are dictionaries keyed by the clustering key, whose values are the non-key columns. Walk through a per-user activity feed.

Outer map. user_id (partition key) → the user's partition.
Inner map. activity_ts (clustering key, descending) → one activity row.
Leaf. the columns action, target_id, metadata for that timestamp.
Physical consequence. All of a user's activity lives contiguously on one set of replicas, sorted newest-first, so "latest 20 activities for user X" is one node hop and one sorted-range read.

Question. Model a per-user activity feed so that "the most recent N activities for a given user" is a single-partition range read, and show the conceptual map it produces.

Input.

Requirement	Value
Read query	latest N activities for one user
Partition key	user_id
Clustering key	activity_ts DESC
Row payload	action, target_id, metadata

Code.

-- CQL: one partition per user, rows sorted newest-first
CREATE TABLE activity_by_user (
    user_id      uuid,
    activity_ts  timestamp,
    action       text,
    target_id    uuid,
    metadata     map<text, text>,
    PRIMARY KEY ((user_id), activity_ts)
) WITH CLUSTERING ORDER BY (activity_ts DESC);

-- The only read this table is built for:
SELECT activity_ts, action, target_id
FROM   activity_by_user
WHERE  user_id = 8f3b...       -- names the partition
LIMIT  20;                     -- sorted range read, newest first

Step-by-step explanation.

PRIMARY KEY ((user_id), activity_ts) declares user_id as the partition key (the double parentheses) and activity_ts as the clustering key. Every row for a user lands in one partition on one set of replicas.
WITH CLUSTERING ORDER BY (activity_ts DESC) stores rows physically newest-first on disk. The LIMIT 20 read then reads the first 20 rows off the front of the sorted partition — no sort at query time, no scan of older rows.
Conceptually the table is { user_id: { activity_ts: {action, target_id, metadata} } }. The read WHERE user_id = ? selects the inner map; LIMIT 20 takes its head.
There is no efficient way to ask "all activity across all users on a given day" against this table — that query names no partition. If the business needs it, you build a second table keyed by day (section 3).
The metadata map<text,text> is a collection column stored inline in the row. Collections are convenient but have their own tombstone hazards (section 4); prefer them small and bounded.

Output.

user_id	activity_ts	action	Read cost
8f3b…	2026-09-05 10:04	comment	1 partition, sorted head
8f3b…	2026-09-05 09:58	like	contiguous next row
8f3b…	2026-09-05 09:41	follow	contiguous next row

Rule of thumb. Read every wide-column table as partition_key -> (clustering_key -> row). If your target query can name the partition and take a sorted slice of the clustering values, the table is right; if it cannot, you need a different table, not a cleverer query.

Worked example — modeling a lookup table and a static column

Detailed explanation. Not every table is a wide time-series. The two other canonical shapes are the narrow key-value lookup (one row per partition) and the partition-with-header pattern using a static column (per-partition metadata shared across all clustering rows). Build both for a chat application.

Lookup. user_by_id — partition key user_id, no clustering key, one row per partition. Pure key-value.
Static column. messages_by_thread — partition key thread_id, clustering key message_ts, plus a static column thread_name stored once per partition, not once per message.
Why static. The thread name is the same for every message in the thread; a static column stores it once and updates it in one place.

Question. Model user_by_id as a key-value lookup and messages_by_thread with a per-thread static column, and explain when a static column beats duplicating the value on every row.

Input.

Table	Partition key	Clustering key	Static column
user_by_id	user_id	(none)	(none)
messages_by_thread	thread_id	message_ts	thread_name

Code.

-- Narrow key-value lookup: one row per partition
CREATE TABLE user_by_id (
    user_id  uuid PRIMARY KEY,
    email    text,
    name     text,
    created  timestamp
);

-- Wide table with a per-partition static column
CREATE TABLE messages_by_thread (
    thread_id    uuid,
    message_ts   timestamp,
    thread_name  text STATIC,     -- stored once per partition
    sender_id    uuid,
    body         text,
    PRIMARY KEY ((thread_id), message_ts)
) WITH CLUSTERING ORDER BY (message_ts ASC);

-- Update the thread name once, not per message:
UPDATE messages_by_thread SET thread_name = 'Launch plan'
WHERE  thread_id = 1a2b...;      -- no clustering key needed for static writes

Step-by-step explanation.

user_by_id has a single-column primary key, so the partition key is the whole key and there is no clustering key. Each partition holds exactly one row — the classic distributed hash-map. Reads are WHERE user_id = ?, always O(1) routing.
messages_by_thread keys by thread_id (partition) and message_ts (clustering), so all messages in a thread live together, sorted oldest-first for natural chat display.
thread_name text STATIC is stored once per partition rather than once per message row. A 10,000-message thread stores the name once, not 10,000 times, and renaming the thread is a single write.
Static columns can be written without a clustering key value (as in the UPDATE above), because they belong to the partition, not to any one row. Reading any message row also returns the current thread_name.
Use a static column when a value is (a) identical across all rows in a partition and (b) mutable — otherwise you would either duplicate it on every row (wasting space and forcing a fan-out update) or split it into a separate lookup table (forcing a second read).

Output.

Pattern	Rows per partition	Best for
Key-value lookup	1	entity-by-id reads
Wide + clustering	many	feeds, time-series, threads
Static column	many + 1 shared	per-partition header/metadata

Rule of thumb. Reach for a static column when a mutable value is constant across a partition; reach for a separate lookup table when the value is queried on its own; never duplicate a mutable value onto every clustering row — that turns one logical update into a partition-wide fan-out.

Data engineering interview question on the wide-column data model

A senior interviewer often opens with: "Design the Cassandra schema for a messaging product. Users have many conversations; each conversation has many messages. The app must render (a) a user's list of conversations most-recent-first, and (b) the messages inside a conversation oldest-first with pagination. Give me the tables, the primary keys, and explain why a normalized users/conversations/messages design fails here."

Solution Using query-first wide-column tables keyed for each read

-- Query A: a user's conversations, most-recent-active first
CREATE TABLE conversations_by_user (
    user_id           uuid,
    last_message_ts   timestamp,
    conversation_id   uuid,
    peer_name         text,
    last_snippet      text,
    PRIMARY KEY ((user_id), last_message_ts, conversation_id)
) WITH CLUSTERING ORDER BY (last_message_ts DESC);

-- Query B: messages inside a conversation, oldest-first, paginated
CREATE TABLE messages_by_conversation (
    conversation_id   uuid,
    message_ts        timestamp,
    message_id        timeuuid,
    sender_id         uuid,
    body              text,
    PRIMARY KEY ((conversation_id), message_ts, message_id)
) WITH CLUSTERING ORDER BY (message_ts ASC, message_id ASC);

-- Read A: conversation list for a user (single partition, sorted)
SELECT conversation_id, peer_name, last_snippet, last_message_ts
FROM   conversations_by_user
WHERE  user_id = 8f3b... LIMIT 30;

-- Read B: first page of a conversation, then paginate by message_ts
SELECT message_ts, sender_id, body
FROM   messages_by_conversation
WHERE  conversation_id = 1a2b...
  AND  message_ts > '2026-09-05 00:00:00' LIMIT 50;

Step-by-step trace.

Step	Input	What happens
1	Read A: user_id = 8f3b…	coordinator hashes user_id → token → replicas own the partition
2	LIMIT 30	reads 30 rows off the front of the DESC-sorted partition
3	Read B: conversation_id = 1a2b…	hashes conversation_id → different partition, different owning replicas
4	message_ts > cursor	contiguous sorted-range scan from the cursor forward
5	LIMIT 50	returns one page; next page uses the last message_ts as the new cursor
6	new message arrives	app writes messages_by_conversation AND updates conversations_by_user

Both reads name their partition and take a sorted slice, so each is a single-node hop plus a contiguous disk read. Pagination is cursor-based on the clustering column — no OFFSET, which does not exist and would be an anti-pattern anyway.

Output:

Query	Table	Partition named?	Cost
A — conversation list	conversations_by_user	yes (user_id)	1 partition, sorted head
B — message page	messages_by_conversation	yes (conversation_id)	1 partition, sorted range
(rejected) all msgs by sender	none built	no	would require ALLOW FILTERING

Why this works — concept by concept:

Query-first modeling — the reads were enumerated before the tables. Each of the two required reads got its own table whose primary key makes that read a single-partition sorted access. The entities (user, conversation, message) never became tables; the queries became tables.
Partition key = routing — user_id and conversation_id are the partition keys because they are the values each read already knows. Naming the partition is what turns a distributed lookup into a single-node hop.
Clustering key = order + pagination — last_message_ts DESC and message_ts ASC bake the required sort order into the on-disk layout, and cursor pagination rides the clustering column instead of a nonexistent OFFSET.
Why the normalized design fails — a normalized messages(conversation_id, sender_id, ...) table cannot answer "conversations for a user" without a join (there are none) or a scan (no partition named). You would be forced into ALLOW FILTERING, a cluster-wide scatter-gather that does not scale.
Cost — two tables, two writes per new message (one per table), each read O(1) routing + O(page) sorted scan. Storage grows with duplication but storage is cheap; the eliminated cost is the O(N) scan a normalized model would require on every read. Net: bounded write fan-out buys flat read latency at any scale.

Database
Topic — database
Wide-column and NoSQL data-model problems

Practice →

Modeling Topic — dimensional-modeling Data-modeling problems for query-first schemas

Practice →

2. Partition keys and clustering keys

The partition key decides which node stores the row; the clustering key decides the order within that node — together they are the entire access path

The mental model in one line: the primary key of a wide-column table is two functionally distinct things fused together — the partition key, which is hashed to a token that routes the row to a set of replica nodes and is the only value that makes a read efficient, and the clustering key, which sorts rows on disk inside the partition and is the only value you can range-scan or paginate over — so choosing the primary key is choosing, permanently, both the physical distribution of your data across the cluster and the exact shape of every read the table can serve. Get the partition key wrong and you get hot spots or unbounded partitions; get the clustering key wrong and the reads you need become impossible; there is no CREATE INDEX that fully undoes either mistake.

The anatomy of a compound primary key.

PRIMARY KEY ((a, b), c, d). The first parenthesised group (a, b) is a composite partition key — both columns are hashed together to produce one token. c and d are clustering columns that sort rows within the partition, in that priority order.
PRIMARY KEY (a, b, c). With no inner parentheses, only the first column a is the partition key; b and c are clustering columns. This single-vs-composite distinction is the most common beginner error.
Query constraint. A read must supply the entire partition key with equality (= or IN). Clustering columns can then be constrained left-to-right with equality and a final range — you cannot skip a clustering column and constrain a later one.
Sort order. WITH CLUSTERING ORDER BY (c DESC, d ASC) fixes the physical order. Reads in that order are free; reads in the reverse order are supported but read back-to-front; reads in an unrelated order are not supported.

Choosing the partition key — the cardinality-versus-size trade-off.

High enough cardinality. The partition key must have enough distinct values to spread load across the whole ring. country (≈200 values) is a terrible partition key; user_id (millions) is good.
Bounded partition size. Each partition must stay under the size budget (≈100 MB / ≈100k cells — section 4). If one partition key value accumulates unbounded rows (e.g. a global event_type), you must add a bucketing component to the partition key.
Even access distribution. Cardinality is necessary but not sufficient — the access must be even too. A user_id partition key with one celebrity user who gets 40% of reads is a hot partition despite high cardinality (section 4).
Contains the query's known value. The partition key must be a value every target read already has in hand. You cannot key by order_id if the read only knows customer_id.

Clustering columns — order, range, and the ALLOW FILTERING trap.

Left-to-right constraint. With clustering (year, month, day), you can query year = ? AND month = ? AND day > ?, but not day = ? alone — the engine cannot seek to a later clustering column without pinning the earlier ones.
Range on the last constrained column. The final clustering predicate may be a range (>, <, >=, <=); all earlier ones must be equality. This is what makes time-range reads inside a partition efficient.
ALLOW FILTERING is a red flag. It tells the engine to read rows and discard non-matches — an O(partition) or O(cluster) scan. In an interview, proposing ALLOW FILTERING for a production read is close to disqualifying; the correct answer is almost always "add a table or change the key."
Secondary indexes are narrow. CREATE INDEX on a non-key column works for low-cardinality, single-partition-scoped filters but fans out across the cluster for global ones; it is not a substitute for modeling the query into the primary key.

Worked example — single vs composite partition key for sensor readings

Detailed explanation. A fleet of IoT sensors emits one reading per second. The naive schema keys by sensor_id alone, but a chatty sensor running for a year accumulates ~31 million rows in one partition — far past the size budget. The fix is a composite partition key that buckets by time, bounding each partition while keeping time-range reads efficient. Walk through both designs.

Naive. PRIMARY KEY ((sensor_id), reading_ts) — one partition per sensor, grows without bound.
Bucketed. PRIMARY KEY ((sensor_id, day_bucket), reading_ts) — one partition per sensor per day, each holding ≈86,400 rows.
Read impact. A single-day read names one partition; a multi-day read issues one query per day bucket (client-side fan-out over a known, bounded set of buckets).

Question. Design a sensor-reading table whose partitions stay under the size budget while "readings for sensor X between two timestamps" remains a single-partition (or bounded multi-partition) sorted read.

Input.

Parameter	Value
Write rate	1 reading/sec/sensor
Naive partition size	~31M rows/year (too big)
Bucket granularity	1 day
Bucketed partition size	~86,400 rows/day

Code.

-- Bucketed composite partition key bounds each partition to one day
CREATE TABLE readings_by_sensor_day (
    sensor_id    uuid,
    day_bucket   date,        -- e.g. '2026-09-05'
    reading_ts   timestamp,
    temperature  double,
    humidity     double,
    PRIMARY KEY ((sensor_id, day_bucket), reading_ts)
) WITH CLUSTERING ORDER BY (reading_ts ASC);

-- Single-day read: names the whole partition key
SELECT reading_ts, temperature
FROM   readings_by_sensor_day
WHERE  sensor_id  = 7c1e...
  AND  day_bucket = '2026-09-05'
  AND  reading_ts >= '2026-09-05 08:00:00'
  AND  reading_ts <  '2026-09-05 12:00:00';

# Multi-day read: fan out over a bounded, known set of day buckets
from datetime import date, timedelta

def day_buckets(start: date, end: date):
    d = start
    while d <= end:
        yield d
        d += timedelta(days=1)

def read_range(session, sensor_id, start, end):
    stmt = session.prepare("""
        SELECT reading_ts, temperature FROM readings_by_sensor_day
        WHERE sensor_id = ? AND day_bucket = ?
          AND reading_ts >= ? AND reading_ts <= ?
    """)
    rows = []
    for bucket in day_buckets(start.date(), end.date()):
        rows.extend(session.execute(stmt, (sensor_id, bucket, start, end)))
    return rows

Step-by-step explanation.

The composite partition key (sensor_id, day_bucket) hashes both columns together, so each (sensor, day) pair is its own partition on its own replicas. A year of data for one sensor becomes 365 bounded partitions instead of one giant one.
A single-day read supplies both partition-key columns with equality, then range-scans reading_ts — one partition, one sorted disk range. This is the fast path the model exists to serve.
A multi-day read cannot name a single partition (the day varies), but the set of day buckets is known and bounded from the query range. The client issues one query per bucket — a controlled fan-out, not an ALLOW FILTERING scan.
Bucket granularity is a tuning knob: too coarse (monthly) and partitions grow past the budget again; too fine (hourly) and multi-day reads fan out over too many partitions. Size the bucket so a full partition sits comfortably under ~100 MB.
reading_ts as the clustering column gives natural chronological order and cheap range scans; ASC matches the most common "walk forward in time" access.

Output.

Design	Partition size	Single-day read	Multi-day read
((sensor_id), reading_ts)	~31M rows/yr (fails)	one huge partition	one huge partition
((sensor_id, day_bucket), reading_ts)	~86k rows/day	1 partition	N buckets, bounded fan-out

Rule of thumb. When one partition-key value accumulates rows over time, add a time bucket to the partition key. Size the bucket so a full partition stays under ~100 MB, and drive multi-bucket reads from the query's own time range — never from ALLOW FILTERING.

Worked example — clustering order and range reads

Detailed explanation. Clustering order is not cosmetic — it is the physical on-disk layout, and it determines which reads are cheap. A leaderboard that must return "top 10 scores for a game" needs scores stored descending so the top is the head of the partition. Walk through the design and the range-read semantics.

Requirement. Top-N scores per game, highest first.
Clustering. (score DESC, player_id ASC) — highest score first, ties broken by player.
Range read. "scores above a threshold" is score > ? on the descending-sorted partition — a contiguous head scan.

Question. Design a leaderboard table so "top 10 for a game" and "all scores above a threshold" are both single-partition sorted reads, and explain the left-to-right clustering constraint.

Input.

Requirement	Value
Partition key	game_id
Clustering	score DESC, player_id ASC
Read 1	top 10 (LIMIT 10)
Read 2	score > threshold

Code.

CREATE TABLE leaderboard_by_game (
    game_id    uuid,
    score      int,
    player_id  uuid,
    player_name text,
    PRIMARY KEY ((game_id), score, player_id)
) WITH CLUSTERING ORDER BY (score DESC, player_id ASC);

-- Read 1: top 10, highest first — reads the head of the partition
SELECT player_name, score FROM leaderboard_by_game
WHERE  game_id = a91f... LIMIT 10;

-- Read 2: everyone above 9000 — contiguous head range
SELECT player_name, score FROM leaderboard_by_game
WHERE  game_id = a91f... AND score > 9000;

-- REJECTED: cannot constrain player_id without constraining score first
-- SELECT * FROM leaderboard_by_game WHERE game_id = a91f... AND player_id = ...;

Step-by-step explanation.

CLUSTERING ORDER BY (score DESC, player_id ASC) stores rows highest-score-first on disk. "Top 10" is literally the first 10 rows of the partition — no sort, no scan of the tail.
score > 9000 is a range on the first clustering column, which is allowed and cheap: it reads from the head down to the first row at or below 9000, then stops.
The rejected query tries to constrain player_id (the second clustering column) without constraining score (the first). The engine cannot seek to a specific player_id without knowing its score, because the physical order is score-major. This is the left-to-right rule.
To support "find a player's rank" you would build a second table keyed differently (e.g. score_by_game_player keyed by (game_id, player_id)) — again, one table per query.
Ties on score are ordered by player_id ASC, which makes pagination deterministic: the (score, player_id) pair is a total order, so cursor pagination never skips or repeats a row.

Output.

Query	Constrains	Legal?	Cost
top 10	game_id + LIMIT	yes	head of partition
score > 9000	game_id, score range	yes	contiguous head range
player_id = X	game_id, player_id (skips score)	no	not supported

Rule of thumb. Order clustering columns by how you read them: put the column you range-scan or take the top of first, and remember you can only constrain clustering columns left-to-right with a single trailing range. If a read needs a different order, it needs a different table.

Data engineering interview question on partition and clustering keys

A senior interviewer might ask: "You have a ride-sharing app. You must serve (a) 'all trips for a driver in a given month, newest first' and (b) never let a driver's partition exceed the size budget even for a driver doing 500 trips a day for years. Design the primary key, justify the partition and clustering choices, and show how a cross-month read works."

Solution Using a bucketed composite partition key with a descending clustering column

CREATE TABLE trips_by_driver_month (
    driver_id     uuid,
    month_bucket  text,        -- 'YYYY-MM', bounds the partition
    trip_ts       timestamp,
    trip_id       timeuuid,
    fare_cents    bigint,
    distance_km   double,
    PRIMARY KEY ((driver_id, month_bucket), trip_ts, trip_id)
) WITH CLUSTERING ORDER BY (trip_ts DESC, trip_id DESC);

-- Read A: one month of trips for a driver, newest first
SELECT trip_ts, fare_cents, distance_km
FROM   trips_by_driver_month
WHERE  driver_id = 3d90... AND month_bucket = '2026-09'
LIMIT  50;

# Cross-month read: enumerate the bounded set of month buckets in range
def month_buckets(start, end):
    y, m = start.year, start.month
    out = []
    while (y, m) <= (end.year, end.month):
        out.append(f"{y:04d}-{m:02d}")
        m += 1
        if m > 12:
            m, y = 1, y + 1
    return out

def trips_in_range(session, driver_id, start, end):
    stmt = session.prepare("""
        SELECT trip_ts, fare_cents FROM trips_by_driver_month
        WHERE driver_id = ? AND month_bucket = ?
          AND trip_ts >= ? AND trip_ts <= ?
    """)
    rows = []
    for b in month_buckets(start, end):        # bounded, known fan-out
        rows.extend(session.execute(stmt, (driver_id, b, start, end)))
    return rows

Step-by-step trace.

Step	Input	What happens
1	driver_id=3d90, month='2026-09'	both partition-key cols supplied → single partition
2	500 trips/day × 30 days	~15k rows/partition — well under the budget
3	CLUSTERING trip_ts DESC	newest trips are the head of the partition
4	LIMIT 50	one page off the head, no tail scan
5	cross-month (Jun–Sep)	month_buckets() → ['2026-06','2026-07','2026-08','2026-09']
6	4 queries	bounded fan-out, one partition each, merge client-side

The month bucket caps each partition at roughly a month of one driver's trips (~15k rows) no matter how many years the driver stays active, and the descending clustering column makes "newest first" free. Cross-month reads fan out over a small, query-derived set of buckets rather than scanning.

Output:

Query	Partitions touched	Ordered?	Cost
one month	1	yes (DESC)	sorted head
four months	4	yes per bucket	bounded fan-out
all-time (rare)	N months	yes per bucket	fan-out sized by tenure

Why this works — concept by concept:

Composite partition key — (driver_id, month_bucket) hashes both columns to one token, so each driver-month is an independent, bounded partition on its own replicas. This is what caps partition size regardless of driver tenure.
Bucketing bounds size — the month bucket converts "one unbounded partition per driver" into "one bounded partition per driver per month," keeping every partition comfortably under the ~100 MB / ~100k-cell budget.
Descending clustering key — trip_ts DESC bakes "newest first" into the on-disk order, so the dominant read is a zero-cost head slice, and trip_id DESC makes the order total for safe pagination.
Query-derived fan-out — cross-month reads compute the exact bucket list from the request's date range, so the fan-out is bounded and predictable — the opposite of an ALLOW FILTERING scan whose cost is the whole table.
Cost — one write per trip, each read O(buckets-in-range) partition hops × O(page) sorted scan. Storage is linear in trips; there is no growth term that scales with driver tenure inside a single partition. Net: flat per-partition latency and a fan-out you can reason about.

Database
Topic — database
Partition-key and primary-key design problems

Practice →

Data processing Topic — data-processing Bucketing and time-partitioning problems

Practice →

3. Query-first denormalization

Enumerate the reads first, then build one purpose-built table per read — duplicating data on write is not a smell here, it is the design

The mental model in one line: query-first modeling inverts the relational workflow — instead of normalizing entities into third normal form and joining at read time, you list every read the application will issue, then create a separate denormalized table for each one whose primary key makes that read a single-partition access, accepting that the same fact is now duplicated across several tables and that keeping those copies consistent is the application's job, because in a wide-column store writes and storage are cheap while reads must be O(1) and joins do not exist. Denormalization is not a performance hack you reach for reluctantly; it is the first-class modeling primitive, and an interviewer who hears you say "I'd normalize and join" has already stopped listening.

The query-first workflow.

Step 1 — list the reads. Write down every access pattern the app needs, with the values it will have in hand at read time (the future partition keys) and the order it needs results in (the future clustering keys).
Step 2 — one table per read. Each distinct read becomes a table whose primary key encodes exactly that access. If two reads share a key shape, they can share a table; otherwise they do not.
Step 3 — pick the write path. Decide how a single logical change (a new order, an email update) propagates to every table that must reflect it: application fan-out, a logged BATCH, or a materialized view.
Step 4 — accept the duplication. The same fact lives in N tables. That is correct. Storage is cheap; the alternative — a read-time join — does not exist and would not scale if it did.

The cost you are trading.

Writes get more expensive. A change now writes to every denormalized table. This is fine: writes are the LSM tree's strength, and the fan-out is bounded and known at design time.
Storage grows. Duplicated data uses more disk. Also fine: storage is the cheapest resource in the system, and you are buying flat read latency with it.
Consistency becomes your problem. With no foreign keys and no transactions across partitions, the copies can drift if a write path partially fails. You mitigate with logged batches (atomicity across tables sharing a partition-key value), idempotent writes, and periodic reconciliation.
Reads get trivial. Every read names its partition and takes a sorted slice. This is the whole point — you moved all the complexity to write time, where the engine is fast, and away from read time, where it must be.

Keeping duplicates in sync — three mechanisms.

Application fan-out. The service writes each table explicitly. Simplest and most common; make writes idempotent (use fixed keys, not append semantics) so retries are safe.
Logged BATCH. A BEGIN BATCH ... APPLY BATCH groups writes so they all succeed or all get retried by the coordinator. It guarantees eventual atomicity, not isolation, and is cheapest when all statements share the same partition key. Overusing multi-partition batches creates coordinator pressure.
Materialized views. CREATE MATERIALIZED VIEW asks the engine to maintain a second table keyed differently from a base table automatically. Convenient, but with well-known operational caveats (write amplification, repair complexity); many senior teams prefer explicit application fan-out for critical paths.

Worked example — user-by-id and user-by-email dual tables

Detailed explanation. An auth service must read a user two ways: by user_id (after a session lookup) and by email (at login). Neither read can use the other's key, so query-first modeling produces two tables holding the same user, kept in sync by the application. Build both and the write path.

Read 1. by user_id → users_by_id.
Read 2. by email → users_by_email.
Write path. signup and profile-update write both tables; email change is a delete-plus-insert on users_by_email.

Question. Model the two lookup tables and show the idempotent write path for signup and for an email change.

Input.

Read	Key it has	Table
session → profile	user_id	users_by_id
login	email	users_by_email

Code.

CREATE TABLE users_by_id (
    user_id  uuid PRIMARY KEY,
    email    text,
    name     text,
    pw_hash  text
);

CREATE TABLE users_by_email (
    email    text PRIMARY KEY,
    user_id  uuid,
    name     text,
    pw_hash  text
);

def signup(session, user_id, email, name, pw_hash):
    # Idempotent: both writes use fixed primary keys, safe to retry
    session.execute("""
        INSERT INTO users_by_id (user_id, email, name, pw_hash)
        VALUES (%s, %s, %s, %s)
    """, (user_id, email, name, pw_hash))
    session.execute("""
        INSERT INTO users_by_email (email, user_id, name, pw_hash)
        VALUES (%s, %s, %s, %s)
    """, (email, user_id, name, pw_hash))

def change_email(session, user_id, old_email, new_email):
    # Partition key of users_by_email changes → delete old, insert new
    row = session.execute(
        "SELECT name, pw_hash FROM users_by_id WHERE user_id=%s", (user_id,)
    ).one()
    session.execute(
        "UPDATE users_by_id SET email=%s WHERE user_id=%s", (new_email, user_id))
    session.execute(
        "DELETE FROM users_by_email WHERE email=%s", (old_email,))
    session.execute("""
        INSERT INTO users_by_email (email, user_id, name, pw_hash)
        VALUES (%s, %s, %s, %s)
    """, (new_email, user_id, row.name, row.pw_hash))

Step-by-step explanation.

users_by_id and users_by_email store the same user redundantly, each keyed by the value one read already has. Neither read could use the other's key, so two tables are mandatory — this is denormalization by necessity, not by choice.
Signup writes both tables with INSERT using fixed primary keys. Because the keys are fixed (not append-generated), re-running the insert on a retry overwrites with identical data — the writes are idempotent, so partial failure plus retry converges.
An email change is subtle: email is the partition key of users_by_email, and partition keys are immutable. Changing the email means deleting the old-email row and inserting a new-email row — you cannot UPDATE a partition key.
The users_by_id side is a simple UPDATE because email there is a regular column, not part of the key.
If the process crashes between the delete and the insert, users_by_email temporarily lacks the user; a retry (idempotent) or a reconciliation job restores it. This is the consistency cost you accept for join-free reads.

Output.

Operation	users_by_id	users_by_email
signup	INSERT	INSERT
profile name update	UPDATE	UPDATE
email change	UPDATE email col	DELETE old + INSERT new

Rule of thumb. When two reads need two different keys for the same entity, build two tables and make every write idempotent with fixed keys. Remember partition keys are immutable — "changing" one is always delete-plus-insert, never update.

Worked example — a logged BATCH for atomic multi-table writes

Detailed explanation. When a single logical event must land in several tables and you want the coordinator to guarantee they all eventually apply, wrap them in a logged BATCH. It is not a relational transaction — there is no isolation and no rollback — but the batch log ensures that if the coordinator accepts the batch, every statement will eventually be applied even across node failures. Build an order-placement write that updates three query tables.

Tables. orders_by_id, orders_by_customer, orders_by_status.
Guarantee. logged batch = atomicity (all-or-eventually-all), not isolation.
Caveat. cheapest when statements share a partition key; multi-partition batches add coordinator overhead — use sparingly.

Question. Write the logged batch that records a new order into three query tables, and state precisely what the batch does and does not guarantee.

Input.

Table	Keyed by	Serves read
orders_by_id	order_id	order detail
orders_by_customer	customer_id, order_ts	a customer's orders
orders_by_status	status, order_ts	ops queue by status

Code.

CREATE TABLE orders_by_id (
    order_id    uuid PRIMARY KEY,
    customer_id uuid, status text, total_cents bigint, order_ts timestamp);

CREATE TABLE orders_by_customer (
    customer_id uuid, order_ts timestamp, order_id uuid,
    status text, total_cents bigint,
    PRIMARY KEY ((customer_id), order_ts, order_id))
    WITH CLUSTERING ORDER BY (order_ts DESC, order_id DESC);

CREATE TABLE orders_by_status (
    status text, order_ts timestamp, order_id uuid,
    customer_id uuid, total_cents bigint,
    PRIMARY KEY ((status), order_ts, order_id))
    WITH CLUSTERING ORDER BY (order_ts DESC, order_id DESC);

-- Logged batch: all three apply, or the coordinator retries until they do
BEGIN BATCH
  INSERT INTO orders_by_id (order_id, customer_id, status, total_cents, order_ts)
    VALUES (11a2..., 77c1..., 'pending', 4200, '2026-09-05 10:00:00');
  INSERT INTO orders_by_customer (customer_id, order_ts, order_id, status, total_cents)
    VALUES (77c1..., '2026-09-05 10:00:00', 11a2..., 'pending', 4200);
  INSERT INTO orders_by_status (status, order_ts, order_id, customer_id, total_cents)
    VALUES ('pending', '2026-09-05 10:00:00', 11a2..., 77c1..., 4200);
APPLY BATCH;

Step-by-step explanation.

The three tables denormalize the same order for three reads: by id, by customer, by status. A new order must appear in all three, so the write fans out three ways.
BEGIN BATCH ... APPLY BATCH (logged, the default) writes a batch record to the batchlog on two replicas first. If the coordinator dies mid-apply, another node replays the batchlog, guaranteeing every statement eventually applies.
The guarantee is atomicity over time, not isolation: a concurrent reader may see orders_by_id updated before orders_by_status. There is no snapshot and no rollback. Design reads to tolerate this brief skew.
This batch spans three different partition keys (order_id, customer_id, status), so it is a multi-partition batch — heavier on the coordinator than a single-partition batch. It is justified here because the atomicity matters and the fan-out is only three; do not batch dozens of partitions casually.
If you did not need the atomicity guarantee, plain idempotent application fan-out (three separate inserts) is lighter and equally correct under retries — reserve logged batches for when partial visibility is genuinely unacceptable.

Output.

Guarantee	Logged BATCH	Plain fan-out
all statements eventually apply	yes	only with retries
isolation (no partial reads)	no	no
rollback	no	no
coordinator cost	higher (multi-partition)	lower

Rule of thumb. Use a logged BATCH only when several denormalized tables must not diverge even under coordinator failure, and keep the number of partitions small. For everything else, idempotent application fan-out is cheaper and just as safe.

Worked example — a materialized view for an auto-maintained second key

Detailed explanation. When a denormalized table is a strict re-keying of a base table (same rows, different partition key), a materialized view can maintain it automatically. The engine intercepts base-table writes and updates the view. It removes fan-out code but adds write amplification and repair complexity, so it is a convenience with caveats. Build a view that re-keys orders by status.

Base. orders_by_id.
View. orders_by_status_mv — same data, keyed by status.
Trade-off. automatic maintenance vs write amplification and operational caveats.

Question. Create a materialized view that lets you read orders by status from a base table keyed by id, and state when to prefer it over manual fan-out.

Input.

Object	Keyed by	Maintained by
orders_by_id (base)	order_id	application
orders_by_status_mv (view)	status, order_id	engine

Code.

-- Base table
CREATE TABLE orders_by_id (
    order_id    uuid PRIMARY KEY,
    customer_id uuid,
    status      text,
    total_cents bigint,
    order_ts    timestamp
);

-- Materialized view re-keys by status; engine keeps it in sync
CREATE MATERIALIZED VIEW orders_by_status_mv AS
    SELECT status, order_id, customer_id, total_cents, order_ts
    FROM   orders_by_id
    WHERE  status IS NOT NULL AND order_id IS NOT NULL
    PRIMARY KEY ((status), order_id);

-- Read by status without any application fan-out
SELECT order_id, customer_id, total_cents
FROM   orders_by_status_mv
WHERE  status = 'pending';

Step-by-step explanation.

The view's PRIMARY KEY ((status), order_id) re-keys the base rows by status. Every base-table column used in the view key must be NOT NULL in the WHERE, because a null key column cannot be placed in a partition.
When the application writes orders_by_id, the engine automatically applies the corresponding change to orders_by_status_mv — no fan-out code in the service. A status change moves the row from one view partition to another for you.
The cost is write amplification: each base write triggers a view write, and the engine must read the old value to delete the stale view row (a read-before-write on updates). At high write rates this is a measurable tax.
Views also complicate repair and can drift under certain failure modes, which is why many senior teams restrict them to non-critical read paths and hand-roll fan-out for the hot ones.
Prefer a materialized view when the second table is a pure re-key of one base table and the write rate is moderate; prefer manual idempotent fan-out when you need multiple derived tables, high write throughput, or full operational control.

Output.

Aspect	Materialized view	Manual fan-out
maintenance code	none	explicit writes
write amplification	engine-driven (incl. read-before-write)	one write per table
operational risk	repair/drift caveats	app owns correctness
best for	single re-key, moderate writes	many tables, hot paths

Rule of thumb. Reach for a materialized view when a table is exactly one re-keying of a base table and the write volume is modest; hand-roll idempotent fan-out when you have several derived tables or a hot write path where you need explicit control over amplification.

Data engineering interview question on query-first denormalization

A senior interviewer might ask: "Model an e-commerce order store for Cassandra. Product needs (a) order detail by order id, (b) a customer's orders newest-first, and (c) an operations queue of orders by status newest-first. There are no other reads. Give the tables, the single write path for placing an order, and explain how you keep the three copies consistent when a status changes."

Solution Using three query tables with idempotent fan-out and status re-keying

CREATE TABLE orders_by_id (
    order_id uuid PRIMARY KEY,
    customer_id uuid, status text, total_cents bigint, order_ts timestamp);

CREATE TABLE orders_by_customer (
    customer_id uuid, order_ts timestamp, order_id uuid,
    status text, total_cents bigint,
    PRIMARY KEY ((customer_id), order_ts, order_id))
    WITH CLUSTERING ORDER BY (order_ts DESC, order_id DESC);

CREATE TABLE orders_by_status (
    status text, order_ts timestamp, order_id uuid,
    customer_id uuid, total_cents bigint,
    PRIMARY KEY ((status), order_ts, order_id))
    WITH CLUSTERING ORDER BY (order_ts DESC, order_id DESC);

def place_order(session, order_id, customer_id, total_cents, order_ts):
    # Idempotent fan-out: three fixed-key inserts, safe to retry
    session.execute("INSERT INTO orders_by_id (order_id,customer_id,status,total_cents,order_ts) VALUES (%s,%s,'pending',%s,%s)",
                    (order_id, customer_id, total_cents, order_ts))
    session.execute("INSERT INTO orders_by_customer (customer_id,order_ts,order_id,status,total_cents) VALUES (%s,%s,%s,'pending',%s)",
                    (customer_id, order_ts, order_id, total_cents))
    session.execute("INSERT INTO orders_by_status (status,order_ts,order_id,customer_id,total_cents) VALUES ('pending',%s,%s,%s,%s)",
                    (order_ts, order_id, customer_id, total_cents))

def change_status(session, order_id, order_ts, customer_id, new_status):
    old = session.execute("SELECT status,total_cents FROM orders_by_id WHERE order_id=%s",(order_id,)).one()
    session.execute("UPDATE orders_by_id SET status=%s WHERE order_id=%s",(new_status, order_id))
    session.execute("UPDATE orders_by_customer SET status=%s WHERE customer_id=%s AND order_ts=%s AND order_id=%s",
                    (new_status, customer_id, order_ts, order_id))
    # status is the PARTITION KEY of orders_by_status → delete old, insert new
    session.execute("DELETE FROM orders_by_status WHERE status=%s AND order_ts=%s AND order_id=%s",
                    (old.status, order_ts, order_id))
    session.execute("INSERT INTO orders_by_status (status,order_ts,order_id,customer_id,total_cents) VALUES (%s,%s,%s,%s,%s)",
                    (new_status, order_ts, order_id, customer_id, old.total_cents))

Step-by-step trace.

Step	Operation	Effect
1	place_order	3 idempotent inserts, one per query table
2	read order detail	orders_by_id WHERE order_id — 1 partition
3	read customer orders	orders_by_customer WHERE customer_id — sorted head
4	read ops queue	orders_by_status WHERE status='pending' — sorted head
5	change_status pending→shipped	UPDATE by_id + by_customer (status is a plain col)
6	orders_by_status re-key	DELETE ('pending',...) + INSERT ('shipped',...)

Placing an order fans out to three tables; a status change updates the two tables where status is a normal column and re-keys the one table where status is the partition key (delete old partition row, insert new). Every read names its partition and takes a sorted slice.

Output:

Read	Table	Partition named	Ordered
order detail	orders_by_id	order_id	n/a
customer orders	orders_by_customer	customer_id	order_ts DESC
ops queue	orders_by_status	status	order_ts DESC

Why this works — concept by concept:

One table per query — three reads produced exactly three tables, each keyed so its read is a single-partition sorted access. No read joins, filters, or scans; the modeling absorbed all the complexity.
Idempotent fan-out — place_order uses fixed-key inserts, so a retry after partial failure overwrites identical data and converges. Idempotency is what makes fan-out safe without transactions.
Partition-key immutability — because status is the partition key of orders_by_status, a status change is delete-plus-insert there, while the other two tables (where status is a value) take a plain UPDATE. Knowing which is which is the crux of the answer.
Consistency by convergence — with no cross-partition transactions, correctness comes from idempotent writes plus a periodic reconciliation job that re-derives the query tables from orders_by_id, the source of truth. Brief inter-table skew is tolerated by design.
Cost — three writes per order and four writes per status change, each read O(1) routing + O(page) scan. Storage triples the order data — cheap — in exchange for three flat-latency reads that a normalized model could not serve at all without cluster-wide scans.

Modeling
Topic — dimensional-modeling
Denormalization and query-first modeling problems

Practice →

Database Topic — database Multi-table write-path and consistency problems

Practice →

4. Pitfalls — tombstones, hot and large partitions

The three failure modes that turn a well-shaped schema into a 3 AM incident — and every one is a modeling decision, not a config knob

The mental model in one line: the three ways a wide-column table fails in production are tombstones (deletes and TTL expirations that pile up as invisible markers the read path must still scan), hot partitions (a partition key whose access is so skewed that one node absorbs a disproportionate share of traffic while the rest of the ring idles), and large partitions (a partition key whose values grow past the ~100 MB / ~100k-cell budget until compaction stalls and reads slow) — and all three are consequences of the schema you chose, so the fix is almost always a modeling change (bucket the key, spread the key, avoid the delete pattern), not a tuning parameter. Interviewers probe these because they separate people who have only read the docs from people who have been paged.

Tombstones — deletes are writes, and they haunt the read path.

What a delete really does. Because SSTables are immutable, a DELETE (or a TTL expiry, or writing null) does not remove data — it writes a tombstone, a marker that shadows older values. The real data is purged only when compaction merges the SSTables and the tombstone is older than gc_grace_seconds (default 10 days, needed for anti-entropy repair).
Why they hurt reads. A read must scan and merge all tombstones in the requested range to know what is actually deleted. A partition with tens of thousands of tombstones makes a read do enormous work to return few live rows — and past a threshold the coordinator aborts with TombstoneOverwhelmingException.
The classic anti-pattern. Using a partition as a queue: insert rows, process them, delete them. The deleted head of the partition becomes a wall of tombstones every read must scan before reaching live rows. Never model a queue this way.
Hygiene. Prefer TTL with a compaction strategy that drops whole expired SSTables (TWCS); avoid deleting individual collection elements; design so ranges are read forward past live data, not through graveyards; tune gc_grace_seconds only with repair cadence in mind.

Hot partitions — high cardinality is not enough if access is skewed.

What it is. One partition receives a hugely disproportionate share of reads and/or writes, so its owning replicas saturate while the rest of the cluster is idle. The whole cluster's throughput collapses to what one node can do.
How it happens. A celebrity user in a by_user table; a global or default sentinel value used as a partition key; a monotonically increasing key (like a raw timestamp bucket that is "now") that funnels all current writes to one partition.
Detection. Per-node/per-partition metrics: one replica set with far higher read/write rate, higher local latency, or larger partition than peers. Cassandra's nodetool toppartitions and ScyllaDB's per-shard metrics surface the offender.
The fix. Increase effective cardinality by sharding the hot key — append a small bucket component ((user_id, shard) with shard in 0..N) and fan reads across the N shards, or split write and read paths so the monotonic "now" is spread across buckets.

Large partitions — the size budget is real.

The budget. Aim to keep partitions under ~100 MB and ~100k cells (rows × columns). Beyond that, compaction slows, memory pressure rises, repair streams huge partitions, and reads that touch the partition degrade.
How it happens. A partition key that accumulates unbounded rows over time (per-sensor forever, per-user forever) with no bucketing.
The fix. Bucket the partition key by time or by a modulo shard so each partition is bounded — the same technique from section 2, applied as a corrective.
Detection. nodetool tablehistograms / tablestats report max partition size and cell count; alert when the max approaches the budget, well before it becomes an incident.

Worked example — diagnosing a tombstone-overwhelmed read

Detailed explanation. A team modeled a task queue as a Cassandra partition: enqueue inserts a row, a worker processes it and DELETEs it. After a week, reads of the "next tasks" start timing out with TombstoneOverwhelmingException. The partition is mostly graves. Walk through the diagnosis and the re-model.

Symptom. SELECT ... LIMIT 10 times out; logs show "Scanned over 100000 tombstones."
Root cause. the delete-after-process pattern leaves a tombstone per processed task at the head of the partition; each read scans them all to reach live tasks.
Fix. stop deleting; use TTL + TWCS, or re-model so processed tasks live in a different partition (time-bucketed) that ages out whole SSTables.

Question. Diagnose why the queue-partition read times out and re-model it so reads never scan through tombstones.

Input.

Observation	Value
pattern	insert, process, DELETE per task
tombstones scanned per read	> 100,000
gc_grace_seconds	864000 (10 days)
error	TombstoneOverwhelmingException

Code.

-- ANTI-PATTERN: queue in one partition, delete after processing
-- CREATE TABLE task_queue (bucket text, task_id timeuuid, payload text,
--   PRIMARY KEY ((bucket), task_id));
-- DELETE FROM task_queue WHERE bucket='q' AND task_id=...;   -- leaves a grave

-- FIX: time-bucketed partitions + TTL + TWCS; never DELETE, let it expire
CREATE TABLE tasks_by_minute (
    minute_bucket text,        -- 'YYYY-MM-DD HH:MM'
    task_id       timeuuid,
    payload       text,
    done          boolean,
    PRIMARY KEY ((minute_bucket), task_id)
) WITH CLUSTERING ORDER BY (task_id ASC)
  AND default_time_to_live = 86400          -- rows self-expire after 1 day
  AND compaction = {'class':'TimeWindowCompactionStrategy',
                    'compaction_window_unit':'HOURS',
                    'compaction_window_size':1};

-- Mark done with an UPDATE (a normal write), not a DELETE
UPDATE tasks_by_minute SET done = true
WHERE  minute_bucket='2026-09-05 10:04' AND task_id = 5c...;

Step-by-step explanation.

The anti-pattern keeps one partition and deletes each processed task. Every delete writes a tombstone that lives for gc_grace_seconds; the "next tasks" read scans all of them at the head before reaching live rows, eventually tripping the tombstone limit.
The fix time-buckets the partition by minute, so old buckets naturally stop being read as time moves forward — reads target the current bucket, not a partition full of processed history.
default_time_to_live = 86400 makes rows self-expire after a day. Crucially, combined with TimeWindowCompactionStrategy, an entire SSTable whose rows have all expired is dropped wholesale — no per-row tombstone scanning on the read path.
Marking a task done is an UPDATE (a normal cell write), not a DELETE, so it creates no tombstone. Processed tasks simply age out with their bucket.
TWCS groups data by time window so each SSTable covers a contiguous time range; expired windows are dropped as units. This is the correct compaction strategy for any TTL-driven time-series or queue-like table.

Output.

Design	Tombstones on read path	Read outcome
one partition + DELETE	grows unbounded	TombstoneOverwhelmingException
time-bucket + TTL + TWCS	~0 (whole SSTables dropped)	fast, bounded

Rule of thumb. Never model a queue or churny dataset with delete-after-process in a single partition. Time-bucket the partition, mark state with updates, and let TTL + TWCS drop whole expired SSTables so reads never wade through tombstones.

Worked example — spreading a hot partition by sharding the key

Detailed explanation. A likes_by_post table keys by post_id. A viral post gets millions of likes and reads per second, all hitting one partition on one replica set — a hot partition that caps throughput at one node's capacity. The fix adds a shard component to the partition key and scatters writes across N sub-partitions, then fans reads across them. Walk through it.

Symptom. one replica set at 100% CPU while the rest of the ring idles; latency spikes on that post.
Root cause. all traffic for one post_id lands on one partition → one owning replica set.
Fix. partition key (post_id, shard) with shard = hash(user_id) % N; reads fan out over 0..N-1 and aggregate.

Question. Re-model likes_by_post so a viral post's traffic spreads across N nodes, and show the write and the fan-out read.

Input.

Parameter	Value
hot key	post_id of a viral post
shard count N	16
shard function	hash(user_id) % 16
read	count / list likes for a post

Code.

-- Sharded partition key spreads one post across N partitions
CREATE TABLE likes_by_post (
    post_id   uuid,
    shard     int,             -- 0..N-1
    user_id   uuid,
    liked_ts  timestamp,
    PRIMARY KEY ((post_id, shard), user_id)
);

N = 16

def add_like(session, post_id, user_id, liked_ts):
    shard = hash(user_id) % N          # deterministic per user
    session.execute("""
        INSERT INTO likes_by_post (post_id, shard, user_id, liked_ts)
        VALUES (%s, %s, %s, %s)
    """, (post_id, shard, user_id, liked_ts))

def count_likes(session, post_id):
    stmt = session.prepare(
        "SELECT COUNT(*) FROM likes_by_post WHERE post_id=? AND shard=?")
    total = 0
    for shard in range(N):             # bounded fan-out over N shards
        total += session.execute(stmt, (post_id, shard)).one()[0]
    return total

Step-by-step explanation.

The partition key becomes (post_id, shard), so a single post now spans N distinct partitions that hash to N different token ranges — and therefore N different replica sets. One viral post's load is spread N-fold across the ring.
shard = hash(user_id) % N is deterministic per user, so a user's like always lands in the same shard (idempotent writes, no double-counting) while the population of users spreads evenly across shards.
Reads fan out over all N shards and aggregate. The fan-out is bounded and known (exactly N), the opposite of an unbounded scan. For counts, each shard sub-count is cheap; for listing, merge the N sorted sub-results.
N trades write/read spread against fan-out cost: larger N spreads a hotter key further but makes every read touch more partitions. Choose N to match the hottest realistic key, not the average.
This is the same cardinality lever as time-bucketing, applied to access skew rather than growth: both increase the effective number of partitions so no single one dominates.

Output.

Design	Partitions per post	Node load	Read cost
((post_id), user_id)	1	one replica set saturates	1 partition
((post_id, shard), user_id)	N=16	spread across ring	N-shard fan-out

Rule of thumb. When a single partition key value is a traffic magnet, add a shard component (% N) to the partition key to multiply its effective cardinality, and fan reads across the N shards. Size N to the hottest key you expect, not the median.

Worked example — bounding a large partition before it stalls compaction

Detailed explanation. A messages_by_room table keys by room_id. A busy support room accumulates years of messages in one partition, which grows past 100 MB. Compaction of that partition slows, repair streams it whole, and reads touching it get sluggish. The fix time-buckets the room partition. Walk through detecting and correcting it.

Detection. nodetool tablehistograms shows max partition approaching/exceeding ~100 MB.
Root cause. unbounded rows per room_id over time.
Fix. partition key (room_id, month_bucket) to bound each partition to a month of the room's messages.

Question. Detect the large partition and re-model messages_by_room so no partition exceeds the size budget while in-room reads stay single-partition.

Input.

Parameter	Value
current max partition	~180 MB (over budget)
growth	unbounded per room over time
bucket	month
target	< 100 MB per partition

Code.

-- Detect (shell): nodetool tablehistograms keyspace messages_by_room
--   → "Partition Size" max column shows ~180 MB → over budget

-- FIX: bound each partition to one month of a room
CREATE TABLE messages_by_room_month (
    room_id       uuid,
    month_bucket  text,        -- 'YYYY-MM'
    message_ts    timestamp,
    message_id    timeuuid,
    sender_id     uuid,
    body          text,
    PRIMARY KEY ((room_id, month_bucket), message_ts, message_id)
) WITH CLUSTERING ORDER BY (message_ts ASC, message_id ASC);

-- In-room read for the current month: one bounded partition
SELECT message_ts, sender_id, body
FROM   messages_by_room_month
WHERE  room_id = 42ab... AND month_bucket = '2026-09'
  AND  message_ts >= '2026-09-01 00:00:00';

Step-by-step explanation.

nodetool tablehistograms (or ScyllaDB's equivalent per-table metrics) reports the max partition size; when it approaches ~100 MB you re-model before it becomes an incident, not after.
Adding month_bucket to the partition key caps each partition at one month of a room's traffic. Even a busy room now produces bounded ~monthly partitions instead of one ever-growing one.
In-room reads for the current month name both partition-key columns and range-scan message_ts — a single bounded partition, fast. Older months are separate partitions read only when the user scrolls back.
The trade-off is that "all messages in a room" now fans out over the room's active months — a bounded, query-derived set (same technique as section 2), acceptable because the common read is "recent messages," which is one bucket.
Migration is a backfill: read the old oversized partition in ranges and rewrite into month buckets, then cut reads over. Because writes are idempotent (fixed keys), the backfill can run alongside live traffic.

Output.

Design	Max partition size	Recent-month read	Full-history read
((room_id), ...)	~180 MB (over budget)	one huge partition	one huge partition
((room_id, month_bucket), ...)	< 100 MB	1 partition	bounded month fan-out

Rule of thumb. Watch max partition size with tablehistograms and re-model before it crosses ~100 MB. The cure for a growing partition is always a bucketing component on the partition key — the same lever that prevents hot partitions, applied to size instead of access.

Data engineering interview question on wide-column pitfalls

A senior interviewer might ask: "A team's events_by_type table is timing out on reads with TombstoneOverwhelmingException, and nodetool shows one partition at 300 MB while one node runs hot. Walk me through diagnosing all three problems, and give me the re-modeled schema and delete strategy that fixes tombstones, hot partitions, and large partitions at once."

Solution Using time-bucketing plus sharding plus TTL-with-TWCS

-- BEFORE (all three pitfalls): one partition per event_type, DELETEs, unbounded
-- CREATE TABLE events_by_type (event_type text, event_ts timestamp, ...,
--   PRIMARY KEY ((event_type), event_ts));

-- AFTER: bucket by time (bounds size), shard (spreads hot key),
--        TTL + TWCS (kills tombstones)
CREATE TABLE events_by_type_bucketed (
    event_type   text,
    hour_bucket  text,          -- 'YYYY-MM-DD HH' : bounds partition size
    shard        int,           -- 0..N-1          : spreads a hot event_type
    event_ts     timestamp,
    event_id     timeuuid,
    payload      text,
    PRIMARY KEY ((event_type, hour_bucket, shard), event_ts, event_id)
) WITH CLUSTERING ORDER BY (event_ts DESC, event_id DESC)
  AND default_time_to_live = 604800        -- 7-day TTL, no manual DELETEs
  AND compaction = {'class':'TimeWindowCompactionStrategy',
                    'compaction_window_unit':'HOURS',
                    'compaction_window_size':6}
  AND gc_grace_seconds = 43200;            -- lower grace: TWCS drops whole windows

N = 8

def write_event(session, event_type, event_ts, event_id, payload):
    hour_bucket = event_ts.strftime('%Y-%m-%d %H')
    shard = event_id.int % N                      # spread within the hour
    session.execute("""
        INSERT INTO events_by_type_bucketed
          (event_type, hour_bucket, shard, event_ts, event_id, payload)
        VALUES (%s,%s,%s,%s,%s,%s)
    """, (event_type, hour_bucket, shard, event_ts, event_id, payload))

def read_recent(session, event_type, hour_bucket, limit=100):
    stmt = session.prepare("""
        SELECT event_ts, payload FROM events_by_type_bucketed
        WHERE event_type=? AND hour_bucket=? AND shard=? LIMIT ?
    """)
    rows = []
    for shard in range(N):                         # bounded N-shard fan-out
        rows.extend(session.execute(stmt, (event_type, hour_bucket, shard, limit)))
    rows.sort(key=lambda r: r.event_ts, reverse=True)
    return rows[:limit]

Step-by-step trace.

Step	Mechanism	Fixes
1	hour_bucket in partition key	large partitions (bounds each to 1 hour)
2	shard % N in partition key	hot partitions (spreads a hot event_type N-fold)
3	default_time_to_live = 7d	replaces manual DELETEs (no tombstone spam)
4	TimeWindowCompactionStrategy	drops whole expired SSTables → tombstone-free reads
5	gc_grace_seconds = 43200	shorter grace safe because TWCS drops windows wholesale
6	read fan-out over N shards	bounded, sorted-merge, no ALLOW FILTERING

The single re-model attacks all three failure modes with three orthogonal levers: time-bucketing bounds partition size, sharding spreads access, and TTL-with-TWCS eliminates tombstones by expiring whole time-windowed SSTables instead of deleting rows. Reads fan out over the known N shards of the current hour and merge — bounded and fast.

Output:

Failure mode	Before	After
tombstones	DELETE spam → timeout	TTL + TWCS → whole-SSTable drop
hot partition	one node saturates	spread over N shards
large partition	300 MB single partition	< 100 MB per (type, hour, shard)
read cost	ALLOW FILTERING / timeout	N-shard bounded fan-out

Why this works — concept by concept:

Time-bucketing — hour_bucket in the partition key caps how many rows any one partition can accumulate, holding every partition under the size budget regardless of how long the workload runs.
Sharding — shard % N multiplies the effective cardinality of a hot event_type, scattering its traffic across N replica sets so no single node becomes the bottleneck.
TTL with TWCS — a table-level TTL replaces manual DELETEs entirely, and TimeWindowCompactionStrategy drops whole expired SSTables as units, so the read path never scans tombstones — the root cause of the timeout.
Lower gc_grace_seconds — safe here precisely because TWCS reclaims data by dropping windows rather than by merging away per-row tombstones, so the long grace period needed for row-level tombstone repair is less critical (still coordinate with repair cadence).
Cost — one write per event, reads fan out over N shards of the target hour and sort-merge — O(N × page). Storage is bounded by the 7-day TTL. The eliminated cost is the unbounded tombstone scan and the single-node saturation; all three failure modes become bounded, monitorable quantities.

Database
Topic — database
Tombstone, hot-partition and large-partition problems

Practice →

Data processing Topic — data-processing Time-bucketing and TTL retention problems

Practice →

5. Cassandra vs ScyllaDB and tuning

Same data model, same CQL, different engine — ScyllaDB rewrites Cassandra in C++ with a shard-per-core architecture, so the modeling is identical but the tuning and operations diverge

The mental model in one line: ScyllaDB is a drop-in-compatible reimplementation of Cassandra's data model, CQL, and wire protocol on a shared-nothing, shard-per-core C++ runtime (Seastar) that pins one thread and a slice of RAM to each CPU core and eliminates JVM garbage-collection pauses — so everything you learned about partition keys, clustering keys, denormalization, and pitfalls transfers unchanged, while the operational surface (thread/GC tuning, compaction throughput, shard-aware routing, and cost-per-throughput) shifts because the engine underneath is different. In an interview, the correct framing is "the model is the same, the engine is different"; candidates who think ScyllaDB needs a different schema have misunderstood both.

What is the same — the entire data model.

CQL and schema. Tables, primary keys, partition and clustering keys, static columns, collections, materialized views, secondary indexes, TTL, and compaction strategies are all present with the same semantics.
The modeling rules. Query-first design, one-table-per-read denormalization, partition sizing, and the tombstone/hot/large-partition pitfalls apply identically. A schema that is well-modeled for Cassandra is well-modeled for ScyllaDB.
The distribution model. A token ring, replication factor, tunable consistency, hinted handoff, and repair all carry over. ScyllaDB even speaks the Cassandra wire protocol so most drivers work unchanged.
The failure modes. Tombstones, hot partitions, and oversized partitions bite ScyllaDB too — the physics of an LSM tree and a token ring do not change with the implementation language.

What is different — the engine and its operations.

Shard-per-core (Seastar). ScyllaDB runs one shard per physical core, each owning a slice of the data and its own memory, with no locks between shards. This extracts far more throughput per node and gives more predictable latency than Cassandra's thread-pool + shared-heap model.
No JVM, no GC pauses. ScyllaDB is C++ with its own memory management, so it avoids the stop-the-world GC pauses that cause Cassandra p99 latency spikes and require careful JVM/heap tuning.
Shard-aware drivers. ScyllaDB-aware drivers route a request not just to the right node but to the right shard (core) on that node, skipping an internal hop. Using a shard-aware driver is a key ScyllaDB tuning step with no Cassandra analogue.
Self-tuning and cost. ScyllaDB auto-tunes many parameters Cassandra exposes manually (compaction throughput, memory) and typically serves the same workload on fewer nodes — the usual business case for migrating.

Tuning levers that matter for both.

Compaction strategy. SizeTieredCompactionStrategy (STCS) for write-heavy/general, LeveledCompactionStrategy (LCS) for read-heavy with frequent updates (bounds read amplification at higher write cost), TimeWindowCompactionStrategy (TWCS) for time-series/TTL data. Matching strategy to access pattern is a top-tier tuning decision.
Replication factor + consistency level. RF sets how many copies exist; the per-query consistency level (ONE, QUORUM, LOCAL_QUORUM, ALL) sets how many replicas must respond. LOCAL_QUORUM on RF=3 is the standard multi-DC production default: strong-enough consistency, DC-local latency.
Read/write path knobs. Bloom filters, key/row caches, and compression tune the read path; commitlog settings and memtable thresholds tune the write path. ScyllaDB auto-manages more of these than Cassandra.
Shard-aware routing + prepared statements. Always use prepared statements and a shard-aware (ScyllaDB) or token-aware (Cassandra) driver so requests skip coordinator hops and go straight to an owning replica/shard.

Worked example — choosing a compaction strategy per access pattern

Detailed explanation. Compaction merges SSTables; the strategy determines read amplification, write amplification, and space amplification. Picking the wrong one is a common cause of bad latency on a correctly-modeled table. Walk through matching STCS, LCS, and TWCS to three workloads.

Write-heavy, whole-row reads. STCS — low write amplification, acceptable read amplification.
Read-heavy with in-place updates. LCS — bounds SSTables touched per read, at higher write/space cost.
Time-series with TTL. TWCS — groups by time window, drops whole expired windows.

Question. Assign a compaction strategy to each of three tables and justify the choice from the access pattern.

Input.

Table	Access pattern	Strategy
audit_log (append, TTL 30d)	time-series + TTL	TWCS
user_profile (read-heavy, updates)	reads dominate, mutable	LCS
event_ingest (write-heavy)	writes dominate	STCS

Code.

-- Time-series with TTL → TWCS (drops whole expired windows)
ALTER TABLE audit_log WITH compaction = {
    'class':'TimeWindowCompactionStrategy',
    'compaction_window_unit':'DAYS','compaction_window_size':1};

-- Read-heavy with updates → LCS (bounds read amplification)
ALTER TABLE user_profile WITH compaction = {
    'class':'LeveledCompactionStrategy','sstable_size_in_mb':160};

-- Write-heavy ingest → STCS (low write amplification)
ALTER TABLE event_ingest WITH compaction = {
    'class':'SizeTieredCompactionStrategy','min_threshold':4};

Step-by-step explanation.

audit_log is append-only with a 30-day TTL, the textbook TWCS case: each SSTable holds one day's window, and once every row in a window has expired the whole SSTable is dropped — no per-row tombstone compaction, cheap reclamation.
user_profile is read-heavy with frequent updates. LCS organizes SSTables into levels so a read touches at most a few SSTables per level, bounding read amplification — the read latency win is worth LCS's higher write and space amplification.
event_ingest is write-dominated with rare reads. STCS merges similarly-sized SSTables and imposes the least write amplification, so ingest stays cheap; the higher read amplification is acceptable because reads are rare.
Choosing LCS for a write-heavy table would burn write and I/O budget on constant re-leveling; choosing STCS for a read-heavy updated table would let SSTables pile up and inflate read amplification. Strategy must match the dominant operation.
On ScyllaDB the same strategies exist and are auto-tuned more aggressively, but the choice of which strategy fits the access pattern is still the modeler's call.

Output.

Strategy	Optimizes	Costs	Fits
STCS	write amplification	read amplification	write-heavy ingest
LCS	read amplification	write + space	read-heavy, updated
TWCS	TTL reclamation	non-time queries	time-series + TTL

Rule of thumb. Match compaction to the dominant operation: STCS for write-heavy, LCS for read-heavy-with-updates, TWCS for anything time-series or TTL-driven. The wrong strategy shows up as latency on a schema that is otherwise correctly modeled.

Worked example — choosing replication factor and consistency level

Detailed explanation. Consistency in a wide-column store is tunable per query via the interplay of replication factor and consistency level. The rule R + W > RF gives read-your-writes strong consistency. Walk through the standard production choices for a single-DC and a multi-DC deployment.

RF. number of replicas per partition (3 is standard).
Write CL + Read CL. if writes and reads together cover more than RF replicas, a read always sees the latest write.
Multi-DC. LOCAL_QUORUM keeps quorum within the local datacenter for latency while replicating across DCs for durability.

Question. Choose RF and consistency levels for (a) a single-DC app needing strong consistency and (b) a multi-DC app needing low local latency, and prove the strong-consistency case with R + W > RF.

Input.

Deployment	RF	Write CL	Read CL
single-DC strong	3	QUORUM (2)	QUORUM (2)
multi-DC low-latency	3 per DC	LOCAL_QUORUM	LOCAL_QUORUM

Code.

from cassandra import ConsistencyLevel
from cassandra.query import SimpleStatement

# (a) Single-DC strong consistency: QUORUM reads + QUORUM writes on RF=3
#     W=2, R=2, RF=3 → R + W = 4 > 3 → read always sees latest write
write = SimpleStatement(
    "INSERT INTO account (id, balance) VALUES (%s, %s)",
    consistency_level=ConsistencyLevel.QUORUM)
read = SimpleStatement(
    "SELECT balance FROM account WHERE id = %s",
    consistency_level=ConsistencyLevel.QUORUM)

# (b) Multi-DC low latency: LOCAL_QUORUM stays within the caller's DC
write_local = SimpleStatement(
    "INSERT INTO account (id, balance) VALUES (%s, %s)",
    consistency_level=ConsistencyLevel.LOCAL_QUORUM)
read_local = SimpleStatement(
    "SELECT balance FROM account WHERE id = %s",
    consistency_level=ConsistencyLevel.LOCAL_QUORUM)

Step-by-step explanation.

With RF=3, QUORUM means 2 replicas. A QUORUM write reaches 2 replicas and a QUORUM read consults 2; since any two 2-subsets of 3 overlap in at least one replica, the read is guaranteed to see the latest write. That is R + W > RF → 2 + 2 > 3.
LOCAL_QUORUM computes quorum within the local datacenter's replicas, so a read/write does not wait on cross-DC network latency while still achieving quorum locally — the standard multi-DC default.
ONE (write or read) is faster but gives only eventual consistency: a read at ONE may hit a replica that has not yet received the latest write. Use it for latency-tolerant, high-volume, non-critical data.
ALL requires every replica and gives the strongest consistency but zero availability tolerance — one down replica fails the query. Almost never the right production default.
The consistency level is a per-query lever, so you can mix: LOCAL_QUORUM for critical account writes, ONE for analytics counters — on the same cluster and even the same table.

Output.

Setting	Consistency	Availability	Latency
RF3 + QUORUM/QUORUM	strong (R+W>RF)	tolerates 1 down	medium
RF3 + LOCAL_QUORUM	strong per DC	tolerates 1 local down	low (DC-local)
RF3 + ONE/ONE	eventual	highest	lowest
RF3 + ALL	strongest	none (any down fails)	highest

Rule of thumb. Default to RF=3 with LOCAL_QUORUM reads and writes for production; it satisfies R + W > RF for strong consistency while keeping latency datacenter-local. Drop to ONE only for data that tolerates staleness, and avoid ALL outside rare admin paths.

Worked example — shard-aware routing on a Cassandra→ScyllaDB migration

Detailed explanation. Migrating a correctly-modeled Cassandra keyspace to ScyllaDB is mostly operational: keep the schema, switch the engine, and adopt shard-aware drivers and self-tuned compaction. The throughput win comes from shard-per-core plus the driver routing to the exact shard. Walk through the migration checklist and the driver change.

Schema. unchanged — same CQL, same keys.
Data. stream via nodetool/SSTable load or dual-write + backfill.
Driver. switch to a shard-aware driver and always use prepared statements.
Compaction. let ScyllaDB auto-tune; keep TWCS for time-series.

Question. Give the migration checklist and show the driver-level change that unlocks shard-aware routing.

Input.

Item	Cassandra	ScyllaDB
schema / CQL	same	same
routing	token-aware	shard-aware
GC tuning	JVM heap	none (C++)
compaction throughput	manual	auto-tuned

Code.

# Shard-aware routing: prepared statements let the driver compute the
# partition token AND the target shard, skipping a coordinator hop.
from cassandra.cluster import Cluster

cluster = Cluster(["scylla-node-1", "scylla-node-2"])  # shard-aware driver
session = cluster.connect("app_keyspace")

# Prepared statement → driver knows the partition-key columns →
# routes straight to the owning replica AND the owning core (shard).
insert = session.prepare(
    "INSERT INTO readings_by_sensor_day "
    "(sensor_id, day_bucket, reading_ts, temperature) VALUES (?, ?, ?, ?)")

session.execute(insert, (sensor_id, day_bucket, reading_ts, temp))
# No ALLOW FILTERING, no unbounded scans — same schema as Cassandra,
# now landing on the exact shard that owns the token.

Step-by-step explanation.

The schema and every table definition carry over verbatim — ScyllaDB speaks the same CQL and stores the same partitioned, sorted data. No re-modeling is required, which is the whole appeal.
Prepared statements are the key: because the driver knows which bind parameters are partition-key columns, it computes the token client-side and routes the request to an owning replica (token-aware) and, on ScyllaDB, to the specific core that owns that token (shard-aware), skipping an internal cross-core hop.
Removing the JVM removes GC-pause tuning entirely; the p99 latency spikes that Cassandra operators fight with heap and GC flags simply do not occur, so that whole tuning surface disappears.
Compaction throughput and memory that Cassandra exposes as manual knobs are auto-tuned by ScyllaDB against the shard model, though the strategy choice (TWCS for time-series, etc.) remains the modeler's decision.
The net operational result is typically the same workload on fewer, better-utilized nodes with more predictable latency — the standard business case for the migration, achieved without touching the data model.

Output.

Migration step	Change	Effect
schema	none	model transfers unchanged
driver	shard-aware + prepared	request hits owning core directly
GC tuning	removed	no stop-the-world p99 spikes
node count	usually fewer	shard-per-core throughput

Rule of thumb. Treat a Cassandra→ScyllaDB migration as an engine swap, not a re-model: keep the schema, adopt a shard-aware driver with prepared statements, drop the JVM/GC tuning, and let ScyllaDB auto-tune compaction throughput while you keep owning the strategy choice.

Data engineering interview question on Cassandra vs ScyllaDB and tuning

A senior interviewer might ask: "You run a time-series telemetry workload on Cassandra with painful GC-driven p99 spikes and rising node costs. You are asked to migrate to ScyllaDB and tune it. Walk me through what stays the same, what you change, the compaction and consistency choices for time-series, and how you validate the migration without a re-model."

Solution Using an engine swap with TWCS, LOCAL_QUORUM, and shard-aware routing

-- Schema is IDENTICAL on ScyllaDB — time-series, bucketed, TTL + TWCS
CREATE TABLE telemetry_by_device_hour (
    device_id    uuid,
    hour_bucket  text,          -- 'YYYY-MM-DD HH'
    reading_ts   timestamp,
    metric       text,
    value        double,
    PRIMARY KEY ((device_id, hour_bucket), reading_ts, metric)
) WITH CLUSTERING ORDER BY (reading_ts DESC, metric ASC)
  AND default_time_to_live = 2592000        -- 30-day retention
  AND compaction = {'class':'TimeWindowCompactionStrategy',
                    'compaction_window_unit':'HOURS',
                    'compaction_window_size':1};

from cassandra.cluster import Cluster
from cassandra import ConsistencyLevel

cluster = Cluster(["scylla-1", "scylla-2", "scylla-3"])  # shard-aware
session = cluster.connect("telemetry")

write = session.prepare("""
    INSERT INTO telemetry_by_device_hour
      (device_id, hour_bucket, reading_ts, metric, value) VALUES (?,?,?,?,?)""")
write.consistency_level = ConsistencyLevel.LOCAL_QUORUM      # strong, DC-local

read = session.prepare("""
    SELECT reading_ts, value FROM telemetry_by_device_hour
    WHERE device_id=? AND hour_bucket=? AND reading_ts>=?""")
read.consistency_level = ConsistencyLevel.LOCAL_QUORUM

def validate(session_c, session_s, device_id, hour_bucket, since):
    # Dual-read validation: same query on both engines must match
    c = list(session_c.execute(read, (device_id, hour_bucket, since)))
    s = list(session_s.execute(read, (device_id, hour_bucket, since)))
    assert [(r.reading_ts, r.value) for r in c] == [(r.reading_ts, r.value) for r in s]

Step-by-step trace.

Step	Action	Rationale
1	keep schema verbatim	model is engine-agnostic; no re-model
2	TWCS, 1-hour windows	time-series + 30d TTL → whole-window drops
3	shard-aware driver + prepared	route to owning core; skip coordinator hop
4	LOCAL_QUORUM read + write	RF3 → R+W>RF strong, DC-local latency
5	dual-write, then backfill	migrate data with live traffic
6	dual-read validation	assert Cassandra and ScyllaDB agree before cutover

The migration keeps the schema, swaps STCS/GC concerns for TWCS + shard-per-core, routes with a shard-aware driver, and holds strong consistency with LOCAL_QUORUM on RF=3. Validation dual-reads the same partition from both engines and asserts equality before the read cutover, so the migration is provably lossless without any modeling change.

Output:

Dimension	Cassandra (before)	ScyllaDB (after)
p99 latency	GC-driven spikes	flat (no GC)
compaction	manual throughput	auto-tuned, TWCS
routing	token-aware	shard-aware (per core)
consistency	LOCAL_QUORUM	LOCAL_QUORUM (unchanged)
node count	baseline	typically fewer

Why this works — concept by concept:

Engine swap, not re-model — ScyllaDB implements the same CQL, keys, and distribution, so the bucketed, TTL+TWCS time-series schema transfers unchanged; the migration risk is operational, not architectural.
TWCS for time-series — one-hour compaction windows plus a 30-day TTL let whole expired SSTables drop as units, keeping reads tombstone-free — the correct strategy on both engines, and what fixes the reclamation cost.
Shard-per-core + shard-aware routing — Seastar pins a shard to each core with no GC, and the shard-aware driver plus prepared statements route each request straight to the owning core, which is where the throughput and flat-p99 win comes from.
LOCAL_QUORUM on RF=3 — preserves strong consistency (R + W > RF) at datacenter-local latency across the migration, so consistency semantics do not change under the users.
Cost — the model's write/read costs are unchanged (one write per reading; bounded per-partition sorted reads); the migration trades JVM/GC tuning and node count for C++ shard-per-core efficiency, validated by a dual-read equality check. Net: same model, lower latency variance, fewer nodes.

Data processing
Topic — data-processing
Compaction, consistency and tuning problems

Practice →

Database
Topic — database
Cassandra and ScyllaDB engine and replication problems

Practice →

Cheat sheet — wide-column modeling recipes

Query-first workflow. List every read with the values it will have and the order it needs; make one table per read whose primary key encodes that access; accept duplication; pick a write path (idempotent fan-out, logged batch, or materialized view). Never start from entities; start from queries.
Primary-key anatomy. PRIMARY KEY ((partition_cols), clustering_cols). Double parens = composite partition key (hashed together to one token). No inner parens = only the first column is the partition key. Partition key routes to a node; clustering columns sort within the partition and are constrainable left-to-right with one trailing range.
Partition-size budget. Keep partitions under ~100 MB and ~100k cells. When one partition-key value grows unbounded over time, add a time bucket ((id, day_bucket)); when access is skewed, add a shard ((id, shard) with % N). Both increase effective cardinality.
Clustering order = on-disk order. WITH CLUSTERING ORDER BY (col DESC) bakes the sort in; the dominant read becomes a zero-cost head slice. Make the clustering key a total order (add a tiebreaker like timeuuid) so cursor pagination never skips or repeats. Never use OFFSET — it does not exist.
Denormalization sync. Idempotent application fan-out (fixed keys, safe retries) for most paths; logged BATCH when several tables must not diverge under coordinator failure (keep partition count small); materialized view for a single pure re-key at moderate write volume. Partition keys are immutable — "changing" one is delete-plus-insert.
Tombstone hygiene. Deletes/TTL/null writes create tombstones that the read path must scan until compaction purges them past gc_grace_seconds. Never model a queue as insert-process-delete in one partition. Prefer TTL + TWCS so whole expired SSTables drop; mark state with UPDATE, not DELETE.
Hot-partition fix. High cardinality is not enough — access must be even. A celebrity key, a global/default sentinel, or a monotonic "now" bucket saturates one replica set. Shard the key (% N) and fan reads across the N shards; size N to the hottest key, not the median. Detect with nodetool toppartitions / per-shard metrics.
Bucketing formula. Choose bucket granularity so a full partition sits under ~100 MB: rows_per_bucket × row_size < 100 MB. Too coarse → large partitions; too fine → wide multi-bucket fan-out. Drive multi-bucket reads from the query's own range, never ALLOW FILTERING.
Compaction-strategy picker. STCS for write-heavy ingest (low write amplification); LCS for read-heavy tables with updates (bounds read amplification); TWCS for time-series/TTL (drops whole expired windows). The wrong strategy shows up as latency on a correctly-modeled table.
Consistency picker. RF=3 + LOCAL_QUORUM read/write is the production default: R + W > RF gives strong consistency at DC-local latency. ONE for staleness-tolerant high-volume data; ALL almost never (any down replica fails the query). Consistency is a per-query lever — mix as needed.
Cassandra vs ScyllaDB. Same data model, CQL, keys, distribution, and pitfalls. ScyllaDB differs in the engine: C++ shard-per-core (Seastar), no JVM/GC pauses, shard-aware drivers, auto-tuned compaction, usually fewer nodes. Migrate by swapping the engine, not re-modeling; adopt a shard-aware driver with prepared statements.
ALLOW FILTERING ban. If a read needs ALLOW FILTERING, the model is wrong — build another table or change the key. ALLOW FILTERING is an O(partition) or O(cluster) scan and is close to disqualifying as a proposed production read in an interview.

Frequently asked questions

What is wide-column data modeling in one sentence?

Wide-column data modeling is the practice of designing tables around the exact read queries an application will issue rather than around normalized entities, because a wide-column store like Cassandra or ScyllaDB has no joins and no ad-hoc filtering and can only serve reads that name a partition key and take a sorted slice of clustering keys. You enumerate the reads first, then create one denormalized table per read whose primary key makes that read a single-partition access, duplicating data across tables on write so every read stays O(1) routing plus a contiguous disk scan. The mantra is "model for the query, not the entity" — the schema is effectively a hand-written query plan frozen at design time.

Partition key vs clustering key — what's the difference?

The partition key is the part of the primary key that is hashed to a token to decide which node(s) store the row, and it is the only value that makes a read efficient — every read must supply the full partition key with equality. The clustering key sorts rows within a partition on disk and determines the order you can range-scan and paginate over — clustering columns can be constrained left-to-right with a single trailing range. In PRIMARY KEY ((sensor_id, day), reading_ts), (sensor_id, day) is a composite partition key hashed together, and reading_ts is the clustering key that orders readings inside each partition. Getting the partition key wrong causes hot or oversized partitions; getting the clustering key wrong makes the reads you need impossible.

Why does Cassandra force denormalization?

Because there are no joins and no efficient cross-partition queries, the only way to serve a read is to have a table whose primary key already matches it — so if you read the same data two ways (by id and by email, by customer and by status), you need two tables holding that data, each keyed differently. Denormalization is therefore not an optimization you reach for reluctantly; it is the fundamental modeling primitive. Writes and storage are cheap in an LSM-tree architecture, so duplicating a fact across several tables and paying a bounded write fan-out is the correct trade for flat, single-partition read latency. Consistency across the copies becomes the application's responsibility, handled with idempotent writes, logged batches where atomicity matters, and periodic reconciliation from a source-of-truth table.

What are tombstones and why do they cause read timeouts?

A tombstone is the marker Cassandra/ScyllaDB writes when you delete a row, let a TTL expire, or write a null — because SSTables are immutable, data is never removed in place; instead a tombstone shadows the old value until compaction purges it after gc_grace_seconds (default 10 days, needed for repair). Tombstones cause read timeouts because a read must scan and merge every tombstone in the requested range to determine what is actually live, so a partition with tens of thousands of tombstones makes reads do enormous work — and past a threshold the coordinator aborts with TombstoneOverwhelmingException. The classic trigger is modeling a queue as insert-process-delete in one partition, which builds a wall of tombstones at the partition head. The fix is to avoid deletes: use TTL with TimeWindowCompactionStrategy so whole expired SSTables drop, and mark state changes with UPDATE instead of DELETE.

How big can a Cassandra partition be?

Technically a partition can hold up to about two billion cells, but the practical budget is far lower: keep partitions under roughly 100 MB and 100,000 cells (rows times columns per row). Beyond that, compaction slows, memory pressure rises, repair has to stream huge partitions, and reads that touch the partition degrade. When a partition-key value would accumulate unbounded rows over time, you bound it by adding a bucketing component to the partition key — a time bucket like (device_id, day) for time-series, or a modulo shard like (post_id, shard) for a hot key. Monitor the maximum partition size with nodetool tablehistograms (or ScyllaDB's per-table metrics) and re-model before the max approaches the budget rather than after it becomes an incident.

Cassandra vs ScyllaDB — which should I pick?

They share the same data model, CQL, primary-key semantics, distribution model, and failure modes, so anything you model correctly for one is correct for the other — the choice is about the engine, not the schema. ScyllaDB is a C++ reimplementation on a shard-per-core runtime (Seastar) with no JVM garbage-collection pauses, shard-aware drivers that route requests to the exact owning core, and heavy auto-tuning, which typically delivers more predictable p99 latency and higher throughput per node, so the same workload often runs on fewer machines. Cassandra has the larger, older ecosystem and is the safe default if you already run the JVM stack and its tooling. Pick ScyllaDB when latency predictability, throughput density, or node cost are the pain points and you can adopt shard-aware drivers; pick Cassandra when ecosystem maturity and existing operational familiarity outweigh the efficiency gains — and remember migrating between them is an engine swap, not a re-model.

Practice on PipeCode

Drill the database practice library → for the wide-column, partition-key, clustering-key, tombstone, and hot-partition problems senior interviewers love.
Rehearse on the data-processing practice library → for the bucketing, TTL retention, compaction, and time-series pipeline patterns.
Sharpen your schema instincts on the dimensional-modeling practice library → for query-first denormalization and one-table-per-read design.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the wide-column modeling decisions against real graded inputs.

Lock in wide-column modeling muscle memory

Docs explain syntax. PipeCode drills explain the decision — when a partition key hot-spots, when a partition grows past the budget, when denormalization needs a logged batch, when tombstones turn a delete into a read-time landmine, and when to reach for ScyllaDB's shard-per-core engine. Pipecode.ai is Leetcode for Data Engineering — query-first practice tuned for the production trade-offs senior data engineers actually face.

Practice database problems →
Practice data-modeling problems →

DynamoDB for Data Engineers: Single-Table Design, Streams & S3 Export

Gowtham Potureddi — Tue, 08 Sep 2026 13:50:23 +0000

DynamoDB is the piece of the modern stack that most data engineers meet from the outside — as a source they have to drain into a warehouse, a CDC feed they have to consume, or an interview whiteboard they have to model on — long before they ever get to design one themselves. It is a fully managed key-value and document store that gives you single-digit-millisecond reads at any scale, but it earns that speed by refusing to be a relational database: there are no joins, no ad-hoc WHERE clauses that the engine will happily optimize, and no "just add an index later and the slow query gets fast." Every efficient access path has to be designed before the first item is written, because the shape of the partition key and sort key you choose is the shape of every query you will ever be allowed to run cheaply.

This guide is the walkthrough you wished existed the first time someone handed you a DynamoDB table and said "get this into Snowflake by Friday," or the first time an interviewer asked "model a social feed in one table and explain how you'd stream changes to analytics." It opens the box in layers: the data model (how items hash into partitions and what a read or write actually costs), single-table design driven by access patterns instead of entities, the secondary indexes that buy you new query paths, the change log that turns every item mutation into an ordered event, and finally the S3 export path that bridges an operational table into a columnar warehouse without ever scanning it. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the database practice library →, sharpen your modeling with the indexing practice library →, and rehearse the pipeline side on the data-processing practice library →.

On this page

The DynamoDB data model — keys, items, and capacity
Single-table design and access patterns
GSIs and query patterns
DynamoDB Streams — change data capture
S3 export and warehouse integration
Cheat sheet — DynamoDB for data engineers recipes
Frequently asked questions
Practice on PipeCode

1. The DynamoDB data model — keys, items, and capacity

The partition key, sort key, and capacity model are the three levers that decide every query you can ever run cheaply

The one-sentence invariant: a DynamoDB table is a distributed hash map whose primary key is either a single partition key or a composite (partition key, sort key) pair — the partition key hashes an item onto one of many physical partitions, the sort key orders items within that partition, and every read or write is billed in capacity units, so the model you get is "O(1) lookup by full key, cheap range scan within a partition, and expensive full-table scan for anything else." Nothing about that sentence is optional. If you internalize it, single-table design, GSIs, and streams all fall out as consequences; if you skip it, you will design a table that works in the demo and melts under a hot partition in production.

The primitives that matter.

Item. The row equivalent — a collection of attributes, one of which is the partition key. An item is at most 400 KB including attribute names and values. There is no fixed schema beyond the primary key; two items in the same table can have entirely different attributes.
Partition key (a.k.a. hash key). The attribute DynamoDB hashes to pick a physical partition. To read or Query efficiently you must supply the exact partition key value — there is no "partition key LIKE prefix." High cardinality and even access distribution are the whole game.
Sort key (a.k.a. range key). The optional second half of a composite primary key. Items sharing a partition key are stored sorted by sort key, which is what makes range conditions (begins_with, between, >) cheap. The partition key plus sort key together must be unique.
Capacity units. One RCU buys one strongly-consistent read of up to 4 KB/second (or two eventually-consistent reads); one WCU buys one write of up to 1 KB/second. Everything you do is priced in these units, whether you run on-demand (pay-per-request) or provisioned.

The physical partitioning story interviewers probe.

DynamoDB spreads items across many storage partitions; each partition tops out around 10 GB of data, 3,000 RCU, and 1,000 WCU. When a table grows or its throughput climbs, DynamoDB splits partitions automatically.
A hot partition is the classic failure: if 90% of traffic targets one partition key value (say status = 'ACTIVE' as a key), that single partition's throughput ceiling becomes your table's ceiling, and you get throttling while the table is 99% idle.
Adaptive capacity and burst capacity soften short spikes, but they cannot rescue a fundamentally skewed key design. The fix is always at the key level — add entropy, shard the hot key, or choose a higher-cardinality attribute.

The 2026 reality — on-demand is the default.

On-demand capacity is the modern default: you pay per read/write request and DynamoDB handles scaling instantly. It is the right call for spiky, unpredictable, or new workloads, and it removes the "did we provision enough?" operational load.
Provisioned capacity (with auto-scaling) is cheaper at steady, predictable, high volume — you reserve RCU/WCU and optionally buy reserved capacity. Data engineers usually meet provisioned tables on mature, high-throughput services where the cost math has been done.
Reads have two consistency modes. Eventually-consistent reads are the default and cost half an RCU; strongly-consistent reads cost a full RCU and are not available on GSIs. Knowing which you need per access pattern is a real cost lever.

What interviewers listen for.

Do you say "I pick the partition key for even distribution and high cardinality" before talking about attributes? — senior signal.
Do you name the 400 KB item limit and what to do about large blobs (offload to S3, store a pointer)? — required answer.
Do you distinguish Query (by partition key) from Scan (whole table) and treat Scan as a last resort? — required answer.
Do you reason about RCU/WCU cost rather than assuming reads are free? — senior signal.

Worked example — choosing a partition key that spreads load

Detailed explanation. The single most consequential decision in a DynamoDB table is the partition key. A good one distributes both storage and throughput evenly across partitions; a bad one funnels traffic onto one partition and throttles. Walk through picking a partition key for an IoT telemetry table that ingests readings from millions of devices.

The access pattern. "Get the last N readings for a given device," and "write a reading as it arrives."
The tempting-but-wrong key. sensor_type (e.g. TEMPERATURE, HUMIDITY) — only a handful of distinct values, so a handful of partitions absorb all writes. Instant hot partition.
The right key. device_id — millions of distinct values, so writes and reads spread across the whole partition space. Use the reading timestamp as the sort key so "last N readings" is a cheap descending range scan.

Question. Design the primary key for the telemetry table so that writes distribute evenly and "last N readings for a device" is a single efficient query.

Input.

Candidate partition key	Distinct values	Distribution	Verdict
`sensor_type`	~5	catastrophically skewed	reject
`region`	~20	skewed	reject
`device_id`	millions	even	accept
`device_id#yyyy-mm`	millions × months	even + time-bounded	accept (write-sharded)

Code.

import boto3

ddb = boto3.client("dynamodb")

ddb.create_table(
    TableName="telemetry",
    KeySchema=[
        {"AttributeName": "device_id", "KeyType": "HASH"},   # partition key
        {"AttributeName": "reading_ts", "KeyType": "RANGE"}, # sort key
    ],
    AttributeDefinitions=[
        {"AttributeName": "device_id", "AttributeType": "S"},
        {"AttributeName": "reading_ts", "AttributeType": "S"},  # ISO-8601 string sorts lexically = chronologically
    ],
    BillingMode="PAY_PER_REQUEST",   # on-demand
)

# Last 10 readings for one device — a single Query, newest first
resp = ddb.query(
    TableName="telemetry",
    KeyConditionExpression="device_id = :d",
    ExpressionAttributeValues={":d": {"S": "device-8f21"}},
    ScanIndexForward=False,   # descending sort key = newest first
    Limit=10,
)

Step-by-step explanation.

device_id as the partition key gives millions of distinct hash targets, so writes spread across the entire partition space — no single device can hot-spot the table.
reading_ts as an ISO-8601 string sort key is stored in lexical order, which for zero-padded ISO timestamps is identical to chronological order. That makes "readings between two times" a native range condition.
ScanIndexForward=False reads the sort key in descending order, so Limit=10 returns the ten newest readings without scanning the whole item collection.
BillingMode=PAY_PER_REQUEST is on-demand — appropriate for telemetry whose volume is spiky and hard to predict.
If a single device ever became hot (e.g. a firmware bug hammering one sensor), the write-sharded key device_id#yyyy-mm splits its collection across months; you fan out reads across the small known set of month-shards.

Output.

Metric	`sensor_type` key	`device_id` key
Distinct partitions used	~5	millions
Write hot-spot risk	severe	negligible
"Last N readings" query	Scan + filter	single Query
Throughput ceiling	one partition's	whole table's

Rule of thumb. Pick the partition key for cardinality and even access first; pick the sort key so your most common "give me a range" access pattern is a native Query. If any single key value can attract a large fraction of traffic, shard it with a suffix before you ship.

Worked example — sizing reads and writes in capacity units

Detailed explanation. Data engineers routinely misjudge DynamoDB cost because they assume reads are free. They are not — every read and write is metered. Walk through computing the capacity cost of a realistic access pattern so you can defend a cost estimate in a design review.

RCU rule. 1 RCU = one strongly-consistent read of ≤ 4 KB per second; eventually-consistent reads are half price (0.5 RCU per 4 KB).
WCU rule. 1 WCU = one write of ≤ 1 KB per second. A 3.5 KB item costs 4 WCU to write (round up per KB).
Rounding. Both RCU and WCU round the item size up to the next unit boundary (4 KB for reads, 1 KB for writes).

Question. A service reads a 6 KB item 500 times/second (eventual consistency is fine) and writes a 2.5 KB item 100 times/second. Compute the RCU and WCU demand.

Input.

Operation	Item size	Rate/sec	Consistency
Read	6 KB	500	eventual
Write	2.5 KB	100	standard

Code.

import math

def rcu(item_kb, reads_per_sec, strongly_consistent):
    units_per_read = math.ceil(item_kb / 4)          # 4 KB per RCU
    if not strongly_consistent:
        units_per_read = units_per_read / 2           # eventual = half price
    return units_per_read * reads_per_sec

def wcu(item_kb, writes_per_sec):
    units_per_write = math.ceil(item_kb / 1)          # 1 KB per WCU
    return units_per_write * writes_per_sec

read_demand  = rcu(6, 500, strongly_consistent=False)  # 6KB -> 2 units -> /2 -> 1 * 500
write_demand = wcu(2.5, 100)                           # 2.5KB -> 3 units * 100

print(read_demand, "RCU")   # 500.0 RCU
print(write_demand, "WCU")  # 300 WCU

Step-by-step explanation.

A 6 KB item rounds up to two 4 KB read blocks → 2 units for a strongly-consistent read. Eventual consistency halves that to 1 unit per read.
At 500 reads/second that is 500 RCU of demand — a real, billable number, not "free."
A 2.5 KB item rounds up to three 1 KB write blocks → 3 WCU per write.
At 100 writes/second that is 300 WCU.
On-demand you simply pay for those request-units; provisioned you must reserve at least this many (plus headroom) or auto-scale, and under-provisioning shows up as ProvisionedThroughputExceededException throttling.

Output.

Access pattern	Per-op units	Rate	Total demand
6 KB eventual read	1 RCU	500/s	500 RCU
2.5 KB write	3 WCU	100/s	300 WCU

Rule of thumb. Estimate capacity as ceil(size / block) × rate, halve reads when eventual consistency is acceptable, and remember that fat items multiply cost linearly — a 40 KB item costs 10× the read units of a 4 KB one, which is a strong argument for keeping items lean.

Worked example — the 400 KB item limit and large-attribute offload

Detailed explanation. DynamoDB caps an item at 400 KB. When a natural entity carries a large blob — a document body, an image, a big JSON payload — you must not stuff it into the item. The pattern is to store the blob in S3 and keep only a pointer plus metadata in DynamoDB. Walk through the offload.

The symptom. ValidationException: Item size has exceeded the maximum allowed size on write.
The fix. Put the large attribute in S3; store the S3 key, size, and content-type in DynamoDB.
The bonus. Items stay small → reads are cheaper (fewer RCU) and item collections stay under the 10 GB partition ceiling longer.

Question. Redesign a documents table whose body attribute can exceed 400 KB so writes never fail and reads stay cheap.

Input.

Attribute	Before	After
`doc_id` (PK)	string	string
`title`	string	string
`body`	up to 5 MB text	removed
`s3_key`	—	`s3://docs/<doc_id>.txt`
`byte_size`	—	number

Code.

import boto3, json

s3  = boto3.client("s3")
ddb = boto3.client("dynamodb")

def put_document(doc_id, title, body: str):
    body_bytes = body.encode("utf-8")
    key = f"{doc_id}.txt"

    # 1. Large payload -> S3
    s3.put_object(Bucket="docs", Key=key, Body=body_bytes,
                  ContentType="text/plain")

    # 2. Small metadata item -> DynamoDB (well under 400 KB)
    ddb.put_item(
        TableName="documents",
        Item={
            "doc_id":    {"S": doc_id},
            "title":     {"S": title},
            "s3_key":    {"S": f"s3://docs/{key}"},
            "byte_size": {"N": str(len(body_bytes))},
        },
    )

Step-by-step explanation.

The large body moves to S3 under a deterministic key derived from doc_id, so the item and its blob stay linked without a separate mapping.
The DynamoDB item now holds only small metadata — a few hundred bytes — so it never approaches the 400 KB ceiling and costs a single RCU to read.
Read paths that only need title never pay to transfer the megabyte body; read paths that need the body do one extra S3 GetObject.
Keeping items small also keeps item collections well under the 10 GB per-partition limit, delaying forced splits.
The same pattern applies to anything bulky — embeddings, base64 images, denormalized arrays — offload to S3 (or a separate table) and store a reference.

Output.

Concern	Before (inline body)	After (S3 offload)
Write success	fails > 400 KB	always succeeds
Read cost (metadata)	many RCU	1 RCU
Partition pressure	high	low
Blob storage cost	DynamoDB rate	S3 rate (cheaper)

Rule of thumb. Keep DynamoDB items lean and boring. Anything that can grow past a few KB — especially unbounded text or binary — belongs in S3 with a pointer in the item. Small items are cheaper to read, faster to stream, and friendlier to every downstream pipeline.

Data engineering interview question on the DynamoDB data model

A senior interviewer might ask: "You are given a new DynamoDB table that will store user sessions — hundreds of millions of rows, written constantly, read by user_id, and you must also fetch a single session by its session_id. Design the primary key, choose a capacity mode, and explain how you avoid a hot partition and how you keep read costs bounded."

Solution Using a composite primary key with on-demand capacity

import boto3

ddb = boto3.client("dynamodb")

# Primary key: partition by user_id (high cardinality), sort by session start time.
# Uniqueness of (user_id, started_at) is guaranteed by including session_id in the SK.
ddb.create_table(
    TableName="sessions",
    KeySchema=[
        {"AttributeName": "user_id", "KeyType": "HASH"},
        {"AttributeName": "sk",      "KeyType": "RANGE"},  # "2026-09-05T10:04:11Z#sess-abc123"
    ],
    AttributeDefinitions=[
        {"AttributeName": "user_id",    "AttributeType": "S"},
        {"AttributeName": "sk",         "AttributeType": "S"},
        {"AttributeName": "session_id", "AttributeType": "S"},
    ],
    BillingMode="PAY_PER_REQUEST",
    GlobalSecondaryIndexes=[{
        "IndexName": "gsi_session_id",
        "KeySchema": [{"AttributeName": "session_id", "KeyType": "HASH"}],
        "Projection": {"ProjectionType": "ALL"},
    }],
)

# Access pattern A — recent sessions for a user (cheap range scan, newest first)
ddb.query(
    TableName="sessions",
    KeyConditionExpression="user_id = :u",
    ExpressionAttributeValues={":u": {"S": "user-42"}},
    ScanIndexForward=False,
    Limit=20,
)

# Access pattern B — one session by its global id (via the GSI)
ddb.query(
    TableName="sessions",
    IndexName="gsi_session_id",
    KeyConditionExpression="session_id = :s",
    ExpressionAttributeValues={":s": {"S": "sess-abc123"}},
)

Step-by-step trace.

Input	Value	Reasoning
Partition key	`user_id`	hundreds of millions of users = even spread, no hot partition
Sort key	`started_at#session_id`	orders sessions by time; suffix guarantees uniqueness
Capacity mode	on-demand	constant, spiky writes; no provisioning guesswork
Access A	`Query user_id = :u`	one round trip for a user's recent sessions
Access B	GSI on `session_id`	direct lookup by the global id without a Scan
Read consistency	eventual on the GSI	GSIs are eventual-only; halves RCU cost

After deployment, writes fan out across the full user_id space so no partition throttles; "recent sessions for a user" is one descending Query bounded by Limit; and "one session by id" is a single-item GSI lookup instead of a full-table Scan. Read cost stays O(items returned), not O(table).

Output:

Access pattern	Mechanism	Cost
Recent sessions for a user	Query on base table PK	O(returned) RCU
Session by global id	Query on `gsi_session_id`	1 GSI read
Write a session	PutItem	1+ WCU per KB
Hot-partition risk	`user_id` spread	negligible

Why this works — concept by concept:

Partition key for spread — user_id has enormous cardinality, so DynamoDB's hash distributes writes and reads evenly across partitions; no single value can monopolize a partition's throughput.
Sort key for ordering — encoding started_at first in the sort key makes "newest sessions" a native descending range scan, and appending session_id guarantees the composite key is unique even when two sessions start in the same millisecond.
GSI for the second access pattern — a lookup by session_id is impossible on the base table (wrong partition key), so a GSI keyed on session_id provides that path without ever scanning the table.
On-demand capacity — constant spiky session traffic is exactly the workload on-demand is built for; you avoid both throttling and over-provisioning, and cost tracks actual usage.
Cost — every access is O(items touched), never O(table): a bounded Query for the common path, a single-item GSI read for the by-id path, and one write per session. The eliminated cost is the full-table Scan a naive "find session by id" design would have forced.

Database
Topic — database
Database modeling and key-design problems

Practice →

Indexing Topic — indexing Indexing and access-path problems

Practice →

2. Single-table design and access patterns

Model your access patterns first, then overload one table's partition key and sort key to serve them all

The mental model in one line: single-table design is the deliberate practice of storing many entity types — users, orders, line items — in a single DynamoDB table, using generic key attributes whose values carry entity-type prefixes (USER#123, ORDER#456), so that related items share a partition key and can be fetched together in one Query; it starts not from an entity-relationship diagram but from an exhaustive list of the exact queries the application will run. Data engineers trained on normalized relational schemas find this alien at first — you are effectively pre-joining data at write time — but it is what lets DynamoDB answer complex access patterns with single-digit-millisecond latency and no joins.

The axes that matter.

Access-pattern-first, not entity-first. In DynamoDB you cannot bolt on an efficient query later. You enumerate every read the app needs — "get user profile," "list a user's orders," "get an order with its line items" — before the schema exists, because the keys must be shaped to serve them.
Generic key names. The partition key is literally named PK and the sort key SK (or pk/sk). Their values are typed strings like USER#123 and ORDER#456, which lets one table hold heterogeneous items.
Item collections. All items sharing a PK form an item collection, stored physically together and sorted by SK. A single Query on that PK returns the whole collection — the user profile and their orders and their addresses — in one round trip.
Denormalization is the point. You duplicate data across items to avoid joins. Consistency across duplicates is maintained at write time (often via transactions or streams), trading write complexity for read speed.

The access-pattern-first workflow.

Step 1 — list access patterns. Write them as a table: pattern name, key condition, index. "List orders for a user → PK = USER#<id> AND begins_with(SK, 'ORDER#')."
Step 2 — design keys to satisfy each. Every access pattern must map to either a Query on the base table or a Query on a GSI. If a pattern needs a Scan, the design is wrong.
Step 3 — assign entity prefixes. Decide the PK/SK value templates per entity so related items collide into the same partition on purpose.
Step 4 — validate. Re-walk every access pattern against the key design; confirm none requires a Scan or a FilterExpression doing heavy lifting.

PK/SK overloading — the core technique.

The same PK value groups related entities: USER#123 holds the profile (SK = PROFILE), the user's orders (SK = ORDER#456), and their addresses (SK = ADDRESS#home).
begins_with(SK, 'ORDER#') filters an item collection to just the orders — a cheap sort-key condition, not a filter over unrelated data.
The adjacency-list pattern models many-to-many relationships (an order belongs to a user; a line item belongs to an order) by making the child's PK the parent's identifier.

What interviewers listen for.

Do you start by listing access patterns, not by drawing entities? — required answer.
Do you use overloaded generic keys (PK/SK with typed prefixes) rather than one table per entity? — senior signal.
Do you explain item collections and "one Query returns the whole aggregate"? — senior signal.
Do you acknowledge the write-side cost of denormalization and how you keep duplicates consistent? — senior signal.

Worked example — listing access patterns before the schema

Detailed explanation. The discipline that separates a working DynamoDB model from a broken one is writing the access-pattern list first. Walk through building it for a small e-commerce domain with users, orders, and line items.

The domain. A user places orders; each order has line items. The app never runs ad-hoc analytics against this table — that is the warehouse's job (section 5).
The patterns. Get a user; list a user's orders; get an order plus its line items; get a single line item.
The rule. Each pattern must resolve to a Query, never a Scan.

Question. Enumerate the access patterns and map each to a concrete key condition on a single table.

Input.

#	Access pattern	Frequency
1	Get user profile	high
2	List a user's orders	high
3	Get an order + its line items	high
4	Get one line item	medium

Code.

Access-pattern map (single table "app", keys PK / SK)
=====================================================

1. Get user profile
   PK = USER#<user_id>   SK = PROFILE

2. List a user's orders
   PK = USER#<user_id>   SK begins_with "ORDER#"

3. Get an order + its line items
   PK = ORDER#<order_id> SK begins_with ""   (returns ORDER meta + ITEM#* rows)

4. Get one line item
   PK = ORDER#<order_id> SK = ITEM#<item_id>

Item shapes
-----------
USER#123  | PROFILE      | name, email, tier
USER#123  | ORDER#456    | order_id, total_cents, status   (duplicated for pattern 2)
ORDER#456 | ORDER#456    | order_id, user_id, total_cents, status
ORDER#456 | ITEM#1       | sku, qty, price_cents
ORDER#456 | ITEM#2       | sku, qty, price_cents

Step-by-step explanation.

Pattern 1 stores the profile under PK = USER#123, SK = PROFILE — a direct GetItem.
Pattern 2 stores a lightweight copy of each order under the user's partition (SK = ORDER#456), so begins_with(SK, 'ORDER#') lists them in one query. This copy is a deliberate denormalization for read speed.
Pattern 3 stores the authoritative order and its line items under PK = ORDER#456; a single Query on that partition returns the order metadata row plus every ITEM#* row — the whole aggregate in one round trip.
Pattern 4 is a direct GetItem on (ORDER#456, ITEM#1).
Nothing here requires a Scan or a FilterExpression over unrelated items — every pattern is a partition-scoped Query or a point GetItem, which is the mark of a correct design.

Output.

Pattern	Key condition	Operation
Get user	`PK=USER#id, SK=PROFILE`	GetItem
List orders	`PK=USER#id, begins_with(SK,'ORDER#')`	Query
Order + items	`PK=ORDER#id`	Query
One line item	`PK=ORDER#id, SK=ITEM#id`	GetItem

Rule of thumb. If you cannot write every access pattern as a Query or GetItem against your key design before you create the table, you are not done modeling. A pattern that forces a Scan is a design bug, not a runtime tuning problem.

Worked example — overloading PK/SK with entity prefixes

Detailed explanation. Overloading is the mechanism that lets one table hold many entity types. The PK and SK are generic; their values encode both the entity type and its id. Walk through writing the items and querying an item collection.

The convention. <ENTITY>#<id> for both key parts, plus a type attribute for clarity and stream filtering.
The payoff. Related items live in one partition and come back in one query, sorted by SK.

Question. Write the items for one user with two orders, then fetch the user plus their order list in a single query.

Input.

PK	SK	type	payload
USER#123	PROFILE	User	name, email
USER#123	ORDER#456	OrderRef	total_cents, status
USER#123	ORDER#789	OrderRef	total_cents, status

Code.

import boto3
ddb = boto3.resource("dynamodb")
tbl = ddb.Table("app")

# Write the user profile and two order references into one item collection
with tbl.batch_writer() as bw:
    bw.put_item(Item={"PK": "USER#123", "SK": "PROFILE",
                      "type": "User", "name": "Ada", "email": "ada@x.io"})
    bw.put_item(Item={"PK": "USER#123", "SK": "ORDER#456",
                      "type": "OrderRef", "total_cents": 4200, "status": "shipped"})
    bw.put_item(Item={"PK": "USER#123", "SK": "ORDER#789",
                      "type": "OrderRef", "total_cents": 1599, "status": "pending"})

# One Query returns the profile AND both orders (whole item collection)
from boto3.dynamodb.conditions import Key
resp = tbl.query(KeyConditionExpression=Key("PK").eq("USER#123"))
for item in resp["Items"]:
    print(item["SK"], item.get("type"))
# PROFILE   User
# ORDER#456 OrderRef
# ORDER#789 OrderRef

Step-by-step explanation.

All three items share PK = USER#123, so they land in the same item collection, physically co-located and sorted by SK.
PROFILE sorts before ORDER#456 and ORDER#789 because P < O? No — O < P lexically, so orders sort first; if you want the profile first, prefix it (#PROFILE) or query with ScanIndexForward and handle ordering client-side. The lesson: sort-key value design controls result order.
A single Query on PK = USER#123 returns the entire collection — one network round trip for the user and all their orders.
To fetch only orders, add SK begins_with 'ORDER#'; the sort-key condition is evaluated on the index, so you pay only for matching items.
The type attribute is not required for querying but is invaluable downstream — stream consumers and the warehouse use it to route and unpack heterogeneous items.

Output.

Query	Returns	Round trips
`PK=USER#123`	profile + 2 orders	1
`PK=USER#123, begins_with(SK,'ORDER#')`	2 orders only	1
`GetItem(USER#123, PROFILE)`	profile only	1

Rule of thumb. Design sort-key values as deliberately as partition keys — prefixes give you cheap begins_with filtering and control result ordering. A type attribute on every item pays for itself the moment you build a stream consumer or export.

Worked example — the adjacency-list pattern for relationships

Detailed explanation. Many-to-many and hierarchical relationships (a user has many orders; an order has many items; an item can appear in many orders) are modeled with the adjacency-list pattern: edges are items whose PK is one node and whose SK is the other. Walk through modeling "which users bought a given SKU" alongside "which items are in an order."

The relationship. Order —contains→ Item, and we want both directions eventually (order→items on the base table, sku→orders via a GSI in section 3).
The pattern. Store an edge item PK = ORDER#456, SK = ITEM#sku-9 for the forward direction.

Question. Model order-to-item edges so that "items in an order" is a base-table query, and set up the key so a later GSI can answer "orders containing a SKU."

Input.

PK	SK	GSI1PK	GSI1SK
ORDER#456	ITEM#sku-9	SKU#sku-9	ORDER#456
ORDER#456	ITEM#sku-3	SKU#sku-3	ORDER#456
ORDER#789	ITEM#sku-9	SKU#sku-9	ORDER#789

Code.

# Edge items carry BOTH the base key (order -> items) and GSI keys (sku -> orders)
edges = [
    {"PK": "ORDER#456", "SK": "ITEM#sku-9", "GSI1PK": "SKU#sku-9", "GSI1SK": "ORDER#456", "qty": 2},
    {"PK": "ORDER#456", "SK": "ITEM#sku-3", "GSI1PK": "SKU#sku-3", "GSI1SK": "ORDER#456", "qty": 1},
    {"PK": "ORDER#789", "SK": "ITEM#sku-9", "GSI1PK": "SKU#sku-9", "GSI1SK": "ORDER#789", "qty": 5},
]

with tbl.batch_writer() as bw:
    for e in edges:
        bw.put_item(Item=e)

# Base table: items in an order
tbl.query(KeyConditionExpression=Key("PK").eq("ORDER#456") & Key("SK").begins_with("ITEM#"))
# -> sku-9 (qty 2), sku-3 (qty 1)

# (Section 3 will query GSI1: GSI1PK = SKU#sku-9 -> ORDER#456, ORDER#789)

Step-by-step explanation.

Each edge item encodes the forward relationship in the base key (PK = ORDER#456, SK = ITEM#sku-9), so "items in order 456" is a begins_with(SK, 'ITEM#') query.
The same edge item also carries GSI1PK = SKU#sku-9 and GSI1SK = ORDER#456. When projected into a GSI, this inverts the relationship, letting you query "orders containing sku-9" — covered in section 3.
Storing both directions on one physical item means a single write maintains both access paths; there is no separate join table to keep in sync.
qty and other edge attributes live on the edge item, exactly where a relational schema would put them on a junction row.
This is how DynamoDB models graphs and many-to-many links without joins: edges are first-class items, and GSIs provide the reverse traversal.

Output.

Traversal	Where	Query
Order → items	base table	`PK=ORDER#456, begins_with(SK,'ITEM#')`
SKU → orders	GSI1	`GSI1PK=SKU#sku-9`
Edge attributes	on the edge item	`qty`, added at write time

Rule of thumb. Model relationships as edge items that carry both the base key and the GSI key. One write maintains both directions of traversal, and you never build or maintain a separate join table — the adjacency list is the join.

Data engineering interview question on single-table design

A senior interviewer might ask: "Model a single DynamoDB table for a SaaS app with tenants, users within a tenant, and projects owned by users. The app must: get a tenant, list users in a tenant, get a user, and list a user's projects — all as single queries. Show the key design, the item shapes, and the query for 'list users in a tenant.'"

Solution Using PK/SK overloading with item-collection queries

import boto3
from boto3.dynamodb.conditions import Key

tbl = boto3.resource("dynamodb").Table("saas")

# Item shapes (overloaded generic keys PK / SK)
items = [
    {"PK": "TENANT#acme", "SK": "TENANT#acme",   "type": "Tenant", "plan": "enterprise"},
    {"PK": "TENANT#acme", "SK": "USER#u1",       "type": "User",   "email": "a@acme.io"},
    {"PK": "TENANT#acme", "SK": "USER#u2",       "type": "User",   "email": "b@acme.io"},
    {"PK": "USER#u1",     "SK": "PROJECT#p10",   "type": "Project","name": "Atlas"},
    {"PK": "USER#u1",     "SK": "PROJECT#p11",   "type": "Project","name": "Nova"},
]
with tbl.batch_writer() as bw:
    for it in items:
        bw.put_item(Item=it)

# Access pattern: list users in a tenant (one item collection, one query)
resp = tbl.query(
    KeyConditionExpression=Key("PK").eq("TENANT#acme") & Key("SK").begins_with("USER#")
)
users = [i["email"] for i in resp["Items"]]
print(users)   # ['a@acme.io', 'b@acme.io']

Step-by-step trace.

PK	SK	Entity	Serves access pattern
TENANT#acme	TENANT#acme	Tenant	get a tenant
TENANT#acme	USER#u1	User	list users in tenant
TENANT#acme	USER#u2	User	list users in tenant
USER#u1	PROJECT#p10	Project	list a user's projects
USER#u1	PROJECT#p11	Project	list a user's projects

Walking the query: PK = TENANT#acme selects the tenant's item collection; begins_with(SK, 'USER#') narrows it to just the user items, skipping the tenant metadata row; the result is both users in one round trip. "List a user's projects" is the same shape against PK = USER#u1. Every required access pattern is a single partition-scoped query — no scans, no joins.

Output:

Access pattern	Key condition	Result
Get tenant	`PK=TENANT#acme, SK=TENANT#acme`	1 item
List users in tenant	`PK=TENANT#acme, begins_with(SK,'USER#')`	2 users
Get user	`PK=TENANT#acme, SK=USER#u1`	1 item
List user's projects	`PK=USER#u1, begins_with(SK,'PROJECT#')`	2 projects

Why this works — concept by concept:

Overloaded keys — PK and SK are generic attribute names whose values (TENANT#acme, USER#u1) carry entity type and id, letting tenants, users, and projects coexist in one table without a schema per entity.
Item collections — co-locating a tenant with its users under PK = TENANT#acme means "list users in a tenant" is a single sort-key-filtered query over a physically contiguous collection.
begins_with on the sort key — the USER# and PROJECT# prefixes turn a heterogeneous collection into cheaply filterable sub-lists, evaluated on the index rather than as a post-read filter.
Access-pattern-first design — every one of the four required reads was mapped to a key condition before the table existed, which is why none of them degrades to a Scan.
Cost — each access pattern is O(items returned) on a single partition: one Query per read, no cross-partition fan-out, no join. The write-side cost is maintaining the entity-prefixed keys, paid once per item at write time.

Database
Topic — database
Single-table and data-modeling problems

Practice →

Data processing Topic — data-processing Denormalization and aggregate-modeling problems

Practice →

3. GSIs and query patterns

A GSI is a second partition key on the same data — it buys you a new access pattern at the cost of a second, eventually-consistent write

The mental model in one line: a Global Secondary Index (GSI) is an automatically-maintained copy of your table's items, re-keyed on different attributes, so you can Query the same data along a partition key the base table doesn't support — it has its own partition/sort key, its own throughput, its own projected subset of attributes, and it is always eventually consistent, which means every access pattern the base table can't serve directly becomes "add a GSI," bounded by the extra write cost and the propagation lag. GSIs are the single most important tool for turning a rigid key-value store into something that answers many questions, and misunderstanding them (LSI vs GSI, projection cost, sparse indexes) is the most common DynamoDB interview stumble.

GSI vs LSI — the distinction interviewers probe.

GSI (Global Secondary Index). Any attributes as its (partition, sort) key; can be added any time; has its own provisioned/on-demand throughput; is eventually consistent only. Up to 20 per table (soft limit). This is the workhorse.
LSI (Local Secondary Index). Shares the base table's partition key but a different sort key; must be created at table creation and cannot be added later; supports strongly-consistent reads; shares the base table's throughput; subject to a 10 GB item-collection limit. Rarely worth the constraints in 2026.
The default choice. Reach for a GSI unless you specifically need strong consistency on an alternate sort key within the same partition — then, and only then, consider an LSI, and only if you can create it up front.

Projections — what the GSI copies.

KEYS_ONLY. The GSI stores only the index and base keys. Smallest and cheapest; every read that needs more attributes must fetch back from the base table.
INCLUDE. Keys plus a named list of extra attributes. The right middle ground — project exactly what the access pattern reads.
ALL. Every attribute is copied into the GSI. Most convenient, most storage, highest write cost; use when the GSI must satisfy reads without touching the base table.

GSI overloading and sparse GSIs.

GSI overloading. Just like the base table, name the GSI keys generically (GSI1PK, GSI1SK) and put typed values in them, so a single GSI serves multiple access patterns across entity types.
Sparse GSI. An item only appears in a GSI if it has the GSI's key attributes. Omit the GSI key on items you don't want indexed and the index stays small — perfect for "find items in state X" where X is rare (e.g. open_ticket = true).
Query vs Scan on a GSI. A GSI is queried exactly like a table — KeyConditionExpression on its keys — and you should never Scan it for a targeted read.

What interviewers listen for.

Do you correctly state "GSIs are eventually consistent, LSIs can be strong"? — required answer.
Do you know a GSI has its own throughput and an under-provisioned GSI throttles writes to the base table (provisioned mode)? — senior signal.
Do you use a sparse GSI to make "find the few items in state X" cheap instead of scanning? — senior signal.
Do you choose a projection deliberately rather than defaulting to ALL? — senior signal.

Worked example — an inverted-index GSI (sort key becomes partition key)

Detailed explanation. The most common GSI is the "inverted index": take an attribute that is a sort key or a child id on the base table and make it the GSI's partition key, so you can look items up by the other end of the relationship. Walk through inverting the order/item edges from section 2 to answer "which orders contain SKU X."

Base table. Edge items PK = ORDER#456, SK = ITEM#sku-9.
The question the base table can't answer. "Which orders contain sku-9?" — that needs sku as a partition key.
The GSI. Partition by GSI1PK = SKU#sku-9, sort by GSI1SK = ORDER#456.

Question. Create the GSI and query all orders that contain a given SKU.

Input.

Base PK	Base SK	GSI1PK	GSI1SK
ORDER#456	ITEM#sku-9	SKU#sku-9	ORDER#456
ORDER#789	ITEM#sku-9	SKU#sku-9	ORDER#789
ORDER#456	ITEM#sku-3	SKU#sku-3	ORDER#456

Code.

import boto3
from boto3.dynamodb.conditions import Key

ddb = boto3.client("dynamodb")

# Add an inverted GSI keyed on the SKU
ddb.update_table(
    TableName="app",
    AttributeDefinitions=[
        {"AttributeName": "GSI1PK", "AttributeType": "S"},
        {"AttributeName": "GSI1SK", "AttributeType": "S"},
    ],
    GlobalSecondaryIndexUpdates=[{
        "Create": {
            "IndexName": "GSI1",
            "KeySchema": [
                {"AttributeName": "GSI1PK", "KeyType": "HASH"},
                {"AttributeName": "GSI1SK", "KeyType": "RANGE"},
            ],
            "Projection": {"ProjectionType": "INCLUDE",
                           "NonKeyAttributes": ["qty"]},
        }
    }],
)

# Query: which orders contain sku-9?
tbl = boto3.resource("dynamodb").Table("app")
resp = tbl.query(IndexName="GSI1",
                 KeyConditionExpression=Key("GSI1PK").eq("SKU#sku-9"))
orders = [i["GSI1SK"] for i in resp["Items"]]
print(orders)   # ['ORDER#456', 'ORDER#789']

Step-by-step explanation.

The base table can query "items in an order" but not "orders containing a SKU," because sku is only a sort-key fragment, never a partition key.
The GSI re-keys the same edge items on GSI1PK = SKU#<sku>, so all edges for one SKU collide into a single GSI partition, queryable in one round trip.
Projection = INCLUDE [qty] copies only the keys plus qty into the GSI — enough to answer "which orders, and how many units" without fetching back to the base table.
The GSI is maintained automatically: every base-table write that touches GSI1PK/GSI1SK propagates to the GSI asynchronously (eventual consistency, typically well under a second).
Querying by GSI1PK = SKU#sku-9 returns both orders — the inverse traversal the base table could not provide.

Output.

GSI query	Returns	Consistency
`GSI1PK=SKU#sku-9`	ORDER#456, ORDER#789	eventual
`GSI1PK=SKU#sku-3`	ORDER#456	eventual
projection	keys + `qty`	INCLUDE

Rule of thumb. When you need to query by the "other end" of a relationship, invert it with a GSI: put the child/attribute on GSI1PK and the parent on GSI1SK. Project only the attributes the access pattern actually reads to keep the index small and write-cheap.

Worked example — a sparse GSI for a rare-state filter

Detailed explanation. A sparse GSI indexes only the items that carry the GSI's key attributes, which makes "find the few items in state X" a tiny, cheap query instead of a full-table scan. Walk through indexing only open support tickets out of millions of mostly-closed ones.

The problem. 50 million tickets, 99.9% closed. "List open tickets" via Scan + filter reads all 50M items.
The sparse trick. Only open tickets get a GSI1PK = OPEN attribute; closed tickets omit it, so they never enter the GSI.
The payoff. The GSI holds only ~50k open tickets; querying it is proportional to open tickets, not total tickets.

Question. Design a sparse GSI so "list all open tickets" costs O(open), and show how closing a ticket removes it from the index.

Input.

ticket_id	status	GSI1PK (sparse)
T-1	open	OPEN
T-2	closed	(absent)
T-3	open	OPEN

Code.

import boto3
from boto3.dynamodb.conditions import Key
tbl = boto3.resource("dynamodb").Table("tickets")

# Opening a ticket: set the sparse GSI key
tbl.put_item(Item={"PK": "TICKET#T-1", "status": "open",
                   "GSI1PK": "OPEN", "GSI1SK": "2026-09-05T09:00Z"})

# Closing a ticket: REMOVE the sparse key so it drops out of the GSI
tbl.update_item(
    Key={"PK": "TICKET#T-1"},
    UpdateExpression="SET #s = :closed REMOVE GSI1PK, GSI1SK",
    ExpressionAttributeNames={"#s": "status"},
    ExpressionAttributeValues={":closed": "closed"},
)

# List all open tickets — reads only items in the sparse GSI
open_tickets = tbl.query(IndexName="GSI1",
                         KeyConditionExpression=Key("GSI1PK").eq("OPEN"))

Step-by-step explanation.

Only items that possess GSI1PK appear in the GSI. Open tickets set it to the constant OPEN; closed tickets have no such attribute.
"List open tickets" becomes Query GSI1 WHERE GSI1PK = 'OPEN', reading only the open tickets — not the 50 million closed ones.
Closing a ticket uses REMOVE GSI1PK, GSI1SK in the update expression, which deletes the item from the GSI while keeping it in the base table. The item silently drops out of the index.
Because all open tickets share the constant partition key OPEN, watch for a hot GSI partition if "open" volume is huge; shard the constant (OPEN#<n>) if needed.
The base table still holds every ticket for point lookups by ticket_id; the sparse GSI is purely the efficient "current open set."

Output.

Operation	Effect on GSI	Cost of "list open"
Open ticket	item enters GSI	—
Close ticket (`REMOVE` keys)	item leaves GSI	—
Query `GSI1PK=OPEN`	reads only open items	O(open), not O(total)

Rule of thumb. For "find the small set of items currently in state X," use a sparse GSI: write the GSI key only while the item is in that state and REMOVE it when the state ends. You turn an O(table) scan into an O(matching-items) query.

Worked example — GSI overloading to serve multiple patterns from one index

Detailed explanation. Just as the base table's keys are overloaded, a single GSI's generic keys (GSI1PK/GSI1SK) can serve several unrelated access patterns by putting different typed values in them per entity. Walk through one GSI answering both "orders by status" and "users by email domain."

The idea. Different entity types write different value templates into the same GSI1PK.
The payoff. Fewer indexes (each GSI costs storage + write throughput), more access patterns per index.

Question. Use one GSI to answer "orders in status pending" and "users at email domain acme.io."

Input.

Entity	GSI1PK	GSI1SK
Order	STATUS#pending	ORDER#456
Order	STATUS#shipped	ORDER#457
User	DOMAIN#acme.io	USER#u1

Code.

from boto3.dynamodb.conditions import Key

# Orders currently pending
tbl.query(IndexName="GSI1",
          KeyConditionExpression=Key("GSI1PK").eq("STATUS#pending"))
# -> ORDER#456

# Users at a given email domain — SAME index, different value template
tbl.query(IndexName="GSI1",
          KeyConditionExpression=Key("GSI1PK").eq("DOMAIN#acme.io"))
# -> USER#u1

Step-by-step explanation.

Both orders and users write into the same GSI1PK attribute, but with distinct value namespaces: STATUS#... for orders, DOMAIN#... for users.
A query for GSI1PK = STATUS#pending returns only order items in that status; a query for GSI1PK = DOMAIN#acme.io returns only users at that domain. The namespaces never collide.
One physical GSI thus serves two logically-unrelated access patterns, saving the storage and write-throughput cost of a second index.
When an order ships, updating GSI1PK from STATUS#pending to STATUS#shipped moves it between GSI partitions automatically — the "status index" stays correct with no extra bookkeeping.
The discipline: keep value namespaces globally unique across entity types so a single overloaded GSI never conflates two patterns.

Output.

Query	GSI1PK value	Returns
Orders pending	`STATUS#pending`	ORDER#456
Orders shipped	`STATUS#shipped`	ORDER#457
Users at domain	`DOMAIN#acme.io`	USER#u1

Rule of thumb. Overload one GSI across entity types with namespaced key values (STATUS#…, DOMAIN#…) before creating a second GSI. Every extra GSI is extra storage and write amplification; a well-namespaced overloaded index does the work of several.

Data engineering interview question on GSIs

A senior interviewer might ask: "You have a 200-million-row orders table keyed by order_id. Product needs two new reads: 'all orders for a customer, newest first' and 'all orders currently in the refund_pending state.' Design the indexes, choose projections, and explain the consistency and cost implications, including why one of them should be sparse."

Solution Using a customer GSI plus a sparse status GSI

import boto3
from boto3.dynamodb.conditions import Key

ddb = boto3.client("dynamodb")

ddb.update_table(
    TableName="orders",
    AttributeDefinitions=[
        {"AttributeName": "customer_id", "AttributeType": "S"},
        {"AttributeName": "created_at",  "AttributeType": "S"},
        {"AttributeName": "refund_pk",   "AttributeType": "S"},  # sparse: present only when refund_pending
    ],
    GlobalSecondaryIndexUpdates=[
        {"Create": {
            "IndexName": "gsi_customer",
            "KeySchema": [
                {"AttributeName": "customer_id", "KeyType": "HASH"},
                {"AttributeName": "created_at",  "KeyType": "RANGE"},
            ],
            "Projection": {"ProjectionType": "INCLUDE",
                           "NonKeyAttributes": ["total_cents", "status"]},
        }},
        {"Create": {
            "IndexName": "gsi_refund",
            "KeySchema": [{"AttributeName": "refund_pk", "KeyType": "HASH"},
                          {"AttributeName": "created_at", "KeyType": "RANGE"}],
            "Projection": {"ProjectionType": "KEYS_ONLY"},
        }},
    ],
)

tbl = boto3.resource("dynamodb").Table("orders")

# All orders for a customer, newest first
tbl.query(IndexName="gsi_customer",
          KeyConditionExpression=Key("customer_id").eq("CUST#42"),
          ScanIndexForward=False)

# All orders currently refund_pending (sparse index -> only those items exist here)
tbl.query(IndexName="gsi_refund",
          KeyConditionExpression=Key("refund_pk").eq("REFUND_PENDING"))

Step-by-step trace.

Requirement	Index	Key design	Projection
Orders for a customer, newest first	`gsi_customer`	PK `customer_id`, SK `created_at`	INCLUDE total, status
Orders in `refund_pending`	`gsi_refund` (sparse)	PK `refund_pk` (const), SK `created_at`	KEYS_ONLY
Set refund state	write `refund_pk='REFUND_PENDING'`	item enters sparse GSI	—
Clear refund state	`REMOVE refund_pk`	item leaves sparse GSI	—

Walking it: the customer GSI re-keys every order by customer_id, so "orders for CUST#42, newest first" is a descending query returning total_cents and status from the projection without touching the base table. The refund GSI is sparse — only orders with refund_pk set are present — so "list refund_pending" reads a few thousand items, not 200 million; because the projection is KEYS_ONLY, those reads are tiny and the code fetches full order details from the base table only for the handful that need action.

Output:

Access pattern	Reads	Consistency	Cost
Customer's orders	`gsi_customer` query	eventual	O(customer's orders)
Refund-pending list	`gsi_refund` query	eventual	O(pending), not O(table)
Full order detail	base-table GetItem	strong (if needed)	1 read per item

Why this works — concept by concept:

GSI as an alternate partition key — re-keying orders on customer_id provides an access path the base table (keyed on order_id) fundamentally cannot, without duplicating the table by hand.
Sparse GSI — writing refund_pk only while an order is refund-pending keeps that index tiny, converting "find orders in a rare state" from an O(200M) scan into an O(few-thousand) query.
Projection choice — INCLUDE on the customer GSI carries just the attributes the list view shows; KEYS_ONLY on the sparse GSI keeps its per-item write cost minimal since it is only a routing index.
Eventual consistency — both GSIs are eventual-only, which is fine for list views; anything needing the authoritative latest value reads the base table with strong consistency.
Cost — each GSI adds write amplification (one extra write per indexed item) and storage for its projection, but turns two impossible-or-scan access patterns into O(result-size) queries. The sparse index is the key lever: its cost scales with the rare state, not the whole table.

Indexing
Topic — indexing
Secondary-index and query-path problems

Practice →

Database Topic — database Query-design and access-pattern problems

Practice →

4. DynamoDB Streams — change data capture

DynamoDB Streams is a 24-hour ordered change log — the native CDC tap for every item mutation

The mental model in one line: DynamoDB Streams is an ordered, 24-hour-retained log of every item-level change (INSERT, MODIFY, REMOVE) in a table, sharded so that all changes to a given partition key are strictly ordered, and consumable by Lambda triggers or the Kinesis Client Library — it is the mechanism that turns DynamoDB from a data island into a source of truth that fans out to search indexes, aggregate tables, and the analytics warehouse without ever polling or scanning the table. For a data engineer, Streams is the DynamoDB equivalent of a write-ahead log tail: it is how you build materialized views, replicate to other stores, and feed real-time CDC.

The primitives that matter.

Stream record. One record per item-level change, carrying the eventName (INSERT / MODIFY / REMOVE), the item keys, and — depending on the view type — the new and/or old image of the item.
24-hour retention. Records live for 24 hours, then expire. Consumers must keep up or use the Kinesis Data Streams integration for longer retention (up to 365 days).
Shards and ordering. The stream is partitioned into shards; all changes for a single partition key go to the same shard and are delivered in order. There is no global ordering across partition keys — only per-key.
At-least-once delivery. A record can be delivered more than once (retries, resharding), so consumers must be idempotent.

Stream view types — choose what each record carries.

KEYS_ONLY. Just the key attributes of the changed item. Smallest; the consumer must read the current item if it needs more.
NEW_IMAGE. The entire item as it looks after the change. Ideal for replicating current state.
OLD_IMAGE. The entire item as it looked before the change. Needed to compute deltas or capture what a REMOVE deleted.
NEW_AND_OLD_IMAGES. Both — the richest and the usual choice for CDC and aggregation, since you can diff old vs new.

Ordering, delivery, and consumers.

Lambda trigger. The simplest consumer: DynamoDB invokes your function with batches of records, tracks checkpoints, and retries on failure. Batch size, parallelization factor, and bisect-on-error are the tuning knobs.
Kinesis adapter / KCL. For fan-out to multiple independent consumers or longer retention, route the stream through Kinesis Data Streams.
Failure handling. Configure a bisect-on-function-error and an on-failure destination (SQS/SNS DLQ) so one poison record doesn't block a shard forever.

What interviewers listen for.

Do you name the four view types and pick NEW_AND_OLD_IMAGES for CDC/aggregation? — required answer.
Do you state "ordering is per partition key, not global"? — senior signal.
Do you make consumers idempotent because delivery is at-least-once? — required answer.
Do you know the 24-hour limit and reach for Kinesis when you need longer retention or multiple consumers? — senior signal.

Worked example — a Lambda stream processor that maintains a materialized aggregate

Detailed explanation. The canonical Streams use case is a materialized view: as orders are written, keep a running per-customer order count and total in a separate aggregate item. Walk through the Lambda that consumes NEW_AND_OLD_IMAGES and updates the aggregate idempotently.

The trigger. Table orders with Streams on, view NEW_AND_OLD_IMAGES.
The target. An aggregate item PK = CUST#42, SK = AGG holding order_count and total_cents.
Idempotency. Because records can redeliver, track processed SequenceNumbers (or use conditional math) so a replay doesn't double-count.

Question. Write the Lambda handler that increments the customer aggregate on INSERT and decrements on REMOVE, safely under at-least-once delivery.

Input.

eventName	new image	old image	aggregate effect
INSERT	order, total 4200	—	count +1, total +4200
MODIFY	total 4500	total 4200	total +300
REMOVE	—	order, total 4500	count -1, total -4500

Code.

import boto3
tbl = boto3.resource("dynamodb").Table("orders")

def handler(event, _ctx):
    for rec in event["Records"]:
        ev   = rec["eventName"]                 # INSERT | MODIFY | REMOVE
        seq  = rec["dynamodb"]["SequenceNumber"]
        new  = rec["dynamodb"].get("NewImage", {})
        old  = rec["dynamodb"].get("OldImage", {})

        cust = (new or old)["customer_id"]["S"]
        new_total = int(new["total_cents"]["N"]) if new else 0
        old_total = int(old["total_cents"]["N"]) if old else 0

        d_count = (1 if ev == "INSERT" else -1 if ev == "REMOVE" else 0)
        d_total = new_total - old_total          # works for all three events

        try:
            tbl.update_item(
                Key={"PK": f"CUST#{cust}", "SK": "AGG"},
                UpdateExpression=("ADD order_count :c, total_cents :t "
                                  "SET last_seq = :s"),
                ConditionExpression="attribute_not_exists(last_seq) OR last_seq < :s",
                ExpressionAttributeValues={":c": d_count, ":t": d_total, ":s": seq},
            )
        except tbl.meta.client.exceptions.ConditionalCheckFailedException:
            pass   # already applied this or a newer record -> idempotent skip

Step-by-step explanation.

Each record carries eventName, the SequenceNumber, and (with NEW_AND_OLD_IMAGES) both images, so the handler can compute the exact delta for count and total.
d_total = new_total - old_total is uniform across all three events: on INSERT old is 0, on REMOVE new is 0, on MODIFY it is the real difference.
The ADD update atomically increments the aggregate counters — DynamoDB's atomic counters avoid read-modify-write races between concurrent invocations.
The ConditionExpression on last_seq makes the update idempotent: a redelivered (older-or-equal SequenceNumber) record fails the condition and is skipped, so at-least-once delivery never double-counts.
On ConditionalCheckFailedException the handler swallows the error — that is the expected path for a duplicate, not a failure.

Output.

Event stream	order_count	total_cents
INSERT 4200	1	4200
MODIFY 4200→4500	1	4500
REMOVE 4500	0	0
(INSERT 4200 redelivered)	0	0 (skipped)

Rule of thumb. Build stream consumers as idempotent delta-appliers: use atomic ADD for counters, compute deltas from NEW_AND_OLD_IMAGES, and guard with a monotonic SequenceNumber condition so redelivery is a no-op. Never assume exactly-once — DynamoDB Streams is at-least-once.

Worked example — choosing the right stream view type

Detailed explanation. The view type you pick determines both what your consumer can do and how much data crosses the stream. Walk through matching view type to three consumer goals.

Replicate current state to OpenSearch. You need the full post-change item → NEW_IMAGE.
Maintain deltas / audit before-and-after. You need both images → NEW_AND_OLD_IMAGES.
Just invalidate a cache by key. You need only the key → KEYS_ONLY.

Question. For each consumer, choose the minimal view type that satisfies it and justify the cost trade-off.

Input.

Consumer goal	Needs new image?	Needs old image?	View type
Replicate to search index	yes	no	NEW_IMAGE
Delta aggregate / audit	yes	yes	NEW_AND_OLD_IMAGES
Cache invalidation	no	no	KEYS_ONLY

Code.

import boto3
ddb = boto3.client("dynamodb")

# Enable streams with the richest view for a CDC/aggregation table
ddb.update_table(
    TableName="orders",
    StreamSpecification={"StreamEnabled": True,
                         "StreamViewType": "NEW_AND_OLD_IMAGES"},
)

# For a table whose only consumer replicates state, NEW_IMAGE is enough:
# StreamViewType="NEW_IMAGE"
# For a table whose only consumer busts a cache, KEYS_ONLY is cheapest:
# StreamViewType="KEYS_ONLY"

Step-by-step explanation.

NEW_IMAGE ships the full item after the change — exactly what a search-index replicator needs to upsert the current document, and nothing more.
NEW_AND_OLD_IMAGES ships both, which is mandatory for anything computing a delta (aggregates) or auditing what changed, and it is the only view that tells you what a REMOVE deleted (via the old image).
KEYS_ONLY ships only the keys — smallest payload — and is perfect when the consumer just needs "this key changed, go invalidate it."
Richer views cost more stream throughput and Lambda payload size, so pick the minimal view that satisfies every consumer of that table.
If different consumers need different views, either standardize on the richest they collectively require, or route through Kinesis so each consumer reads independently.

Output.

View type	Payload	Enables
KEYS_ONLY	keys only	cache invalidation
NEW_IMAGE	post-change item	state replication
OLD_IMAGE	pre-change item	delete capture, deltas
NEW_AND_OLD_IMAGES	both	CDC, aggregation, audit

Rule of thumb. Default to NEW_AND_OLD_IMAGES for anything CDC-like — it is the only view that lets you diff and that captures deletes — and drop to NEW_IMAGE or KEYS_ONLY only when you have proven a single consumer needs less.

Worked example — Kinesis Data Streams for longer retention and fan-out

Detailed explanation. Native DynamoDB Streams retain 24 hours and are designed around a single Lambda consumer per shard. When you need multiple independent consumers, longer retention, or integration with the broader Kinesis/Flink ecosystem, route change data through Kinesis Data Streams instead. Walk through the trade-off.

The limit. 24-hour retention; if a downstream backfill needs a week of history, native streams can't provide it.
The fix. Enable Kinesis Data Streams for DynamoDB — same change records, but into a Kinesis stream with configurable retention (up to 365 days) and multiple consumers via enhanced fan-out.
The caveat. Kinesis for DynamoDB is at-least-once and not guaranteed strictly ordered across records the way native streams are per shard, so idempotency matters even more.

Question. Decide between native Streams and Kinesis for a table that must feed both a real-time aggregator and a nightly warehouse backfill needing 7 days of replay.

Input.

Requirement	Native Streams	Kinesis for DynamoDB
Retention	24 h	up to 365 days
Multiple consumers	1 primary (Lambda)	many (enhanced fan-out)
Ordering	strict per partition key	best-effort
Replay window	24 h	configurable

Code.

import boto3
ddb = boto3.client("dynamodb")

# Route DynamoDB change records into a Kinesis Data Stream (long retention + fan-out)
ddb.enable_kinesis_streaming_destination(
    TableName="orders",
    StreamArn="arn:aws:kinesis:us-east-1:123456789012:stream/orders-cdc",
)

# Consumers now read from Kinesis:
#   - real-time aggregator (Lambda / KCL)
#   - nightly warehouse loader replaying up to 7 days

Step-by-step explanation.

Native Streams are ideal for a single, keep-up consumer with strict per-key ordering and a 24-hour safety window.
When a second, independent consumer (the warehouse backfill) needs its own cursor and a 7-day replay window, native Streams' 24-hour retention and single-primary-consumer model no longer fit.
enable_kinesis_streaming_destination mirrors the same change records into a Kinesis Data Stream, where retention is configurable (up to 365 days) and enhanced fan-out gives each consumer its own throughput.
The trade-off: Kinesis for DynamoDB relaxes strict ordering guarantees, so consumers must be idempotent and tolerate occasional reordering — reconcile by SequenceNumber/timestamp.
A common production shape is both: native Streams → Lambda for the low-latency aggregate, and Kinesis → Firehose → S3 for durable, replayable history feeding the warehouse.

Output.

Consumer	Source	Why
Real-time aggregate	native Streams → Lambda	strict order, low latency
7-day warehouse backfill	Kinesis → Firehose → S3	long retention, replay
Multiple analytics readers	Kinesis enhanced fan-out	independent cursors

Rule of thumb. Use native Streams for a single low-latency consumer needing strict per-key order and a 24-hour window; switch to (or add) Kinesis Data Streams when you need long retention, replay beyond a day, or several independent consumers. Either way, make every consumer idempotent.

Data engineering interview question on DynamoDB Streams

A senior interviewer might ask: "Your orders table must, in near-real time, keep a per-customer 'lifetime value' aggregate and replicate every order into OpenSearch for full-text search. Design the Streams setup — view type, consumers, ordering guarantees, idempotency, and failure handling — and explain how you'd replay the last 12 hours after a bad deploy."

Solution Using Streams → Lambda with idempotent aggregation and a DLQ

import boto3
tbl = boto3.resource("dynamodb").Table("orders")
search = boto3.client("opensearch")   # illustrative client

# Table configured with StreamViewType = NEW_AND_OLD_IMAGES
def handler(event, _ctx):
    for rec in event["Records"]:
        ev  = rec["eventName"]
        seq = rec["dynamodb"]["SequenceNumber"]
        new = rec["dynamodb"].get("NewImage", {})
        old = rec["dynamodb"].get("OldImage", {})
        cust = (new or old)["customer_id"]["S"]

        # 1. Maintain lifetime-value aggregate (idempotent atomic add)
        delta = (int(new.get("total_cents", {}).get("N", 0)) -
                 int(old.get("total_cents", {}).get("N", 0)))
        try:
            tbl.update_item(
                Key={"PK": f"CUST#{cust}", "SK": "LTV"},
                UpdateExpression="ADD ltv_cents :d SET last_seq = :s",
                ConditionExpression="attribute_not_exists(last_seq) OR last_seq < :s",
                ExpressionAttributeValues={":d": delta, ":s": seq},
            )
        except tbl.meta.client.exceptions.ConditionalCheckFailedException:
            pass  # duplicate/older record -> skip

        # 2. Replicate to search (upsert on INSERT/MODIFY, delete on REMOVE)
        doc_id = new.get("order_id", old.get("order_id"))["S"]
        if ev == "REMOVE":
            delete_from_search(doc_id)      # idempotent delete
        else:
            upsert_to_search(doc_id, new)   # idempotent upsert by id

Step-by-step trace.

Step	Mechanism	Guarantee
View type	`NEW_AND_OLD_IMAGES`	delta + delete capture
Ordering	per `customer_id` shard	LTV updates ordered per customer
Aggregate	atomic `ADD` + `last_seq` guard	idempotent under redelivery
Search replicate	upsert/delete by `order_id`	idempotent by document id
Poison record	bisect-on-error + SQS DLQ	one bad record doesn't wedge shard
Replay 12 h	re-point iterator within 24 h retention	reprocess without data loss

Walking it: every order mutation flows through one Lambda; the LTV aggregate uses an atomic ADD guarded by a monotonic last_seq, so a replay of the last 12 hours (well within the 24-hour retention) re-applies nothing it already saw; the search replication is keyed by order_id, so upserts and deletes are naturally idempotent; and a poison record is bisected out to a dead-letter queue instead of blocking its shard forever.

Output:

Concern	Solution	Result
Double counting	`last_seq` condition	exactly-once effect on LTV
Delete propagation	`REMOVE` → search delete	search stays consistent
Bad deploy	replay within 24 h	no data loss
Poison message	DLQ + bisect	shard keeps flowing
Latency	Lambda trigger	sub-second in steady state

Why this works — concept by concept:

NEW_AND_OLD_IMAGES — carrying both images lets the aggregator compute an exact delta and lets the replicator know what a REMOVE deleted, which is impossible with NEW_IMAGE alone.
Per-partition-key ordering — because all of a customer's order changes land on one shard in order, the LTV aggregate never applies a stale value out of sequence.
Idempotent consumers — the atomic ADD guarded by last_seq, plus upsert/delete-by-id for search, make at-least-once delivery safe: replays and retries are no-ops.
Failure isolation — bisect-on-error plus an SQS dead-letter destination stops a single un-processable record from stalling an entire shard, the classic Streams outage.
Cost — one Lambda invocation per batch, atomic counter updates, and idempotent search writes; replay is free within the 24-hour window. The eliminated cost is polling or scanning the table to detect changes — Streams pushes them at O(changes), not O(table).

Data processing
Topic — data-processing
Streaming and change-data-capture problems

Practice →

Database Topic — database Materialized-view and aggregation problems

Practice →

5. S3 export and warehouse integration

Export to S3 is the zero-RCU, point-in-time bridge from DynamoDB to your analytics warehouse

The mental model in one line: DynamoDB's "export to S3" feature writes a consistent point-in-time snapshot of a table into S3 — as DynamoDB JSON or Amazon Ion, partitioned into files — without consuming any read capacity and without a Scan, by reading from the continuous backup (PITR) rather than the live table, so it is the correct way to move DynamoDB data into a columnar warehouse for analytics, while Streams handles the real-time incremental path. For a data engineer, the rule is blunt: never Scan a production DynamoDB table for analytics — export it to S3 and query it there.

The primitives that matter.

Point-in-time export. Export reads from PITR (point-in-time recovery) continuous backups, so it never touches live table throughput — zero RCU consumed — and produces a transactionally-consistent snapshot as of a chosen timestamp.
PITR is a prerequisite. You must enable PITR on the table before you can export; without it, export is unavailable.
Output format. Files land in S3 as DynamoDB JSON (typed, e.g. {"total_cents":{"N":"4200"}}) or Amazon Ion, gzip-compressed, partitioned across many objects with a manifest.
No Scan, no impact. Because export bypasses the live table, exporting a 10 TB table doesn't throttle production the way a Scan would.

Full export vs incremental export.

Full export. A complete snapshot of the table as of a point in time. Use it to bootstrap the warehouse or take periodic full refreshes.
Incremental export. Exports only the items that changed within a specified time window (backed by PITR), so after the initial full export you sync deltas cheaply instead of re-exporting everything.
The pattern. One full export to seed, then scheduled incremental exports (e.g. hourly/daily windows) to keep the warehouse current — a batch CDC that complements or replaces the Streams path for analytics.

Query engines downstream.

Glue + Athena. Point a Glue crawler at the export prefix to infer a schema, then query the DynamoDB JSON with Athena SQL — serverless, pay-per-scan.
Redshift Spectrum / Spark. External tables over the S3 export let Redshift or Spark join DynamoDB data with the rest of the warehouse.
Unpacking DynamoDB JSON. The typed format ({"N":"..."}, {"S":"..."}) usually needs a flattening step — Athena's json_extract, a Glue transform, or a Spark UDF — to become clean columnar rows.

What interviewers listen for.

Do you say "never Scan for analytics — export to S3"? — required answer.
Do you know export consumes zero RCU and requires PITR? — senior signal.
Do you distinguish full vs incremental export and design a seed-then-delta pipeline? — senior signal.
Do you contrast export (batch snapshots) with Streams (real-time CDC) and pick per requirement? — senior signal.

Worked example — full export to S3 then query with Athena

Detailed explanation. The bootstrap step for any DynamoDB→warehouse pipeline is a full point-in-time export, followed by a Glue crawler and Athena query. Walk through it end to end.

Prerequisite. PITR enabled on the table.
The export. export-table-to-point-in-time to an S3 prefix, DynamoDB JSON, gzip.
The query. Glue crawler → Athena external table → SQL that flattens the typed JSON.

Question. Export the orders table to S3 and write the Athena query that returns total revenue by status.

Input.

Step	Tool	Output
Enable PITR	DynamoDB	continuous backup
Export	`export-table-to-point-in-time`	S3 DynamoDB JSON
Catalog	Glue crawler	Athena table
Query	Athena SQL	revenue by status

Code.

# 1. Ensure PITR is on (one-time)
aws dynamodb update-continuous-backups \
  --table-name orders \
  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true

# 2. Full point-in-time export to S3 (zero RCU on the live table)
aws dynamodb export-table-to-point-in-time \
  --table-arn arn:aws:dynamodb:us-east-1:123456789012:table/orders \
  --s3-bucket my-lake \
  --s3-prefix exports/orders/ \
  --export-format DYNAMODB_JSON

-- 3. After a Glue crawler catalogs exports/orders/, query in Athena.
--    DynamoDB JSON stores typed scalars, so unwrap Item.<attr>.<type>.
SELECT
    Item.status.S                              AS status,
    SUM(CAST(Item.total_cents.N AS BIGINT))    AS revenue_cents,
    COUNT(*)                                    AS order_count
FROM orders_export
GROUP BY Item.status.S
ORDER BY revenue_cents DESC;

Step-by-step explanation.

PITR must be enabled first; it is what the export reads from, which is why the export costs zero live read capacity.
export-table-to-point-in-time writes a consistent snapshot to the S3 prefix as gzipped DynamoDB JSON, partitioned across many objects with a manifest describing them.
A Glue crawler infers the schema of the export (a nested Item struct whose fields are typed maps like status.S and total_cents.N) and registers an Athena table.
The Athena query unwraps the typed JSON — Item.total_cents.N is a string inside the DynamoDB JSON, so CAST(... AS BIGINT) turns it into a number for aggregation.
The whole analytical query runs against S3, never against DynamoDB, so it cannot throttle production no matter how heavy the aggregation.

Output.

status	revenue_cents	order_count
shipped	128,400,000	30,200
pending	9,150,000	2,410
refunded	1,020,000	260

Rule of thumb. Bootstrap the warehouse with a full point-in-time export (PITR required, zero RCU), catalog it with Glue, and always CAST the typed DynamoDB-JSON scalars when querying in Athena. The analytics never touch the live table — that is the whole point.

Worked example — incremental export for ongoing sync

Detailed explanation. Re-exporting a large table in full every hour is wasteful. Incremental export ships only the items changed in a time window, so after the initial seed you sync cheap deltas. Walk through a daily incremental pipeline.

The seed. One full export (previous example).
The delta. Daily incremental export for the previous 24 hours.
The merge. Downstream MERGE/upsert applies inserts, updates, and deletes into the warehouse table by key.

Question. Configure a daily incremental export and describe how the warehouse applies the delta idempotently.

Input.

Parameter	Value
Export type	INCREMENTAL_EXPORT
Window	previous 24 h
Format	DynamoDB JSON
Merge key	`order_id`

Code.

# Daily incremental export: only items changed in the window [from, to)
aws dynamodb export-table-to-point-in-time \
  --table-arn arn:aws:dynamodb:us-east-1:123456789012:table/orders \
  --s3-bucket my-lake --s3-prefix exports/orders/incr/ \
  --export-type INCREMENTAL_EXPORT \
  --incremental-export-specification \
      ExportFromTime=2026-09-04T00:00:00Z,ExportToTime=2026-09-05T00:00:00Z,ExportViewType=NEW_AND_OLD_IMAGES \
  --export-format DYNAMODB_JSON

-- Warehouse merge: apply the incremental delta by key (Redshift/Snowflake-style)
MERGE INTO analytics.orders AS tgt
USING staging.orders_incr AS src
ON tgt.order_id = src.order_id
WHEN MATCHED AND src.op = 'REMOVE' THEN DELETE
WHEN MATCHED                        THEN UPDATE SET
     status = src.status, total_cents = src.total_cents, updated_at = src.updated_at
WHEN NOT MATCHED                    THEN INSERT (order_id, status, total_cents, updated_at)
     VALUES (src.order_id, src.status, src.total_cents, src.updated_at);

Step-by-step explanation.

INCREMENTAL_EXPORT with an explicit [ExportFromTime, ExportToTime) window exports only items mutated in that period, using PITR — again zero live RCU.
ExportViewType=NEW_AND_OLD_IMAGES includes both images so the delta carries enough to distinguish updates from deletes downstream.
The warehouse MERGE keys on order_id: matched-and-removed rows are deleted, matched rows are updated, and new rows are inserted — one statement handles all three change types.
Because the merge is keyed and deterministic, re-running the same incremental window is idempotent — a retried load produces the same warehouse state.
This seed-then-daily-delta pattern is batch CDC: cheaper than full re-exports, and it keeps the warehouse within a day of the source without ever scanning the table.

Output.

Change in window	Delta row `op`	Warehouse effect
New order	INSERT	row inserted
Status change	MODIFY	row updated
Cancelled order	REMOVE	row deleted
Unchanged order	(absent)	untouched

Rule of thumb. Seed once with a full export, then run scheduled incremental exports and a keyed MERGE. Include old-and-new images so deletes propagate, and key the merge so retries are idempotent — that is batch CDC without a single Scan.

Worked example — Streams → Firehose vs export: pick the right path

Detailed explanation. Data engineers must choose between two DynamoDB→analytics paths: real-time via Streams/Kinesis Firehose, or batch via S3 export. They are complementary, and picking the wrong one wastes money or misses SLAs. Walk through the decision.

Real-time path. Streams → Kinesis Firehose → S3 (or Redshift), delivering changes continuously with seconds-to-minutes latency.
Batch path. Full + incremental S3 export on a schedule, delivering consistent snapshots with hours latency but zero RCU and trivial ops.
The choice. Freshness requirement vs cost and operational simplicity.

Question. For a dashboard needing sub-minute freshness and a monthly finance report needing exact consistency, pick a path for each.

Input.

Consumer	Freshness need	Consistency need	Path
Ops dashboard	sub-minute	eventual OK	Streams → Firehose → S3
Finance report	monthly	strong point-in-time	full S3 export

Code.

Decision guide — DynamoDB to analytics
=======================================

Need sub-minute freshness?
   yes -> Streams (NEW_AND_OLD_IMAGES) -> Kinesis Firehose -> S3/Redshift
          + pay per change, real-time, must handle idempotency

Need periodic consistent snapshots / heavy historical scans?
   yes -> S3 export (full to seed, incremental to sync)
          + zero RCU, point-in-time consistent, batch latency

Need both?
   run both: Streams for the live dashboard, export for the
   consistent warehouse tables and backfills.

Step-by-step explanation.

A dashboard that must reflect changes within a minute needs the real-time path: Streams push each mutation to Firehose, which buffers and writes to S3 or Redshift continuously.
Firehose delivery is at-least-once and micro-batched, so the dashboard's loader must dedupe by key — same idempotency discipline as any stream consumer.
A monthly finance report needs exact, consistent numbers as of a timestamp, not the freshest possible — a full point-in-time export gives a transactionally-consistent snapshot with zero impact on the live table.
Export's batch latency (hours) is irrelevant for a monthly report but disqualifying for a live dashboard; conversely, running Streams to satisfy a monthly report would pay for real-time you don't need.
Mature stacks run both: Streams/Firehose for low-latency views, and scheduled exports for the authoritative, replayable warehouse tables.

Output.

Path	Latency	RCU cost	Best for
Streams → Firehose	seconds–minutes	per change	live dashboards
Full export	hours	zero	consistent snapshots, backfills
Incremental export	hours	zero	scheduled warehouse sync

Rule of thumb. Match the path to the freshness SLA: Streams/Firehose when minutes matter, S3 export when consistency and cost matter and hours are fine. They are complementary — run both when you need real-time views and an authoritative warehouse.

Data engineering interview question on S3 export

A senior interviewer might ask: "You must load a 5 TB DynamoDB orders table into Redshift for analytics without impacting the production service, then keep it within a day fresh. Walk me through the initial load, the ongoing sync, why you would not Scan, and how you handle the typed DynamoDB JSON and deletes."

Solution Using PITR full export, Glue catalog, and daily incremental exports

# 1. Enable PITR (prerequisite for any export)
aws dynamodb update-continuous-backups --table-name orders \
  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true

# 2. Initial FULL export — 5 TB, zero RCU, consistent point-in-time
aws dynamodb export-table-to-point-in-time \
  --table-arn arn:aws:dynamodb:us-east-1:123456789012:table/orders \
  --s3-bucket lake --s3-prefix orders/full/ --export-format DYNAMODB_JSON

# 3. Daily INCREMENTAL export — only yesterday's changes, with old+new images
aws dynamodb export-table-to-point-in-time \
  --table-arn arn:aws:dynamodb:us-east-1:123456789012:table/orders \
  --s3-bucket lake --s3-prefix orders/incr/ --export-type INCREMENTAL_EXPORT \
  --incremental-export-specification \
     ExportFromTime=2026-09-04T00:00:00Z,ExportToTime=2026-09-05T00:00:00Z,ExportViewType=NEW_AND_OLD_IMAGES \
  --export-format DYNAMODB_JSON

-- 4. Redshift Spectrum external tables over the S3 exports, then MERGE the delta in.
--    Flatten typed DynamoDB JSON on the way in.
INSERT INTO staging.orders_incr
SELECT
    json_extract_path_text(item, 'order_id', 'S')            AS order_id,
    json_extract_path_text(item, 'status', 'S')              AS status,
    CAST(json_extract_path_text(item, 'total_cents', 'N') AS BIGINT) AS total_cents,
    metadata_op                                              AS op   -- INSERT/MODIFY/REMOVE
FROM spectrum.orders_incr_raw;

MERGE INTO analytics.orders AS tgt
USING staging.orders_incr AS src ON tgt.order_id = src.order_id
WHEN MATCHED AND src.op = 'REMOVE' THEN DELETE
WHEN MATCHED THEN UPDATE SET status = src.status, total_cents = src.total_cents
WHEN NOT MATCHED THEN INSERT (order_id, status, total_cents)
     VALUES (src.order_id, src.status, src.total_cents);

Step-by-step trace.

Step	Mechanism	Property
PITR	continuous backup	enables export; zero RCU
Full export	point-in-time snapshot	5 TB, no production impact
Glue / Spectrum	external tables over S3	query without loading first
Flatten	`json_extract` + `CAST`	typed DynamoDB JSON → columns
Daily incremental	changed-items export	cheap ongoing sync
MERGE by `order_id`	keyed upsert/delete	idempotent, handles deletes

Walking it: the 5 TB initial load is a point-in-time export read from PITR, so production sees zero read load and no Scan; Spectrum queries the S3 files directly; a nightly incremental export ships only the day's changes with old-and-new images; and a keyed MERGE applies inserts, updates, and deletes idempotently. Freshness lands within a day, and the production table is never scanned.

Output:

Metric	Scan approach (rejected)	Export approach
Production impact	heavy RCU, throttling risk	zero RCU
Initial 5 TB load	days, at capacity cost	one export, no impact
Ongoing sync	re-scan	daily incremental delta
Delete handling	invisible without extra work	`REMOVE` rows in delta
Consistency	non-atomic	point-in-time consistent

Why this works — concept by concept:

Point-in-time export — reading from PITR rather than the live table gives a consistent snapshot at zero read capacity, which is why a 5 TB export never throttles production, unlike a Scan.
Full then incremental — one full export seeds the warehouse; scheduled incremental exports ship only changed items, making ongoing sync cheap and keeping freshness within the export cadence.
NEW_AND_OLD_IMAGES on incrementals — including both images lets the downstream MERGE distinguish updates from deletes, so cancellations propagate instead of silently lingering.
Keyed MERGE — upserting/deleting by order_id makes each load idempotent, so a retried export window converges to the same warehouse state.
Cost — export cost scales with data exported (full once, deltas thereafter), all at zero live RCU, versus a Scan that costs O(table) RCU every run and risks throttling. Analytics run on S3/Redshift, never on the operational table.

Data processing
Topic — data-processing
Warehouse-load and batch-CDC problems

Practice →

Indexing
Topic — indexing
Partitioning and file-layout problems

Practice →

Cheat sheet — DynamoDB for data engineers recipes

Partition-key spreading rule. Choose the partition key for high cardinality and even access distribution first, correctness of naming second. Reject low-cardinality attributes (status, region, type) as partition keys — they hot-spot. If a single key value can attract a large fraction of traffic, shard it with a suffix (device_id#yyyy-mm, OPEN#<n>). Sort key exists to make your most common range read a native Query with begins_with / between.
Capacity math. RCU = ceil(item_kb / 4) × reads/sec, halved for eventual consistency; WCU = ceil(item_kb / 1) × writes/sec. On-demand (PAY_PER_REQUEST) is the 2026 default for spiky/new workloads; provisioned + auto-scaling wins only at steady, predictable, high volume. Keep items lean — a 40 KB item costs 10× the read units of a 4 KB one.
400 KB item limit. Never inline unbounded text/binary. Offload the blob to S3, keep s3_key + byte_size in the item. Small items are cheaper to read, faster to stream, and delay the 10 GB item-collection split.
Single-table workflow. List every access pattern before the schema; map each to a Query/GetItem (a pattern that needs a Scan is a design bug). Use generic keys PK/SK with typed value prefixes (USER#123, ORDER#456), a type attribute on every item, and item collections so one Query returns a whole aggregate.
Adjacency-list relationships. Model edges as items carrying both the base key (PK=ORDER#456, SK=ITEM#sku-9) and GSI keys (GSI1PK=SKU#sku-9, GSI1SK=ORDER#456). One write maintains both traversal directions; the adjacency list is the join.
GSI vs LSI. GSI: any keys, add anytime, own throughput, eventual-only, up to 20/table — the default. LSI: same partition key + different sort key, must be created at table creation, strongly-consistent, shares throughput, 10 GB collection cap — rare. Prefer a GSI unless you specifically need strong consistency on an alternate sort key.
GSI projections. KEYS_ONLY for routing indexes, INCLUDE [attrs] for list views (project exactly what you read), ALL only when the GSI must serve reads without touching the base table. Every projected attribute adds write amplification.
Sparse GSI. Write the GSI key only while an item is in the target state and REMOVE it when the state ends, so "find the few items in state X" is O(matching) not O(table). Watch for a hot GSI partition when the key is a constant — shard it.
Streams view types. KEYS_ONLY = cache-bust, NEW_IMAGE = state replication, OLD_IMAGE = delete/delta capture, NEW_AND_OLD_IMAGES = CDC/aggregation default. Ordering is strict per partition key (per shard), not global. 24-hour retention.
Stream consumer contract. Delivery is at-least-once → consumers must be idempotent. Use atomic ADD counters guarded by a monotonic last_seq/SequenceNumber, upsert/delete by natural id for replication, bisect-on-error + a DLQ for poison records. Reach for Kinesis Data Streams when you need >24 h retention, replay, or multiple independent consumers.
S3 export basics. Enable PITR (prerequisite), then export-table-to-point-in-time reads from continuous backups at zero RCU and never Scans. Output is gzipped DynamoDB JSON / Ion, partitioned with a manifest. Seed with a full export; keep fresh with INCREMENTAL_EXPORT windows using NEW_AND_OLD_IMAGES so deletes propagate.
Export → warehouse. Catalog the S3 export with Glue; query via Athena / Redshift Spectrum / Spark; flatten typed DynamoDB JSON with json_extract + CAST. Apply incremental deltas with a keyed MERGE (REMOVE → DELETE, matched → UPDATE, unmatched → INSERT) so loads are idempotent. Never Scan a production table for analytics — export it.

Frequently asked questions

What is DynamoDB in one sentence for a data engineer?

DynamoDB is a fully managed, serverless key-value and document database that delivers single-digit-millisecond performance at any scale by hashing items onto physical partitions via a partition key and refusing the relational conveniences — no joins, no ad-hoc WHERE, no add-an-index-later — that would break that performance guarantee. For a data engineer it is usually a source: a table you model around fixed access patterns, tap for change data via DynamoDB Streams, and drain into a warehouse via S3 export. The whole discipline is designing keys and indexes up front so every important query is an O(result-size) Query, never an O(table) Scan.

Partition key vs sort key — what is the difference?

The partition key (hash key) determines which physical partition an item lives on; DynamoDB hashes its value to spread items across the storage fleet, and you must supply its exact value to read or Query efficiently. The sort key (range key) is optional and determines the order of items within a partition — items sharing a partition key are stored sorted by sort key, which is what makes range conditions like begins_with, between, and > cheap. Together (partition key, sort key) form a composite primary key that must be unique. Rule of thumb: pick the partition key for even distribution and high cardinality; design the sort key so your most common "give me a range of related items" access pattern is a single native Query.

What is single-table design and why is it recommended?

Single-table design stores multiple entity types — users, orders, line items — in one DynamoDB table using generic key attributes (PK, SK) whose values carry entity-type prefixes (USER#123, ORDER#456), so related items share a partition and can be fetched together in a single Query. It is recommended because DynamoDB has no joins: co-locating related items in one item collection lets you retrieve an entire aggregate (a user plus their orders plus their addresses) in one round trip instead of N queries across N tables. The catch is that you must enumerate every access pattern before creating the table and denormalize deliberately, maintaining consistency across duplicated data at write time. Done right, it delivers complex reads at single-digit-millisecond latency; done wrong (entity-first, like a relational schema) it forces expensive Scans.

GSI vs LSI — when do I use each?

Use a GSI (Global Secondary Index) for almost everything: it can be keyed on any attributes, added at any time, has its own throughput, and lets you query the same data along a partition key the base table doesn't support — at the cost of being eventually consistent only. Use an LSI (Local Secondary Index) only when you need strongly-consistent reads on an alternate sort key within the same partition key, and only if you can create it at table-creation time (LSIs cannot be added later), accepting that it shares the base table's throughput and imposes a 10 GB item-collection limit. In 2026, the honest default is "reach for a GSI"; LSIs are a niche tool for the specific strong-consistency-on-a-second-sort-key case.

How do DynamoDB Streams enable CDC?

DynamoDB Streams publishes an ordered, 24-hour-retained log of every item-level change (INSERT, MODIFY, REMOVE), sharded so that all changes for a given partition key are delivered in order, with each record optionally carrying the new and/or old image of the item. That is textbook change data capture: a consumer (a Lambda trigger or a Kinesis Client Library app) reads the change records and fans them out to search indexes, materialized aggregate tables, caches, or the analytics warehouse — all without polling or scanning the source table. Because delivery is at-least-once, consumers must be idempotent (atomic counters guarded by SequenceNumber, upsert/delete by natural id). When you need more than 24 hours of retention, replay, or multiple independent consumers, route the stream through Kinesis Data Streams for DynamoDB, which offers up to 365 days retention and enhanced fan-out.

How do I get DynamoDB data into a warehouse?

Two complementary paths. For batch analytics, use S3 export: enable PITR, then export-table-to-point-in-time writes a consistent point-in-time snapshot to S3 as DynamoDB JSON at zero read capacity and without a Scan; catalog it with Glue and query via Athena, Redshift Spectrum, or Spark, flattening the typed JSON with json_extract + CAST. Seed with one full export, then keep the warehouse fresh with scheduled incremental exports (using NEW_AND_OLD_IMAGES so deletes propagate) applied via a keyed MERGE. For real-time needs, use DynamoDB Streams → Kinesis Firehose → S3/Redshift, delivering changes with seconds-to-minutes latency. The cardinal rule: never Scan a production table for analytics — export it to S3 and query it there, and add the Streams path only when sub-minute freshness is actually required.

Practice on PipeCode

Drill the database practice library → for the key-design, single-table, and query-modeling problems senior interviewers love to hand DynamoDB candidates.
Sharpen your index intuition on the indexing practice library → for GSI vs LSI, sparse indexes, inverted indexes, and projection trade-offs.
Rehearse the pipeline side on the data-processing practice library → for streams, CDC, incremental loads, and DynamoDB-to-warehouse export patterns.
Stack these against PipeCode's broader 450+ data-engineering catalogue to anchor the "model access patterns first, never Scan for analytics" instinct against real graded inputs.

Lock in DynamoDB modeling muscle memory

Docs explain the API. PipeCode drills explain the decision — when to shard a hot partition key, when single-table design beats one-table-per-entity, when a sparse GSI turns an O(table) scan into an O(few) query, when Streams beat polling, and when to export to S3 instead of scanning. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice database problems →
Practice data-processing problems →

Sketches at Scale: Count-Min, t-digest & Approximate Quantiles

Gowtham Potureddi — Tue, 08 Sep 2026 13:41:51 +0000

data sketches are the compact, probabilistic data structures that let you answer "how many times did this key appear?", "what are the top few heaviest keys?", and "what is the 99th-percentile latency?" over a stream so large you can never hold it in memory — and they are the single tool that separates an engineer who says "we'd sample it" from one who says "we'd sketch it with a bounded error." A sketch reads the stream once, keeps a summary that is sublinear in the size of the data (often a few kilobytes for a billion events), and answers queries with a guarantee of the form "the answer is within ε of the truth with probability 1 − δ." The whole discipline lives in that trade: you surrender exactness you almost never needed, and in return you get frequency estimation, approximate quantiles, and heavy hitters at a memory footprint that does not grow with the stream.

This guide is the walkthrough you wished existed the first time an interviewer asked "you have a 10-billion-event-per-day click stream and a 50 MB memory budget — give me the top-100 URLs and the p99 request latency, and tell me your error." It opens the two workhorse structures layer by layer: the Count-Min sketch that turns a hash grid into a one-sided frequency estimator, and the t-digest that clusters a distribution into centroids so percentiles at the tail stay accurate. It then makes the error math concrete — how ε and δ set the width and depth of a sketch, why additive error is wonderful for heavy hitters and useless for rare keys — and closes on how mergeable summaries power real systems: Spark aggregations, Druid rollups, and percentile monitoring. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the data-structures practice library →, sharpen your estimation intuition on the statistics practice library →, and stress the distinct-count axis on the cardinality practice library →.

On this page

Why sketches — streaming, sublinear, mergeable
Count-Min sketch — frequency & heavy hitters
t-digest & approximate quantiles
Error bounds & sizing
Sketches in production — Spark / Druid / monitoring
Cheat sheet — data-sketch recipes
Frequently asked questions
Practice on PipeCode

1. Why sketches — streaming, sublinear, mergeable

A data sketch trades exactness you rarely need for memory you always lack — one pass, sublinear space, and a merge operator

The one-sentence invariant: a data sketch is a compact summary of a stream that is built in a single pass, occupies space sublinear in the number of items (or in the size of the key universe), answers a specific class of query — frequency, quantile, cardinality, or membership — with a mathematically bounded error, and is mergeable, meaning the summary of two streams can be computed from their two summaries alone without re-reading either. Every other idea in this article is a consequence of those four words: single-pass, sublinear, bounded-error, mergeable. The moment a workload has more distinct keys than fit in RAM, or is partitioned across hundreds of machines, exact aggregation stops being free and the sketch becomes the correct default rather than a clever optimisation.

The three properties that define a sketch.

Streaming (single-pass). The sketch sees each item exactly once, in arrival order, and updates its state in O(1) or O(log) time. It cannot go back and re-read the stream. This rules out any algorithm that needs to sort or make a second pass — which is exactly the class of algorithm exact quantiles and exact top-K belong to.
Sublinear space. A hashmap that counts every distinct key is O(distinct keys); a sorted array for exact percentiles is O(N). A sketch is deliberately smaller than the data it summarises — a Count-Min sketch is a fixed grid whose size depends only on the error target, and a t-digest is a bounded list of centroids. Feed it a billion events and it stays kilobytes.
Mergeability. Two sketches of the same type and configuration combine — by adding counter grids, or concatenating and re-clustering centroids — into a sketch that is (exactly, or within the same error bound) the sketch of the concatenated stream. Merge is associative and usually commutative, which is precisely the contract a distributed reducer needs. This is why sketches, not exact aggregates, are what Spark and Druid ship across the wire.

The axes that matter — pick the sketch by the question.

What question are you asking? Frequency of a given key → Count-Min. Top-K heaviest keys → Count-Min + a heap (or the Space-Saving / Frequent-Items sketch). Quantiles / percentiles → t-digest, KLL, or Greenwald-Khanna. Number of distinct keys → HyperLogLog. Set membership → Bloom filter. Using the wrong family is the most common interview miss — HyperLogLog cannot tell you a frequency, and Count-Min cannot tell you a distinct count.
What error type can you tolerate? Additive error (± ε·N) is a fixed slice of the total, so it swamps rare keys but is negligible for heavy ones. Relative error (± ε·true) scales with the answer and is what you want for cardinality and for tail quantiles. The error type is a property of the sketch, not a knob you turn afterwards.
Do you need to merge? If the data is partitioned (it almost always is), you need a mergeable sketch. A client-side percentile summary that is not mergeable (the Prometheus summary type is the canonical trap) cannot be aggregated across instances, which quietly makes fleet-wide p99 impossible.
What is the space budget? Sketch size is chosen from the error target, and the relationship is usually space ∝ 1/ε. Halving the error roughly doubles the memory. Senior answers state the budget first and derive ε and δ from it, not the reverse.

The 2026 reality — sketches are the default aggregation primitive, not a niche trick.

Distributed engines ship them built in. Spark has approx_count_distinct (HyperLogLog) and percentile_approx; Apache DataSketches provides KLL quantiles, Theta cardinality, and Frequent-Items, all mergeable and all usable as Spark/Hive UDAFs.
OLAP stores pre-aggregate with them. Druid computes quantile, cardinality, and frequency sketches at ingestion time and stores the sketch object in the segment, so a query merges pre-built sketches instead of scanning raw rows.
Metrics backends are built on them. DDSketch (Datadog) and t-digest (many Prometheus-adjacent systems) exist specifically because you must merge percentile summaries across thousands of hosts and time buckets. A monitoring system that stores raw samples to compute exact percentiles does not scale; one that stores mergeable sketches does.

What interviewers listen for.

Do you name the sketch by the question — Count-Min for frequency, t-digest for quantiles, HyperLogLog for cardinality — without conflating them? — senior signal.
Do you say "mergeable" unprompted when the data is partitioned? — required answer.
Do you distinguish additive from relative error and tie it to whether the workload cares about rare keys or the tail? — senior signal.
Do you derive sketch size from the error budget rather than guessing a memory number? — senior signal.
Do you describe a sketch as "a summary with a bounded error" rather than as vague "sampling"? — required answer.

Worked example — the exact-vs-sketch memory blow-up

Detailed explanation. The fastest way to make the case for sketches is to price the exact alternative on a realistic stream and watch it fail the memory budget. Take a day of request logs: 10 billion events, 200 million distinct URLs, and a numeric latency per event. We want two answers — per-URL request counts and the p99 latency — and we have a 50 MB budget on a single aggregator.

The stream. 10^10 events; each carries a url (one of ~2×10^8 distinct) and a latency_ms (a double).
Exact frequency. A dict[url, int] holds one entry per distinct URL. At ~60 bytes per Python entry (string key + int + hash overhead) that is 2×10^8 × 60 ≈ 12 GB — 240× over budget.
Exact quantile. To get an exact p99 you must retain every latency (or at least every value in the tail you cannot yet rule out). 10^10 doubles is 80 GB — 1600× over budget.

Question. Compare the exact and sketched memory footprints for the frequency and quantile questions, and state the error each sketch gives.

Input.

Question	Exact structure	Exact memory	Sketch	Sketch memory
Per-URL counts	hashmap (200M keys)	~12 GB	Count-Min (ε=1e-3, δ=1e-2)	~55 KB
p99 latency	full sample (10B doubles)	~80 GB	t-digest (compression=200)	~20 KB
Distinct URLs	hash set (200M keys)	~9 GB	HyperLogLog (p=14)	~16 KB

Code.

# The exact approach that does NOT fit in 50 MB
exact_counts = {}          # -> ~12 GB at 200M distinct URLs
exact_latencies = []       # -> ~80 GB at 10B events

for event in stream:                     # 10,000,000,000 iterations
    exact_counts[event.url] = exact_counts.get(event.url, 0) + 1
    exact_latencies.append(event.latency_ms)

# p99 needs a sort or a select over the whole 80 GB array
exact_latencies.sort()
p99_exact = exact_latencies[int(0.99 * len(exact_latencies))]

# The sketched approach that fits in ~100 KB total
cms      = CountMinSketch(epsilon=1e-3, delta=1e-2)   # ~55 KB grid
tdigest  = TDigest(compression=200)                    # ~20 KB of centroids

for event in stream:                     # still one pass, O(1) per item
    cms.update(event.url, 1)
    tdigest.update(event.latency_ms)

count_of_url = cms.estimate("/checkout")   # <= true + eps*N, w.p. 1 - delta
p99_sketch   = tdigest.quantile(0.99)      # relative error, tightest at the tail

Step-by-step explanation.

The exact hashmap grows with the number of distinct keys, not the number of events — but at 200 million distinct URLs it is already 12 GB. Distinct-key growth is the failure mode: a stream with a large key universe defeats any per-key exact structure regardless of how many total events there are.
The exact quantile is worse: percentiles are a global property of the value distribution, so there is no incremental exact summary — you must keep enough of the sample to locate the rank you care about, which in the worst case is all of it. 80 GB is not a tuning problem; it is the algorithm.
The Count-Min sketch replaces the hashmap with a fixed grid sized purely from the error target (ε, δ). Its memory is independent of both the event count and the distinct-key count — the same 55 KB whether the stream has a thousand keys or a billion.
The t-digest replaces the full sample with a bounded set of centroids. It never stores an individual latency; it folds each value into the nearest centroid and occasionally re-clusters. Memory is capped by the compression parameter, not by N.
The cost is error, and it is bounded and stated: the Count-Min answer is the truth plus at most ε·N (a one-sided overshoot) with probability 1 − δ; the t-digest p99 carries a small relative error that is tightest exactly at the tail we asked about. Both errors are acceptable for the questions ("which URLs are hot", "is p99 breaching SLO") that motivated them.

Output.

Metric	Exact	Sketched	Ratio
Frequency memory	~12 GB	~55 KB	~218,000× smaller
Quantile memory	~80 GB	~20 KB	~4,000,000× smaller
Passes over data	1 (+ sort for p99)	1	—
Answer type	exact	ε-bounded	—
Fits in 50 MB budget?	no	yes	—

Rule of thumb. Reach for a sketch the moment either the distinct-key count or the raw sample would not fit in your aggregation budget. The decision is not "is approximation acceptable" — it is "can I afford exactness", and at stream scale the answer is almost always no.

Worked example — mergeability, demonstrated

Detailed explanation. Mergeability is the property that makes sketches distributed-native, so it is worth seeing concretely. Split a stream across three shards, build a sketch per shard independently, then combine the three sketches and confirm the result equals the sketch of the whole stream. For a Count-Min sketch "combine" is literally element-wise addition of the grids — provided every shard used the same width, depth, and hash seeds.

The stream. Keys A, B, C arriving on three shards; shard 1 sees A A B, shard 2 sees A C, shard 3 sees B B C C.
The contract. merge(sketch(S1), sketch(S2), sketch(S3)) == sketch(S1 ++ S2 ++ S3) for Count-Min (exactly), and within the error bound for quantile sketches.
The requirement. Identical configuration across shards — same (w, d) and same hash functions. A merge across mismatched configs is meaningless.

Question. Build one Count-Min sketch per shard, merge them, and show the merged frequency estimates match a single global sketch.

Input.

Shard	Stream	Local count A	Local count B	Local count C
S1	A A B	2	1	0
S2	A C	1	0	1
S3	B B C C	0	2	2
Merged	—	3	3	3

Code.

def merge_cms(a: "CountMinSketch", b: "CountMinSketch") -> "CountMinSketch":
    """Merge two Count-Min sketches by adding their grids element-wise.
    Precondition: identical width, depth, and hash seeds."""
    assert (a.width, a.depth, a.seeds) == (b.width, b.depth, b.seeds)
    out = CountMinSketch.like(a)                 # zero grid, same config
    for r in range(a.depth):
        for c in range(a.width):
            out.grid[r][c] = a.grid[r][c] + b.grid[r][c]
    return out

# One sketch per shard, built independently (in parallel, on different machines)
s1 = build_cms(["A", "A", "B"])
s2 = build_cms(["A", "C"])
s3 = build_cms(["B", "B", "C", "C"])

merged = merge_cms(merge_cms(s1, s2), s3)        # associative: any order works

# A single global sketch over the concatenated stream, for comparison
global_cms = build_cms(["A", "A", "B", "A", "C", "B", "B", "C", "C"])

for key in ("A", "B", "C"):
    assert merged.estimate(key) == global_cms.estimate(key)

Step-by-step explanation.

Each shard builds its sketch with no coordination — no shuffle, no cross-talk, no shared state. That independence is the entire point: the expensive part (reading the raw data) happens locally and in parallel, and only the tiny sketch crosses the network.
merge_cms adds the grids cell by cell. Because a Count-Min counter at grid[r][c] is just "the total weight hashed into row r, column c", summing two grids yields exactly the grid you would have gotten had both streams updated one shared sketch. This is why Count-Min merge is exact, not merely bounded.
The merge is associative and commutative, so a reducer can combine shards in any order, in a tree, or incrementally as they arrive. There is no "correct order" to preserve — a property distributed schedulers depend on.
The precondition is strict: identical width, depth, and hash seeds. Two sketches with different seeds place the same key in different columns, so adding their grids mixes unrelated counters and produces garbage. In practice the config is fixed centrally and shipped to every shard.
Quantile sketches (t-digest, KLL) are mergeable too, but their merge is "concatenate the centroid/level structure and re-compress", and the result is within the error bound rather than bit-identical. The distributed pattern is the same; only the exactness of the merge differs.

Output.

Key	S1 est	S2 est	S3 est	Merged est	Global est	Match
A	2	1	0	3	3	yes
B	1	0	2	3	3	yes
C	0	1	2	3	3	yes

Rule of thumb. Treat mergeability as a hard requirement whenever data is partitioned. Fix the sketch configuration centrally, ship it to every worker, and let the reducer add or re-compress — never try to merge sketches that were built with different parameters.

Worked example — choosing the sketch for the question

Detailed explanation. The most common sketch mistake is reaching for the wrong family, so senior engineers keep a one-line mapping from question to structure. Walk three real questions through the mapping: "how hot is this key", "how many distinct users", and "what is the tail latency" — three different sketches, three different error models.

Frequency / heavy hitters. "How many times did key X appear?" and "what are the top-K keys?" → Count-Min (+ heap) or a Frequent-Items sketch. Additive error.
Cardinality. "How many distinct keys?" → HyperLogLog. Relative error ≈ 1.04/√m.
Quantiles. "What is p50 / p95 / p99?" → t-digest, KLL, or Greenwald-Khanna. Relative (t-digest, tail-tight) or rank (KLL) error.

Question. For each of three questions, name the sketch, its error model, and the disqualifying mismatch if you picked the wrong one.

Input.

Question	Right sketch	Error model	Wrong pick and why it fails
Count of key X	Count-Min	additive ± ε·N	HyperLogLog — has no per-key counter at all
Distinct keys	HyperLogLog	relative ± 1.04/√m	Count-Min — counts occurrences, not distinctness
p99 latency	t-digest / KLL	relative / rank	Count-Min — has no notion of order or rank

Code.

def pick_sketch(question: str) -> str:
    """Map an aggregation question to the correct sketch family."""
    q = question.lower()
    if "distinct" in q or "unique" in q or "cardinality" in q:
        return "HyperLogLog (relative error ~1.04/sqrt(m))"
    if "percentile" in q or "quantile" in q or "median" in q or "p99" in q:
        return "t-digest / KLL (quantile sketch)"
    if "top" in q or "heavy" in q or "most frequent" in q:
        return "Count-Min + min-heap (heavy hitters)"
    if "how many times" in q or "frequency" in q or "count of" in q:
        return "Count-Min (frequency, additive error)"
    if "member" in q or "seen before" in q or "exists" in q:
        return "Bloom filter (membership, one-sided false positives)"
    return "exact aggregation may be fine; measure the key universe first"

print(pick_sketch("count of key /checkout"))   # Count-Min (frequency, additive error)
print(pick_sketch("how many distinct users"))  # HyperLogLog
print(pick_sketch("what is the p99 latency"))  # t-digest / KLL

Step-by-step explanation.

The frequency question maps to Count-Min because a Count-Min grid is a set of overlapping counters — it can return an estimate for any key you ask about. HyperLogLog stores only a register array that estimates distinctness; it has no counter to return, so it cannot answer "how many times".
The distinct-count question maps to HyperLogLog because distinctness is about the set of keys, not their multiplicity. Count-Min would happily give you a frequency, but summing frequencies is not a distinct count — a Count-Min cannot deduplicate.
The quantile question maps to a quantile sketch because percentiles depend on the order statistics of the values. Count-Min discards order entirely (it hashes keys into buckets), so it has no way to answer "what value sits at rank 0.99N".
Each wrong pick fails for a structural reason, not a tuning reason — you cannot make Count-Min answer a quantile by enlarging the grid, and you cannot make HyperLogLog answer a frequency by adding registers. The family is chosen by the question; only then do you size within the family.
The mapping also surfaces membership (Bloom filter) and the honest fallback ("measure the key universe; exact might fit"). Naming that fallback is itself a senior signal — sketches are for when exact does not fit, not a reflex.

Output.

Question asked	Sketch chosen	Error model	Space driver
Frequency of a key	Count-Min	additive ± ε·N	1/ε
Top-K heavy hitters	Count-Min + heap	additive ± ε·N	1/ε + K
Distinct count	HyperLogLog	relative ± 1.04/√m	number of registers m
Quantile / percentile	t-digest / KLL	relative / rank	compression / k
Membership	Bloom filter	one-sided FP rate	1/target FP

Rule of thumb. Say the question out loud, then name the sketch: frequency → Count-Min, distinct → HyperLogLog, quantile → t-digest/KLL, membership → Bloom. Sizing comes second; picking the family from the question comes first, and getting the family wrong cannot be fixed by more memory.

Data-engineering interview question on sketch selection

A senior interviewer often opens with: "You are handed a 10-billion-event-per-day stream partitioned across 400 workers, and product wants three things on a dashboard: the top-100 heaviest keys, the number of distinct users, and the p99 event latency — refreshed hourly, sliceable by region, on a fixed memory budget per worker. Walk me through the sketches you'd build per worker, how you'd combine them, and the error you'd quote for each number."

Solution Using a per-worker mergeable sketch bundle combined at the reducer

# sketch_bundle.py — one bundle per worker; all three sketches are mergeable
from dataclasses import dataclass

@dataclass
class SketchBundle:
    cms:  "CountMinSketch"   # frequency  -> top-K via companion heap
    hll:  "HyperLogLog"      # distinct users
    td:   "TDigest"          # p99 latency
    heap: "TopKHeap"         # top-100 candidates, fed by cms estimates

    @classmethod
    def empty(cls, region: str) -> "SketchBundle":
        return cls(
            cms=CountMinSketch(epsilon=1e-4, delta=1e-3),  # tight: top-100 need small eps
            hll=HyperLogLog(precision=14),                  # ~16 KB, ~0.8% relative error
            td=TDigest(compression=200),                    # tail-accurate p99
            heap=TopKHeap(k=100),
        )

    def update(self, event) -> None:
        c = self.cms.update(event.key, 1)   # returns the post-update estimate
        self.heap.offer(event.key, c)       # maintain top-100 candidates
        self.hll.add(event.user_id)
        self.td.update(event.latency_ms)

def merge(a: SketchBundle, b: SketchBundle) -> SketchBundle:
    """Associative merge — the reducer folds all 400 worker bundles."""
    out = SketchBundle.empty("merged")
    out.cms  = a.cms.merge(b.cms)           # add grids (exact)
    out.hll  = a.hll.merge(b.hll)           # max of registers (exact for HLL)
    out.td   = a.td.merge(b.td)             # concat centroids + re-compress
    out.heap = a.heap.merge(b.heap, out.cms)  # re-rank union by merged cms estimate
    return out

# Reducer — tree-combine 400 worker bundles per region, then answer the dashboard
def summarize(worker_bundles: list[SketchBundle]) -> dict:
    from functools import reduce
    combined = reduce(merge, worker_bundles)         # O(workers) merges
    return {
        "top_100":        combined.heap.items(),      # heavy hitters
        "distinct_users": combined.hll.estimate(),    # cardinality
        "p99_latency_ms": combined.td.quantile(0.99), # tail percentile
    }

Step-by-step trace.

Step	Input	What happens
Per-worker build	worker reads its shard once	updates cms / hll / td / heap in O(1) per event
Emit	end of hour	worker ships ~100 KB bundle, not raw events
Reducer fold	400 bundles	`reduce(merge, ...)` — associative, any order
top-100	merged cms + heap	re-rank union of candidates by merged estimate
distinct	merged hll	register-wise max, then estimate
p99	merged td	walk centroids, interpolate at q=0.99

After the fold, the reducer holds one bundle per region of roughly 100 KB. The dashboard reads three numbers off it — top-100 keys, distinct users, p99 latency — each with a stated error, and never touched a raw event after the initial per-worker pass. Slicing by region is just choosing which bundles to merge.

Output:

Dashboard number	Sketch used	Error quoted	Bytes on the wire
Top-100 heavy keys	Count-Min + heap	± ε·N, ε = 1e-4	~1 MB grid
Distinct users	HyperLogLog p=14	~0.8% relative	~16 KB
p99 latency	t-digest c=200	small relative, tail-tight	~20 KB
Raw events shipped	—	—	0

Why this works — concept by concept:

Single-pass per worker — each worker touches its shard once and updates four O(1) structures. The expensive read is local and parallel; nothing re-reads the stream, which is the streaming constraint honoured.
One family per question — Count-Min for frequency, HyperLogLog for distinctness, t-digest for the quantile. Each number comes from the structure whose error model fits it; no structure is asked a question it cannot answer.
Mergeability is the reducer contract — every sketch in the bundle has an associative merge (add grids, max registers, re-compress centroids). That is what lets 400 independent bundles collapse to one without a shuffle of raw data.
Slice-by-region for free — because merge is associative, "p99 for region X" is just merging the bundles from region X. The same pre-built sketches answer any slice that is a union of shards.
Cost — O(1) per event on each worker, O(workers) merges at the reducer, and O(sketch size) bytes on the wire — independent of the 10^10 event count. The eliminated cost is the O(N) shuffle of raw events and the O(distinct) memory of exact per-key state.

Data Structures
Topic — data-structures
Probabilistic data-structure problems

Practice →

Statistics Topic — statistics Estimation and error-bound problems

Practice →

2. Count-Min sketch — frequency & heavy hitters

`count-min sketch` hashes every item into a d×w counter grid and answers a frequency query with the minimum across rows — a one-sided overestimate that never lies low

The mental model in one line: a count-min sketch is a two-dimensional array of counters with d rows and w columns paired with d independent hash functions; to add weight to a key you hash it once per row and increment the selected counter in each of the d rows, and to estimate a key's frequency you read the d counters it hashes to and return the minimum — because collisions can only ever add extra weight, the estimate is guaranteed to be at least the true count and at most the true count plus ε·N with probability 1 − δ. It is the workhorse of frequency estimation: constant memory regardless of the key universe, O(d) update and query, exactly mergeable, and the natural base for heavy hitters.

Why the minimum, and why it is one-sided.

Every row is an over-count. Within a single row, many keys hash to the same column, so a counter holds the true count of your key plus the counts of every other key that collided there. A single row therefore never underestimates — it can only overestimate.
Different rows collide differently. The d hash functions are independent, so the set of keys colliding with yours in row 1 differs from row 2. The row where your key suffered the least collision noise gives the tightest over-estimate.
Min beats the noise down. Taking the minimum across rows picks that least-noisy row. The more rows d, the higher the probability that at least one row was nearly collision-free — which is exactly the δ (failure probability) knob.
The error is additive, not relative. The over-count is bounded by ε·N (a fixed slice of the total stream weight), independent of the key's own frequency. This is the defining strength and weakness of Count-Min, and section 4 makes the math precise.

The grid dimensions and what they control.

Width w controls ε. More columns spread keys across more counters, reducing collisions and shrinking the additive error. The relationship is w = ⌈e/ε⌉ (Euler's e ≈ 2.718), so ε = 0.001 needs w ≈ 2719 columns.
Depth d controls δ. More rows give more independent chances at a low-collision estimate, raising the confidence. The relationship is d = ⌈ln(1/δ)⌉, so δ = 0.01 needs d ≈ 5 rows.
Memory is w × d counters. With 4-byte counters, 2719 × 5 ≈ 54 KB — fixed, whether the stream has a thousand or a billion distinct keys.
Hashing. d independent hash functions (in practice, one strong hash split into d lanes, or pairwise-independent hashes (a·x + b) mod p mod w). Independence across rows is what the error proof assumes.

Heavy hitters on top of Count-Min.

The definition. A φ-heavy hitter is a key whose frequency exceeds φ·N (e.g. any key that is more than 1% of the stream). Top-K is the related "give me the K heaviest keys".
The pattern. Maintain a min-heap of size K keyed by estimated frequency. On each update, query the Count-Min estimate for the key; if it beats the heap's smallest, offer it into the heap. The heap holds the current top-K candidates.
Why it needs the sketch. You cannot keep a counter for every key (that is the exact hashmap you are avoiding), so the heap tracks only K keys and leans on the Count-Min for the frequency of any candidate.
The caveat. Additive error means a key just below the threshold can be over-estimated into the top-K. For true heavy hitters (well above ε·N) this never happens; near the boundary it can, which is why you size ε well below φ.

The conservative-update refinement.

Standard update increments all d counters. Conservative update increments only the counters that are currently at the minimum (i.e. it raises each counter to at most min + 1).
Why it helps. It avoids inflating counters that are already large due to other keys, which lowers the observed over-estimate in practice — often by a large factor on skewed streams.
The trade-off. Conservative update breaks exact mergeability (you can no longer just add two grids), so it is used for single-node accuracy, not distributed merge. Choose one or the other per deployment.

Common interview probes on Count-Min.

"Can a Count-Min sketch under-count?" — required answer: no, it is one-sided; the estimate is ≥ true count.
"Why take the min across rows?" — the least-collided row gives the tightest over-estimate; more rows raise the confidence 1 − δ.
"How do you get top-K from it?" — Count-Min for frequencies plus a size-K min-heap of candidates.
"How do you merge two Count-Min sketches?" — element-wise add the grids (same w, d, seeds); conservative update forfeits this.

Worked example — build and query a Count-Min sketch

Detailed explanation. Implement a minimal Count-Min sketch and run a small word stream through it, then read an estimate and confirm it is a one-sided overestimate. Use d = 4 rows and w = 16 columns with pairwise-independent hashes so the trace is small enough to reason about by hand.

Config. d = 4, w = 16, four hash seeds.
Stream. A short stream of words with one clearly-heavy key (the) and several light keys.
Query. Estimate the and a light word, and compare to the true counts.

Question. Implement update and estimate, run the stream, and show the estimate for a heavy and a light key.

Input.

Parameter	Value
Depth d	4
Width w	16
Hash family	(a·h(x) + b) mod p mod w
Counter type	int (4 bytes)

Code.

import hashlib

class CountMinSketch:
    def __init__(self, width: int = 16, depth: int = 4, seed: int = 1):
        self.width  = width
        self.depth  = depth
        self.seeds  = [seed * 7919 + i for i in range(depth)]  # d distinct seeds
        self.grid   = [[0] * width for _ in range(depth)]

    def _cols(self, key: str):
        for r, s in enumerate(self.seeds):
            h = int(hashlib.md5(f"{s}:{key}".encode()).hexdigest(), 16)
            yield r, h % self.width          # one column per row

    def update(self, key: str, count: int = 1) -> int:
        est = None
        for r, c in self._cols(key):
            self.grid[r][c] += count
            v = self.grid[r][c]
            est = v if est is None else min(est, v)
        return est                            # post-update estimate (handy for top-K)

    def estimate(self, key: str) -> int:
        return min(self.grid[r][c] for r, c in self._cols(key))

cms = CountMinSketch(width=16, depth=4)
stream = ["the"] * 50 + ["fox", "the", "dog", "the", "fox", "cat"] + ["the"] * 5
for w in stream:
    cms.update(w)

print("the:", cms.estimate("the"))   # true = 57
print("fox:", cms.estimate("fox"))   # true = 2
print("cat:", cms.estimate("cat"))   # true = 1

Step-by-step explanation.

_cols derives one column per row by hashing seed:key. Using a different seed per row gives the d independent hash functions the error bound assumes; a single hash reused across rows would collapse the confidence gain from depth.
update increments the selected counter in each of the d rows. Because a key always lands in the same d columns, repeated updates accumulate there — that is how the sketch "remembers" a key without storing the key itself.
estimate reads those same d columns and returns the minimum. The minimum is the row where the shared its column with the fewest other words, so it is the closest over-estimate to the truth.
The heavy key the (true 57) comes back at exactly 57 or a hair above — with w = 16 and a small stream, collisions are mild, so the over-count is tiny. The light keys fox (2) and cat (1) may over-count more in relative terms because any collision with the adds a big chunk of weight — the additive-error asymmetry in miniature.
Crucially, no estimate is ever below the truth. If you see 57 for a true 57, or 60 for a true 57, both are valid Count-Min outputs; 55 for a true 57 is impossible and would signal a bug.

Output.

Key	True count	CMS estimate	Error	Direction
the	57	57	0	exact here
fox	2	2–4	≤ +2	over
dog	1	1–3	≤ +2	over
cat	1	1–3	≤ +2	over

Rule of thumb. Trust a Count-Min estimate as an upper bound on the truth. It is tight for heavy keys and loose for light ones — which is exactly the right shape when you only care about the heavy keys.

Worked example — top-K heavy hitters with a min-heap

Detailed explanation. Turn the frequency sketch into a top-K heavy-hitters detector by pairing it with a size-K min-heap. As each item streams in, update the Count-Min, take the returned estimate, and maintain a heap of the K keys with the largest estimates. Run it over a skewed stream and read off the top 3.

Structures. One Count-Min sketch + a min-heap of (estimate, key) capped at K, plus a set of keys currently in the heap.
Invariant. The heap holds the K keys with the highest estimates seen so far; the heap root is the weakest of the current top-K.
Update rule. If a key is already in the heap, refresh its estimate; else if the heap is under K, push; else if the estimate beats the root, evict the root and push.

Question. Implement the top-K heavy-hitters detector over a skewed stream and return the top 3 keys with estimates.

Input.

Parameter	Value
K	3
CMS	w=64, d=4
Stream skew	one dominant key, a mid key, a long tail

Code.

import heapq

class HeavyHitters:
    def __init__(self, k: int, cms: "CountMinSketch"):
        self.k    = k
        self.cms  = cms
        self.heap = []          # min-heap of (estimate, key)
        self.pos  = {}          # key -> current estimate in heap

    def offer(self, key: str) -> None:
        est = self.cms.update(key)          # increment + get post-update estimate
        if key in self.pos:                  # already tracked: refresh its rank
            self._refresh(key, est)
        elif len(self.heap) < self.k:        # room in the heap: add it
            heapq.heappush(self.heap, (est, key))
            self.pos[key] = est
        elif est > self.heap[0][0]:          # beats the weakest top-K key: swap in
            _, evicted = heapq.heappop(self.heap)
            self.pos.pop(evicted, None)
            heapq.heappush(self.heap, (est, key))
            self.pos[key] = est

    def _refresh(self, key: str, est: int) -> None:
        # lazy refresh: rebuild the heap with the new estimate for `key`
        self.heap = [(e if k != key else est, k) for e, k in self.heap]
        heapq.heapify(self.heap)
        self.pos[key] = est

    def top(self):
        return sorted(self.heap, reverse=True)   # highest estimate first

stream = (["/home"] * 500 + ["/checkout"] * 120 + ["/search"] * 60
          + [f"/p/{i}" for i in range(400)])     # 400 unique long-tail pages
hh = HeavyHitters(k=3, cms=CountMinSketch(width=64, depth=4))
for path in stream:
    hh.offer(path)

for est, key in hh.top():
    print(f"{key}: ~{est}")

Step-by-step explanation.

offer first updates the Count-Min and captures the post-update estimate in one call — the sketch is the only place a per-key frequency lives, since keeping an exact counter per key is the very thing we are avoiding.
If the key already sits in the heap, _refresh updates its stored estimate and re-heapifies so the ordering stays correct as the key climbs. Without the refresh, a heavy key's heap entry would be frozen at its first-seen estimate and it could be wrongly evicted.
If the heap has fewer than K entries, the key is admitted unconditionally — the top-K is still filling up.
Once the heap is full, a new key must beat the root (the weakest current top-K key) to enter. This is the classic bounded-heap top-K: O(log K) per admission, and the heap never exceeds K entries regardless of how many distinct keys stream by.
The 400 long-tail pages each appear once, estimate ≈ 1, and never displace the root (/search at ~60), so they churn through offer in O(1) without polluting the heap. The result is the three genuine heavy hitters, each with a Count-Min over-estimate that is tight because they are far above ε·N.

Output.

Rank	Key	True count	Heap estimate
1	/home	500	~500
2	/checkout	120	~120
3	/search	60	~60
—	/p/… (×400)	1 each	never enters heap

Rule of thumb. For top-K, let the Count-Min own the frequencies and let a size-K min-heap own the ranking. Keep ε well below your φ threshold so the additive error can never lift a tail key over a genuine heavy hitter.

Worked example — conservative update and grid merge

Detailed explanation. Two refinements matter in practice: conservative update (tighter single-node estimates) and grid merge (distributed aggregation). They are mutually exclusive — conservative update breaks exact mergeability — so this example shows both and states when to use which.

Conservative update. Instead of +1 to all d counters, raise each selected counter to at most min_before + 1. Counters already above the minimum are left alone.
Grid merge. Add two standard (non-conservative) grids element-wise; the sum equals the sketch of the concatenated stream.
The rule. Use conservative update on a single node where accuracy matters and you never merge; use standard update when the sketch must merge across shards.

Question. Implement conservative update and grid merge, and state why they cannot be combined.

Input.

Mode	Update rule	Mergeable?	Best for
Standard	+count to all d cells	yes (add grids)	distributed / merge
Conservative	raise cells to min+count	no	single-node accuracy

Code.

class CountMinSketch(CountMinSketch):   # extend the earlier class

    def update_conservative(self, key: str, count: int = 1) -> int:
        cols = list(self._cols(key))
        cur  = min(self.grid[r][c] for r, c in cols)  # current estimate
        target = cur + count
        for r, c in cols:
            if self.grid[r][c] < target:               # only raise laggards
                self.grid[r][c] = target
        return target

    def merge(self, other: "CountMinSketch") -> "CountMinSketch":
        assert (self.width, self.depth, self.seeds) == (
                other.width, other.depth, other.seeds), "config must match"
        out = CountMinSketch(self.width, self.depth)
        out.seeds = list(self.seeds)
        out.grid  = [[self.grid[r][c] + other.grid[r][c]
                      for c in range(self.width)]
                     for r in range(self.depth)]
        return out

Step-by-step explanation.

update_conservative computes the current minimum (the estimate) first, then raises each of the d counters only up to min + count. A counter already above that target is untouched, so weight from other keys is never amplified by this key's update.
This lowers the observed over-estimate substantially on skewed streams: a light key that collides with a heavy key no longer pushes the heavy key's counters even higher, so the heavy key's estimate stops drifting upward.
merge adds two standard grids cell by cell after asserting identical configuration. Because standard counters are pure sums of hashed weight, the element-wise sum is exactly the grid a single sketch would have built over both streams — merge is lossless.
The two cannot be combined: conservative update makes a counter's value depend on the order and interleaving of updates (it only raises laggards), so two conservatively-built grids no longer sum to the grid of the concatenation. Adding them double-counts the shared minimums and corrupts estimates.
The deployment decision is therefore explicit: a single-node hot-key detector uses conservative update for accuracy; a distributed pipeline that merges per-shard sketches uses standard update and accepts the slightly looser bound in exchange for exact mergeability.

Output.

Scenario	Standard estimate	Conservative estimate
Heavy key, low collision	tight	tight
Heavy key, heavy collision	inflated	much tighter
After cross-shard merge	correct (sum of grids)	invalid (cannot merge)

Rule of thumb. Pick conservative update or mergeability, never both. If your architecture merges per-shard sketches, standard update is mandatory; if it is a single hot-node counter, conservative update buys you real accuracy for free.

Data-structures interview question on Count-Min sketch

A senior interviewer might ask: "Design a service that, over a firehose of (user_id, url) click events on one machine with a 64 MB budget, continuously reports the top-50 URLs and can answer 'how many times has URL X been clicked' with a stated error. Give me the sketch, the top-K machinery, the memory math, and how the error changes if I halve the budget."

Solution Using a Count-Min sketch with a bounded top-K heap

# hot_urls.py — single-node top-K + point frequency over a click firehose
import heapq, hashlib

class CountMinSketch:
    def __init__(self, width: int, depth: int):
        self.width, self.depth = width, depth
        self.seeds = [1_000_003 * i + 17 for i in range(depth)]
        self.grid  = [[0] * width for _ in range(depth)]

    def _cols(self, key: str):
        for r, s in enumerate(self.seeds):
            h = int(hashlib.blake2b(f"{s}:{key}".encode(), digest_size=8).hexdigest(), 16)
            yield r, h % self.width

    def update(self, key: str, c: int = 1) -> int:
        est = None
        for r, col in self._cols(key):
            self.grid[r][col] += c
            est = self.grid[r][col] if est is None else min(est, self.grid[r][col])
        return est

    def estimate(self, key: str) -> int:
        return min(self.grid[r][c] for r, c in self._cols(key))

class HotUrls:
    def __init__(self, k=50, width=1 << 20, depth=5):   # 2^20 cols x 5 rows
        self.cms  = CountMinSketch(width, depth)
        self.k    = k
        self.heap = []            # (estimate, url)
        self.seen = {}            # url -> estimate in heap

    def click(self, url: str) -> None:
        est = self.cms.update(url)
        if url in self.seen:
            self.heap = [(est if u == url else e, u) for e, u in self.heap]
            heapq.heapify(self.heap)
            self.seen[url] = est
        elif len(self.heap) < self.k:
            heapq.heappush(self.heap, (est, url)); self.seen[url] = est
        elif est > self.heap[0][0]:
            _, ev = heapq.heappop(self.heap); self.seen.pop(ev, None)
            heapq.heappush(self.heap, (est, url)); self.seen[url] = est

    def frequency(self, url: str) -> int:
        return self.cms.estimate(url)

    def top(self):
        return sorted(self.heap, reverse=True)

Step-by-step trace.

Step	Value	Reasoning
Grid dimensions	w = 2^20, d = 5	ε ≈ e/w ≈ 2.6e-6; δ ≈ e^-5 ≈ 0.0067
Counter size	4 bytes	2^20 × 5 × 4 = ~21 MB grid
Heap	50 entries	O(log K) per admission; negligible memory
Point query	min over 5 cells	one-sided; ≤ true + ε·N
Halving budget	w → 2^19	ε doubles (~5.2e-6); error band widens 2×

Over the firehose the grid sits at ~21 MB (well under 64 MB), the heap tracks the 50 heaviest URLs, and frequency(url) answers any point query as an upper bound within ε·N. Halving the budget halves w, which doubles ε and therefore doubles the additive error band — depth d (and thus the confidence 1 − δ) is untouched because memory was taken from width.

Output:

Query	Result	Error guarantee
top(50)	50 heaviest URLs, ranked	tight for keys ≫ ε·N
frequency("/checkout")	estimate ≥ true	≤ true + ε·N, prob 1 − δ
Grid memory	~21 MB	fixed vs event count
After halving budget	ε doubles	band 2× wider, same δ

Why this works — concept by concept:

d×w grid with d hashes — the sketch stores overlapping counters, not keys, so memory is O(w·d) regardless of how many distinct URLs stream by. The key universe can be unbounded; the grid does not grow.
min across rows — returns the least-collided (tightest) over-estimate, and increasing d raises the probability that some row was nearly collision-free — the 1 − δ confidence.
bounded top-K heap — the size-50 min-heap ranks candidates by their Count-Min estimate; it is O(log K) per event and never exceeds K entries, so the long tail costs nothing.
width sets ε, depth sets δ — spending memory on width tightens the additive error; spending it on depth raises confidence. Halving the budget on width doubles ε and leaves δ alone — a controllable, stated degradation.
Cost — O(d) per update and per query (constant, d = 5), O(w·d) space (~21 MB, fixed), O(log K) per top-K admission. The eliminated cost is the O(distinct URLs) memory of an exact hashmap, which the firehose would blow past.

Data Structures
Topic — data-structures
Hashing and counter-sketch problems

Practice →

Cardinality Topic — cardinality Frequency and distinct-count estimation problems

Practice →

3. t-digest & approximate quantiles

`t-digest` summarises a distribution as centroids that are large in the middle and tiny at the tails, so `approximate quantiles` like p99 and p999 stay accurate where you need them most

The mental model in one line: a t-digest is a quantile sketch that clusters streaming values into a bounded set of centroids — each a (mean, count) pair — where the allowed centroid size is governed by a scale function that keeps clusters small near the extremes (q close to 0 or 1) and lets them grow in the middle, so the summary spends its limited memory buying high relative accuracy at the tails and coarse accuracy at the median, which is exactly the trade percentiles monitoring wants because nobody pages on p50 but everyone pages on p99. It is mergeable, order-insensitive in the limit, and answers any quantile query by walking the centroids and interpolating.

Why exact quantiles are expensive, and what t-digest replaces.

Quantiles are global. The p99 of a stream is defined by the rank of every value relative to all others, so there is no per-value incremental exact summary — the naive exact method sorts all N values, an O(N) memory and O(N log N) time cost.
Histograms leak accuracy. Fixed-bucket histograms bound memory but hard-code where the accuracy goes: bucket edges chosen for the middle give terrible tail resolution, and you cannot know the right edges in advance for latency that spans microseconds to seconds.
t-digest adapts. It places many small centroids where quantiles change fast (the tails) and few large ones where they change slowly (the middle), so the same memory yields far better tail accuracy than an equal-width histogram.
The output is one small object. A few hundred centroids — a couple of kilobytes — summarise a stream of any length, and merge cheaply.

The centroid model and the scale function.

A centroid is (mean, count). It claims to represent count values clustered around mean. The digest is a sorted list of centroids covering the value range.
The scale function k(q). t-digest maps the quantile position q ∈ [0,1] through a non-linear scale (commonly k(q) = δ·(asin(2q−1))/(2π) or a log-like variant). A centroid may absorb more mass only if doing so keeps its k-width within one unit.
Tails get small centroids. Because k(q) is steep near q = 0 and q = 1, a centroid near the tail hits its size limit after absorbing very few values, so tail centroids stay tiny and numerous — high resolution.
The middle gets big centroids. Near q = 0.5, k(q) is flat, so a centroid can absorb a large count before splitting — low resolution where you do not care.
Compression δ. The single knob (often called compression, e.g. 100–500) sets the total centroid budget: higher δ means more centroids, more accuracy, more memory. Centroid count is roughly O(δ).

Building and querying.

Buffer and merge. Practical t-digests buffer incoming values, sort the buffer, then merge it into the centroid list in one pass, re-splitting any centroid that exceeds its k-width. Buffering amortises the sort and keeps updates fast.
Quantile query. To find the value at quantile q, accumulate centroid counts left to right until you pass q·N, then interpolate between adjacent centroid means. The tiny tail centroids make that interpolation precise at p99/p999.
Rank query. The inverse — given a value, sum the counts of centroids below it to estimate its percentile (useful for "what percentile is this 800 ms request").

Merging t-digests across shards.

Concatenate then re-compress. Merge is: take the union of both centroid lists, sort by mean, and re-cluster under the scale function into a fresh bounded digest. The result approximates the digest of the combined stream within the same error.
Order-insensitive in the limit. Because merging re-clusters from means, the combined digest is (nearly) independent of shard order — the property a reducer needs.
This is the production superpower. Each shard ships a few kilobytes of centroids; the reducer merges them; fleet-wide p99 falls out without shipping a single raw latency.

t-digest vs KLL vs Greenwald-Khanna.

t-digest. Excellent relative accuracy at the tails, tiny, mergeable, hugely popular — but no strict worst-case rank-error bound; accuracy is empirical and can degrade on adversarial inputs.
KLL. A newer quantile sketch with a provable (ε, rank) guarantee (the answer's rank is within ε·N of the query), mergeable, and the basis of Apache DataSketches quantiles — pick it when you must prove the error.
Greenwald-Khanna (GK). The classic deterministic ε-approximate quantile summary with a worst-case rank bound; mergeable but historically fiddlier to merge than KLL.

Common interview probes on t-digest.

"Why is t-digest more accurate at p99 than a histogram?" — required answer: centroids are small at the tails (scale function), so tail resolution is high.
"What is a centroid?" — a (mean, count) cluster; the digest is a sorted, bounded list of them.
"How do you merge two t-digests?" — union the centroids, sort by mean, re-cluster under the scale function.
"When would you use KLL instead?" — when you need a provable worst-case rank-error bound, not just empirical tail accuracy.

Worked example — build a t-digest and query p99

Detailed explanation. Implement a compact t-digest (buffer-and-merge style) and push a right-skewed latency stream through it, then query p50, p99, and p999 and compare to the exact values. The skew mimics real latency: most requests fast, a heavy right tail.

Config. compression δ = 100.
Stream. 100,000 latencies: a lognormal-like bulk plus a small fraction of slow outliers.
Query. p50, p99, p999.

Question. Implement the digest, ingest the stream, and report the three quantiles against exact.

Input.

Parameter	Value
Compression δ	100
Stream size N	100,000
Distribution	right-skewed latency (ms)
Queries	p50, p99, p999

Code.

import bisect, math

class TDigest:
    def __init__(self, compression: float = 100.0):
        self.delta = compression
        self.centroids = []   # sorted list of [mean, count]
        self.n = 0

    def _k_to_q_limit(self, q: float) -> float:
        # centroid size limit at quantile q: small near 0/1, large near 0.5
        return 4 * self.n * (1.0 / self.delta) * q * (1 - q) + 1

    def update(self, x: float, w: int = 1) -> None:
        self.n += w
        self.centroids.append([x, w])
        if len(self.centroids) > 10 * self.delta:  # buffer full -> compress
            self._compress()

    def _compress(self) -> None:
        self.centroids.sort(key=lambda c: c[0])
        merged, cum = [], 0.0
        for mean, count in self.centroids:
            if not merged:
                merged.append([mean, count]); cum = count; continue
            q = (cum + count / 2) / self.n
            limit = self._k_to_q_limit(q)
            m0, c0 = merged[-1]
            if c0 + count <= limit:                 # absorb into current centroid
                merged[-1][0] = (m0 * c0 + mean * count) / (c0 + count)
                merged[-1][1] = c0 + count
            else:                                   # start a new centroid
                merged.append([mean, count])
            cum += count
        self.centroids = merged

    def quantile(self, q: float) -> float:
        self._compress()
        target, cum = q * self.n, 0.0
        for i, (mean, count) in enumerate(self.centroids):
            if cum + count >= target:
                if i == 0:
                    return mean
                pm, pc = self.centroids[i - 1]
                frac = (target - cum) / count       # interpolate between means
                return pm + (mean - pm) * frac
            cum += count
        return self.centroids[-1][0]

    def merge(self, other: "TDigest") -> "TDigest":
        out = TDigest(self.delta)
        out.centroids = [c[:] for c in self.centroids] + [c[:] for c in other.centroids]
        out.n = self.n + other.n
        out._compress()
        return out

Step-by-step explanation.

update appends each value as a singleton centroid into a buffer and only triggers _compress when the buffer grows past 10·δ. Buffering means the expensive sort happens once per batch, not per value — the standard amortisation that keeps ingestion fast.
_compress sorts the centroids by mean and sweeps left to right, folding each into the previous centroid only if the combined count stays under the scale-function limit _k_to_q_limit(q). That limit is small when q is near 0 or 1 and large near 0.5, which is what forces tiny tail centroids and fat middle centroids.
The limit formula 4·N·(1/δ)·q·(1−q) + 1 is the q(1−q) shape in action: at q = 0.5 it is largest (centroids can be big), and it shrinks toward the tails (centroids must stay small). This is the whole trick that makes p99 accurate.
quantile accumulates centroid counts until it passes q·N, then linearly interpolates between the two straddling centroid means. Because tail centroids are tiny, the interpolation interval at p99 is narrow, so the answer is close to exact.
merge simply unions the two centroid lists and re-compresses — the same code path, which is why merge and update share accuracy behaviour. The digest stays a few hundred centroids no matter how many values or merges it has seen.

Output.

Quantile	Exact (ms)	t-digest (ms)	Relative error
p50	100.0	~100.4	~0.4%
p99	512.0	~510	~0.4%
p999	1180.0	~1173	~0.6%
Centroids stored	100,000 (exact)	~180	—

Rule of thumb. Read a t-digest as "accurate where it is steep" — tails are tight, the middle is coarse. That is the correct default for latency SLOs, where p99 and p999 are the numbers that matter and p50 rarely does.

Worked example — merging t-digests across shards

Detailed explanation. The reason t-digest dominates fleet monitoring is that per-shard digests merge into a global digest that answers p99 for the whole fleet — without shipping raw latencies. Build one digest per shard on disjoint slices of the same latency population, merge them, and compare the merged p99 to a digest built over all values at once.

Setup. 4 shards, each ingesting 25,000 latencies from the same distribution.
Merge. Fold the four digests pairwise into one.
Check. Merged p99 ≈ single-digest p99.

Question. Build four shard digests, merge them, and confirm the merged p99 matches an all-at-once digest.

Input.

Shard	Values	Local p99 (ms)
1	25,000	~509
2	25,000	~514
3	25,000	~511
4	25,000	~513

Code.

from functools import reduce

# Each shard builds its digest independently, in parallel, no coordination
shard_digests = []
for shard_values in (shard1, shard2, shard3, shard4):
    td = TDigest(compression=200)
    for v in shard_values:
        td.update(v)
    shard_digests.append(td)          # only ~2 KB of centroids leaves each shard

# Reducer folds them — merge is associative, so order does not matter
global_td = reduce(lambda a, b: a.merge(b), shard_digests)

# Compare against a digest built over every value at once
reference = TDigest(compression=200)
for v in shard1 + shard2 + shard3 + shard4:
    reference.update(v)

print("merged  p99:", round(global_td.quantile(0.99), 1))
print("single  p99:", round(reference.quantile(0.99), 1))

Step-by-step explanation.

Each shard ingests its 25,000 values into a local digest with no cross-shard communication. The heavy work — touching every latency — happens locally and in parallel, exactly as with Count-Min.
Only the centroid lists (a couple of kilobytes each) leave the shards. This is the bandwidth win: merging p99 across a 400-node fleet moves kilobytes, not the terabytes of raw latency the exact method would require.
reduce(merge, ...) folds the digests in a chain (a tree in a real reducer). Merge is associative because it re-clusters from centroid means, so any fold order yields the same result within the error bound.
The merged p99 lands within a fraction of a percent of the reference digest built over all values at once. Small discrepancies come from re-compression at merge time, bounded by the compression parameter — raise δ to shrink it.
This is precisely the pattern Druid and Spark use: sketch per partition, merge at query. The digest's order-insensitive merge is what makes "p99 by region, then p99 across all regions" a matter of choosing which digests to fold.

Output.

Metric	Value
Merged p99 (ms)	~512
Single-digest p99 (ms)	~512
Difference	< 0.3%
Bytes shipped per shard	~2 KB
Raw latencies shipped	0

Rule of thumb. Build one digest per partition and merge at the reducer. Because merge re-clusters from means, the fold order is irrelevant and any dimensional slice (region, service, time bucket) is just a different set of digests to combine.

Worked example — why the tails are accurate (scale-function intuition)

Detailed explanation. The tail accuracy is not magic; it is the scale function spending centroids where quantiles change fast. Contrast t-digest with an equal-width histogram on the same skewed latency stream and show that the histogram's fixed buckets blur the tail while t-digest resolves it.

Histogram. 200 equal-width buckets from min to max. With a heavy tail, almost all buckets sit in the sparse tail region or the dense head, wasting resolution.
t-digest. ~200 centroids placed by k(q); density follows the data, concentrating at both tails.
The measurement. Estimate p999 with each and compare error.

Question. Compare p999 error between an equal-width histogram and a t-digest of the same size.

Input.

Structure	Size	Placement	p999 error
Equal-width histogram	200 buckets	fixed edges	large (blurred tail)
t-digest	~200 centroids	scale-function adaptive	small

Code.

def equal_width_p999(values, buckets=200):
    lo, hi = min(values), max(values)
    width  = (hi - lo) / buckets
    counts = [0] * buckets
    for v in values:
        idx = min(int((v - lo) / width), buckets - 1)
        counts[idx] += 1
    target, cum = 0.999 * len(values), 0
    for i, c in enumerate(counts):
        cum += c
        if cum >= target:
            return lo + (i + 0.5) * width         # bucket midpoint = coarse guess
    return hi

def tdigest_p999(values, compression=200):
    td = TDigest(compression)
    for v in values:
        td.update(v)
    return td.quantile(0.999)

exact_p999 = sorted(latencies)[int(0.999 * len(latencies))]
print("exact   p999:", round(exact_p999, 1))
print("hist    p999:", round(equal_width_p999(latencies), 1))   # coarse at tail
print("tdigest p999:", round(tdigest_p999(latencies), 1))       # tight at tail

Step-by-step explanation.

The equal-width histogram divides the entire [min, max] range into 200 fixed buckets. With a heavy tail, the max is far out, so each bucket is wide — and the p999 lands inside one wide tail bucket whose midpoint can be off by tens of milliseconds.
The histogram cannot fix this by adding buckets in the middle; its accuracy at the tail is set by the range, not the data density, so a few extreme outliers stretch every bucket and blur the very quantile you asked for.
The t-digest places centroids by quantile position, not value position. Near q = 0.999 the scale function forces tiny centroids, so the digest keeps fine-grained structure exactly where the histogram is coarsest.
The result: the t-digest's p999 sits within a fraction of a percent of exact, while the equal-width histogram's p999 can be off by a large margin — same memory, very different tail resolution.
The lesson generalises: any fixed-bucket scheme hard-codes where accuracy lives, and for latency the tail is unknowable in advance. t-digest (and KLL) adapt to the data, which is why they, not histograms, back serious percentile systems.

Output.

Method	p999 estimate (ms)	Error vs exact
Exact	1180	0
Equal-width histogram	~1240	~5%
t-digest	~1173	~0.6%

Rule of thumb. If the tail matters, do not use equal-width histograms — their accuracy is dictated by the range, so a single outlier blurs every bucket. Use a t-digest (or KLL) whose resolution follows the data to the tail.

Statistics interview question on approximate quantiles

A senior interviewer might ask: "You run 500 service replicas and need a fleet-wide p50/p95/p99/p999 latency dashboard refreshed every 10 seconds, sliceable by service and region, and you cannot ship raw latencies off the hosts. Design the quantile pipeline — the per-replica structure, the merge, the accuracy you'd promise at p999, and how you'd choose the compression."

Solution Using per-replica t-digests merged at the collector

# quantile_pipeline.py — per-replica t-digest, merged per (service, region) slice
from collections import defaultdict
from functools import reduce

# 1. On each replica: fold latencies into a local digest, flush every 10s
class ReplicaAgg:
    def __init__(self, compression=200):
        self.td = TDigest(compression)

    def observe(self, latency_ms: float) -> None:
        self.td.update(latency_ms)

    def flush(self) -> "TDigest":
        out, self.td = self.td, TDigest(self.td.delta)   # swap out, reset
        return out                                        # ~2-4 KB of centroids

# 2. At the collector: group incoming digests by slice key, merge each group
def collect(digests_by_slice: dict[tuple, list["TDigest"]]) -> dict:
    dashboard = {}
    for slice_key, digests in digests_by_slice.items():
        merged = reduce(lambda a, b: a.merge(b), digests)
        dashboard[slice_key] = {
            "p50":  merged.quantile(0.50),
            "p95":  merged.quantile(0.95),
            "p99":  merged.quantile(0.99),
            "p999": merged.quantile(0.999),
        }
    return dashboard

# 3. "All regions for service X" = merge every region's slice digest
def rollup(service: str, dashboard_digests: dict[tuple, "TDigest"]) -> dict:
    parts = [d for (svc, _region), d in dashboard_digests.items() if svc == service]
    merged = reduce(lambda a, b: a.merge(b), parts)
    return {q: merged.quantile(v) for q, v in
            {"p50": .5, "p95": .95, "p99": .99, "p999": .999}.items()}

Step-by-step trace.

Layer	Component	Purpose
Replica	ReplicaAgg t-digest (c=200)	fold latencies locally; flush ~3 KB every 10s
Wire	centroid lists	kilobytes per replica, not raw samples
Collector	merge per (service, region)	associative fold -> one digest per slice
Rollup	merge slices sharing a service	cross-region p99 = union of region digests
Dashboard	quantile(q) per slice	p50/p95/p99/p999 with tail-tight accuracy

After deployment, each of the 500 replicas ships a ~3 KB digest every 10 seconds. The collector merges per slice and can roll up any union of slices, so "p99 for checkout across all regions" is a fold of the checkout digests. p999 stays within roughly half a percent because compression 200 keeps ample tail centroids; raising compression tightens it at a linear memory cost.

Output:

Slice	p50 (ms)	p95 (ms)	p99 (ms)	p999 (ms)
checkout / us-east	42	180	512	1170
checkout / eu-west	45	190	528	1205
checkout / all (rollup)	43	185	519	1188
Bytes/replica/flush	~3 KB	—	—	—

Why this works — concept by concept:

Per-replica t-digest — each replica summarises its own latencies into a few hundred centroids, so the raw samples never leave the host and the ingestion cost is a cheap buffered merge.
Scale-function tail accuracy — compression 200 forces small centroids near q = 0.999, so p999 — the number the SLO cares about — is resolved tightly while the middle stays deliberately coarse.
Associative merge = arbitrary slices — because merging re-clusters from means, any dimensional rollup (service, region, or their union) is just a different set of digests to fold, computed at query time.
Compression is the accuracy knob — higher compression buys more centroids and tighter tails at linear memory cost; you choose it from the p999 error you must promise, not from a guess.
Cost — O(1) amortised per observation on the replica, O(centroids) bytes per flush (~3 KB, independent of request volume), and O(slices × digests) merges at the collector. The eliminated cost is shipping and storing every raw latency to compute exact percentiles.

Statistics
Topic — statistics
Quantile and percentile estimation problems

Practice →

Data Structures Topic — data-structures Streaming summary structure problems

Practice →

4. Error bounds & sizing

Sketch memory is a function of the error you demand — `ε` sets the width, `δ` sets the confidence, and additive vs relative error decides whether the sketch can even answer your question

The one-sentence invariant: every sketch exposes an error budget you must set from the question — for a Count-Min sketch the width is w = ⌈e/ε⌉ and the depth is d = ⌈ln(1/δ)⌉, giving an additive error of at most ε·N with probability 1 − δ at a cost of w·d counters; for a t-digest the compression δ_t sets the centroid budget and the relative tail accuracy — and the single most important judgement is whether the sketch's error is additive (a fixed slice of the total, fatal for rare keys) or relative (scales with the answer, right for cardinality and tails). Sizing is not guesswork: you state the budget or the tolerance, and the formulas give the other.

The Count-Min error theorem, in plain terms.

The guarantee. For a Count-Min sketch with width w = ⌈e/ε⌉ and depth d = ⌈ln(1/δ)⌉, the estimate f̂(x) satisfies f(x) ≤ f̂(x) ≤ f(x) + ε·N with probability at least 1 − δ, where N is the total weight (sum of all counts).
ε is the additive error fraction. The overshoot is bounded by ε times the whole stream's weight — not the key's own count. Halve ε, double the width, halve the overshoot.
δ is the failure probability. With probability up to δ the bound can be exceeded (all d rows happened to collide badly). Add rows to shrink δ; each row multiplies the failure probability by roughly 1/e.
Memory falls out. space = w·d counters = ⌈e/ε⌉·⌈ln(1/δ)⌉. Width dominates the memory because 1/ε grows fast while ln(1/δ) grows slowly — you buy accuracy mostly by widening.

Additive vs relative error — the judgement that decides the sketch.

Additive (± ε·N). Count-Min. The error is a fixed number of "stream units". For a heavy key (count ≫ ε·N) it is negligible; for a rare key (count ≲ ε·N) it swamps the true value. Count-Min is therefore a heavy-hitters tool, not a rare-key tool.
Relative (± ε·true). HyperLogLog (cardinality), t-digest/KLL at the tails. The error scales with the answer, so small answers get small absolute error — the right model when the small values matter.
The trap. Using Count-Min to ask "how many keys appeared exactly once" is hopeless: ε·N is enormous relative to a count of 1, so every rare key's estimate is dominated by collision noise. Know which error type your question needs before sizing.
The tell in an interview. Saying "Count-Min gives additive error, so it's for heavy hitters, not rare keys" is the senior signal that you understand the bound, not just the formula.

t-digest sizing.

Compression δ_t sets the budget. Centroid count is roughly O(δ_t) (a few times δ_t in practice). Compression 100 → ~hundreds of centroids → a couple of kilobytes; compression 500 → more centroids, tighter tails, ~10 KB.
Accuracy is relative and tail-tight. t-digest promises small relative error that is smallest at the extremes; there is no simple closed-form worst-case bound (that is KLL's territory), so you validate empirically against exact on representative data.
Choose from the p-you-care-about. If you page on p999, pick compression high enough that p999 error is within tolerance on your data; p50 will be more than accurate enough regardless.

Working backward from the requirement.

From a memory budget. Given B bytes and 4-byte counters, w·d ≤ B/4. Fix δ (hence d), then w = B/(4d) gives the achievable ε = e/w. State the resulting error.
From an accuracy target. Given a required ε and δ, compute w = ⌈e/ε⌉, d = ⌈ln(1/δ)⌉, and report the memory. If it exceeds budget, either relax ε or accept that the question needs a different structure.
From a heavy-hitter threshold. To reliably find φ-heavy hitters, set ε a factor (say 5–10×) below φ so the additive band cannot lift a sub-threshold key over the line.

Common interview probes on sizing.

"How do you size a Count-Min sketch?" — required answer: w = ⌈e/ε⌉, d = ⌈ln(1/δ)⌉, memory w·d.
"What error does it guarantee?" — additive ε·N, one-sided, with confidence 1 − δ.
"Why is Count-Min bad for rare keys?" — additive error ε·N swamps small counts.
"How do you pick t-digest compression?" — from the p999 error you must hit on representative data; centroid count ≈ O(compression).

Worked example — size a Count-Min from a memory budget

Detailed explanation. Given a 1 MB budget and a required confidence, derive the achievable error and check it against a heavy-hitter threshold. This is the calculation you do on a whiteboard before writing any code.

Budget. 1 MB = 1,048,576 bytes, 4-byte counters → 262,144 counters total.
Confidence. Want δ = 0.001 → d = ⌈ln(1000)⌉ = ⌈6.9⌉ = 7 rows.
Derive ε. w = floor(262144 / 7) = 37,449 columns → ε = e/w ≈ 2.718 / 37449 ≈ 7.3e-5.

Question. For a 1 MB budget and δ = 0.001, compute w, d, ε, and the additive error on a stream of N = 2×10^9. Can it find 0.1%-heavy hitters?

Input.

Given	Value
Budget	1 MB (262,144 counters)
δ	0.001
Stream weight N	2,000,000,000
Heavy-hitter threshold φ	0.001 (0.1%)

Code.

import math

BYTES        = 1_048_576
COUNTER_SIZE = 4
delta        = 0.001
N            = 2_000_000_000
phi          = 0.001

d = math.ceil(math.log(1 / delta))          # depth from delta
w = (BYTES // COUNTER_SIZE) // d             # width from remaining budget
eps = math.e / w                             # achievable additive fraction

additive_error = eps * N                     # worst-case overshoot in counts
threshold      = phi * N                     # a 0.1%-heavy hitter has this many

print(f"d = {d} rows")                       # 7
print(f"w = {w} cols")                       # ~37449
print(f"eps = {eps:.2e}")                    # ~7.3e-5
print(f"additive error = {additive_error:,.0f}")   # ~145,000
print(f"heavy-hitter threshold = {threshold:,.0f}") # 2,000,000
print(f"error / threshold = {additive_error/threshold:.3f}")  # ~0.07

Step-by-step explanation.

Start from confidence: δ = 0.001 needs d = ⌈ln(1/0.001)⌉ = ⌈ln 1000⌉ = ⌈6.9⌉ = 7 rows. Depth is cheap — it grows only logarithmically in 1/δ — so buying high confidence costs little memory.
Spend the rest of the budget on width: 262144 / 7 ≈ 37,449 columns. Width is where the memory goes, and width is what tightens the additive error.
The achievable error fraction is ε = e/w ≈ 7.3×10⁻⁵. On a stream of N = 2×10⁹, the worst-case additive overshoot is ε·N ≈ 145,000 counts.
A 0.1%-heavy hitter has φ·N = 2,000,000 occurrences. The additive error (145,000) is about 7% of that threshold — comfortably below it, so genuine 0.1%-heavy hitters stand well clear of the noise band and cannot be confused with sub-threshold keys.
The check error/threshold ≈ 0.07 ≪ 1 is the sign-off: the sketch fits the budget and resolves the heavy hitters you care about. Had the ratio approached 1, you would need more width (more memory) or a looser threshold.

Output.

Quantity	Value
Depth d	7
Width w	37,449
ε	7.3e-5
Additive error (ε·N)	~145,000
0.1% threshold (φ·N)	2,000,000
Error / threshold	~0.07 (safe)

Rule of thumb. Size depth from δ first (it is cheap), then spend all remaining memory on width to minimise ε. Confirm ε·N is well under your heavy-hitter threshold φ·N — a ratio under ~0.1 means genuine heavy hitters are safe from collision noise.

Worked example — additive vs relative error, side by side

Detailed explanation. Make the additive-vs-relative distinction concrete by asking two questions of the same stream: "how big is the heaviest key" (heavy — additive error fine) and "how many singletons" (rare — additive error fatal). Show numerically why Count-Min answers the first and cannot answer the second.

Stream. N = 10^9 with one key at 5% (5×10^7) and millions of singletons.
Additive band. With ε = 10^-4, the band is ε·N = 10^5.
Verdict. For the 5% key, 10^5 overshoot on 5×10^7 is 0.2% relative — fine. For a singleton, 10^5 overshoot on 1 is a 100,000× error — useless.

Question. Compute the relative impact of the same additive band on a heavy key and a rare key, and state which questions Count-Min can answer.

Input.

Key type	True count	Additive band ε·N	Relative impact
Heavy (5%)	50,000,000	100,000	0.2%
Mid (0.01%)	100,000	100,000	100%
Rare (singleton)	1	100,000	100,000×

Code.

N   = 1_000_000_000
eps = 1e-4
band = eps * N                       # 100,000 additive overshoot

for label, true in [("heavy 5%", 50_000_000),
                    ("mid 0.01%", 100_000),
                    ("rare x1", 1)]:
    relative = band / true
    verdict  = "usable" if relative < 0.1 else "useless"
    print(f"{label:10s} true={true:>12,}  band/true={relative:>10.2%}  -> {verdict}")

# heavy 5%   true=  50,000,000  band/true=     0.20%  -> usable
# mid 0.01%  true=     100,000  band/true=   100.00%  -> useless
# rare x1    true=           1  band/true= 10,000,000.00%  -> useless

Step-by-step explanation.

The additive band is a constant ε·N = 100,000, identical for every key regardless of its own frequency. That constancy is the entire character of additive error.
For the 5% heavy key (true 5×10⁷), a 100,000 overshoot is 0.2% — the estimate is essentially exact in relative terms. Count-Min nails heavy hitters.
For a mid key at the 0.01% mark (true 100,000), the band equals the true value — a potential 100% error. The estimate is worthless; the key is right at the edge of the noise floor.
For a singleton, the band is 100,000× the true value. Count-Min literally cannot distinguish a singleton from any other sub-band key — asking it "how many keys appeared once" is a category error.
The takeaway: Count-Min's noise floor is ε·N. Any question about keys near or below that floor (rare keys, exact tail frequencies, distinct-of-rare) needs a relative-error structure — HyperLogLog for distinctness, or simply exact if the rare-key set fits.

Output.

Key	True	Estimate band	Relative error	Count-Min usable?
Heavy 5%	50,000,000	+0–100,000	~0.2%	yes
Mid 0.01%	100,000	+0–100,000	up to 100%	no
Rare ×1	1	+0–100,000	up to 10⁷%	no

Rule of thumb. Count-Min's error floor is ε·N. If the answers you care about are smaller than that floor, the sketch is the wrong tool — switch to a relative-error structure or exact counting for the rare-key regime.

Worked example — t-digest compression vs accuracy sweep

Detailed explanation. Show the compression knob's effect by sweeping it and measuring p999 error and memory. This is how you choose compression from a promised accuracy rather than guessing.

Sweep. Compression ∈ {50, 100, 200, 500}.
Measure. p999 relative error and approximate centroid count (memory proxy).
Choose. The smallest compression that meets the p999 tolerance.

Question. Sweep compression and pick the smallest value meeting a 1% p999 tolerance.

Input.

Compression	~Centroids	~Memory	p999 error
50	~120	~2 KB	~1.8%
100	~220	~4 KB	~0.9%
200	~420	~7 KB	~0.5%
500	~1000	~16 KB	~0.2%

Code.

import bisect

def p999_error(values, compression) -> float:
    td = TDigest(compression)
    for v in values:
        td.update(v)
    est   = td.quantile(0.999)
    exact = sorted(values)[int(0.999 * len(values))]
    return abs(est - exact) / exact, len(td.centroids)

TOLERANCE = 0.01
choice = None
for c in (50, 100, 200, 500):
    err, ncent = p999_error(latencies, c)
    print(f"compression={c:4d}  centroids~{ncent:4d}  p999_err={err:.2%}")
    if choice is None and err <= TOLERANCE:
        choice = c

print("smallest compression meeting 1% p999:", choice)   # 100

Step-by-step explanation.

Each compression value builds a digest, queries p999, and compares to the exact p999 on the same data. This is the empirical validation t-digest requires — there is no closed-form bound to trust in place of measurement.
Error falls as compression rises: more centroids means smaller tail clusters and tighter interpolation at p999. The relationship is smooth and monotone, so the sweep is quick.
Memory rises roughly linearly with compression (centroid count ≈ O(compression)), so doubling compression roughly doubles the digest size — the standard accuracy-for-memory trade.
The chooser picks the smallest compression that meets the 1% tolerance — here compression 100 at ~0.9% error and ~4 KB. Going higher would only spend memory for accuracy the SLO does not require.
The method generalises: sweep on representative data, pick the cheapest setting that clears your promised tail error, and re-validate if the distribution shifts materially. Never pick compression by folklore ("100 is fine") without checking against your own p999.

Output.

Compression	Centroids	Memory	p999 error	Meets 1%?
50	~120	~2 KB	~1.8%	no
100	~220	~4 KB	~0.9%	yes ← pick
200	~420	~7 KB	~0.5%	yes (overkill)
500	~1000	~16 KB	~0.2%	yes (overkill)

Rule of thumb. Choose t-digest compression by sweeping on representative data and taking the smallest value that clears your p999 tolerance. Memory scales linearly with compression, so higher settings only make sense when the tail error must be tighter.

Data-engineering interview question on sketch sizing

A senior interviewer might ask: "You have 8 MB per worker for a frequency sketch and must reliably surface any key above 0.05% of the stream, plus answer point frequencies with a stated error, on a 5×10^9-weight stream. Size the Count-Min sketch, justify the depth and width split, quote the error, and say what breaks if the stream grows 10×."

Solution Using the ε/δ sizing formulas with a heavy-hitter margin

# size_cms.py — derive (w, d, eps) from budget, confidence, and HH threshold
import math

def size_cms(budget_bytes: int, counter_size: int, delta: float,
             N: int, phi: float, margin: float = 8.0) -> dict:
    counters = budget_bytes // counter_size
    d        = math.ceil(math.log(1 / delta))         # depth from confidence
    w        = counters // d                            # width from remaining budget
    eps      = math.e / w                               # achievable additive fraction

    additive = eps * N
    threshold = phi * N
    # require eps*N to sit at least `margin`x below the HH threshold
    safe = additive * margin <= threshold
    return {
        "d": d, "w": w, "eps": eps,
        "additive_error": additive,
        "hh_threshold": threshold,
        "ratio": additive / threshold,
        "safe_for_heavy_hitters": safe,
        "memory_mb": counters * counter_size / 1e6,
    }

cfg = size_cms(budget_bytes=8 * 1024 * 1024, counter_size=4,
               delta=1e-3, N=5_000_000_000, phi=5e-4)
for k, v in cfg.items():
    print(f"{k:24s} {v}")

Step-by-step trace.

Step	Value	Reasoning
Counters	2,097,152	8 MB / 4 bytes
Depth d	7	⌈ln(1/0.001)⌉
Width w	299,593	counters / d
ε	~9.1e-6	e / w
Additive error ε·N	~45,400	on N = 5e9
HH threshold φ·N	2,500,000	0.05% of 5e9
Ratio	~0.018	ε·N well under threshold

The 8 MB budget yields d = 7, w ≈ 299,593, and ε ≈ 9.1×10⁻⁶. On the 5×10⁹ stream the additive band is ~45,400 — about 1.8% of the 2,500,000 heavy-hitter threshold, so any key above 0.05% is safely separable from noise. Depth is fixed by confidence (cheap, logarithmic); everything else went to width to minimise ε.

Output:

Requirement	Result	Status
Fits 8 MB	2.1M counters	yes
Confidence 1 − δ	0.999 (d=7)	met
ε·N vs 0.05% threshold	~1.8% of it	safe (8× margin)
Point-frequency error	≤ ~45,400, one-sided	stated
Stream grows 10× (N=5e10)	ε·N → ~454,000	still 18% of threshold — degraded but safe

Why this works — concept by concept:

Depth from δ, width from budget — d = ⌈ln(1/δ)⌉ buys confidence logarithmically, so it is set first and cheaply; the remaining ~2.1M counters go entirely to width to drive ε down.
ε·N is the noise floor — the additive error scales with the total stream weight N, not the key's count, so the sizing check is always "is ε·N well below φ·N", i.e. is the margin big enough.
Heavy-hitter margin — requiring ε·N to sit ~8× under the threshold guarantees a sub-threshold key cannot be over-estimated across the line, which is what "reliably surface any key above 0.05%" demands.
Graceful growth — if N grows 10×, ε·N grows 10× too (to ~454,000), still only ~18% of the threshold; the sketch degrades predictably rather than failing, and you would rewiden only if the margin got tight.
Cost — O(w·d) space (fixed at ~8 MB), O(d) per update/query (constant). The error is O(ε·N) additive; the only thing that erodes it is a growing N, which is a known, quantifiable dial — not a surprise.

Statistics
Topic — statistics
Error-bound and confidence-interval problems

Practice →

Cardinality Topic — cardinality Sketch sizing and accuracy problems

Practice →

5. Sketches in production — Spark / Druid / monitoring

The production pattern is always the same — sketch at ingest, store the sketch, merge at query — and Spark, Druid, and metrics backends each expose `mergeable summaries` as first-class objects

The one-sentence invariant: in every production system that scales approximate aggregation — Spark batch jobs, Druid OLAP rollups, and percentile-based monitoring — the sketch is computed once per partition or per ingest interval, the sketch object itself (not the raw data) is stored and shipped, and queries merge the relevant sketches on demand, so the expensive pass over raw data happens once and any slice or rollup is answered by an O(sketch size) merge. Spark exposes approx_count_distinct and percentile_approx plus the Apache DataSketches UDAFs; Druid stores quantile/cardinality/frequency sketches in segments; monitoring systems store t-digest/DDSketch objects so p99 merges across hosts.

Spark — approximate aggregation built in and via DataSketches.

Built-in functions. approx_count_distinct(col, rsd) uses HyperLogLog++ with a target relative standard deviation; percentile_approx(col, p, accuracy) computes quantiles with an accuracy knob (higher = more memory, tighter).
Apache DataSketches UDAFs. The datasketches-spark package adds mergeable KLL quantiles, Theta (set-operation cardinality), and Frequent-Items sketches usable in groupBy(...).agg(...). Because they are mergeable, Spark computes partial sketches per partition and combines them in the shuffle — no raw-row aggregation.
Why it matters. A groupBy(region).agg(sketch(...)) builds one sketch per region by merging partial per-partition sketches; the sketch column can be persisted and later merged across regions without re-reading the source.
The persistence trick. Store the serialized sketch as a column. Tomorrow's "distinct users over the last 7 days" merges 7 daily sketch rows — an O(7) merge instead of a re-scan of a week of events.

Druid — sketches as aggregators at ingestion.

Ingest-time rollup. Druid can apply sketch aggregators during ingestion so each segment stores, per dimension combination, a quantilesDoublesSketch (KLL), thetaSketch or HLLSketch (cardinality), and frequency sketches — instead of raw rows.
Query-time merge. A query over a time range and dimension filter merges the pre-built segment sketches. The cost is proportional to the number of segments touched, not the number of raw events they summarise.
The extension. The druid-datasketches extension provides these aggregators; a quantilesDoublesSketch field answers p50/p95/p99 for any slice by merging segment sketches at query time.
Why it wins. Retention of raw events is expensive and often unnecessary; retaining sketches gives bounded storage and mergeable quantiles/cardinalities across arbitrary time-and-dimension slices.

Monitoring — histograms, summaries, and why mergeability decides.

Prometheus histogram. A fixed set of cumulative buckets per series. Buckets are mergeable across instances (sum the bucket counts), so fleet-wide quantiles are computable via histogram_quantile(...) — but accuracy is bounded by bucket layout, which is hard to pick for wide-range latency.
Prometheus summary. Computes quantiles client-side on each instance and exposes them directly. Summaries are not mergeable — you cannot average per-instance p99s into a fleet p99 — which quietly makes cross-instance percentiles impossible. This is the canonical monitoring trap.
DDSketch. A relative-error quantile sketch (Datadog's) with a guaranteed relative-error bound and full mergeability, so fleet-wide p99 with a promised error is a merge of per-host DDSketches. It is the answer to the summary's non-mergeability.
t-digest backends. Many metrics stores use t-digest for the same reason: mergeable, tail-accurate percentiles across hosts and time.

The universal contract.

Sketch at ingest. Build the sketch where the raw data first lands — per Spark partition, per Druid segment, per monitored host.
Store the sketch. Persist the serialized sketch object; discard or cold-store the raw data.
Merge at query. Any slice, rollup, or time range is a merge of the relevant sketches — associative, cheap, and independent of the raw event count.

Common interview probes on production sketches.

"How does Spark compute an approximate distinct count?" — approx_count_distinct (HyperLogLog++), merged per partition in the shuffle.
"Why store sketches in Druid segments?" — mergeable pre-aggregation; query merges segment sketches instead of scanning raw rows.
"Why can't you average Prometheus summary quantiles?" — summaries are computed client-side and are not mergeable.
"What does DDSketch guarantee that t-digest doesn't?" — a provable relative-error bound with full mergeability.

Worked example — Spark percentile_approx and DataSketches KLL

Detailed explanation. Compute approximate quantiles and distinct counts in Spark two ways — the built-in percentile_approx / approx_count_distinct, and a mergeable DataSketches KLL column you can persist and re-merge later. Group by region so the per-partition-then-merge behaviour is visible.

Built-in. percentile_approx(latency, array(0.5, 0.95, 0.99), 10000) and approx_count_distinct(user_id, 0.01).
DataSketches. Build a KLL sketch per region, store it, and merge across days later.
Grouping. groupBy("region") so Spark merges partial sketches in the shuffle.

Question. Write the Spark job for both approaches and show the mergeable KLL persistence.

Input.

Column	Type	Question
region	string	group key
latency_ms	double	p50/p95/p99
user_id	long	distinct count

Code.

from pyspark.sql import functions as F

# 1. Built-in approximate aggregations, grouped by region
agg = (events
    .groupBy("region")
    .agg(
        F.expr("percentile_approx(latency_ms, array(0.5, 0.95, 0.99), 10000)").alias("pcts"),
        F.approx_count_distinct("user_id", 0.01).alias("distinct_users"),
    )
    .select(
        "region",
        F.col("pcts")[0].alias("p50"),
        F.col("pcts")[1].alias("p95"),
        F.col("pcts")[2].alias("p99"),
        "distinct_users",
    ))
agg.show()

-- 2. Apache DataSketches (SQL / UDAF form): mergeable KLL quantile sketch per region
--    Persist the sketch column so it can be merged across days without re-scanning.
CREATE TABLE region_latency_sketch AS
SELECT
    region,
    kll_sketch_agg(latency_ms) AS latency_kll   -- mergeable sketch object
FROM events
GROUP BY region;

-- 3. Later: fleet p99 for the last 7 days = merge 7 daily sketch rows (no re-scan)
SELECT
    region,
    kll_sketch_get_quantile(kll_sketch_merge(latency_kll), 0.99) AS p99_7d
FROM   region_latency_sketch_daily
WHERE  day >= current_date - INTERVAL 7 DAYS
GROUP  BY region;

Step-by-step explanation.

percentile_approx(col, ps, accuracy) computes the requested quantiles in one pass; the accuracy argument (10,000 here) is the memory/accuracy knob — higher trades RAM for tighter quantiles. Spark builds partial summaries per partition and merges them in the shuffle triggered by groupBy.
approx_count_distinct(user_id, 0.01) is HyperLogLog++ with a 1% target relative standard deviation. Like the quantile function, it is mergeable, so the per-region distinct count is a merge of per-partition HLLs — never an exact countDistinct shuffle of all user IDs.
The DataSketches kll_sketch_agg builds a serializable, mergeable KLL quantile sketch per region and stores it as a column. This is the crucial difference from the built-in: the sketch object is persisted, not just a scalar quantile.
Because the stored KLL is mergeable, "p99 over the last 7 days" is kll_sketch_get_quantile(kll_sketch_merge(daily_sketches), 0.99) — an O(7) merge of daily sketch rows, not a re-scan of a week of raw events. This is the persistence trick that makes rolling windows cheap.
KLL (unlike t-digest) carries a provable (ε, rank) guarantee, so when an SLA requires a stated quantile error, the DataSketches path is preferable to percentile_approx, whose accuracy is empirical.

Output.

region	p50	p95	p99	distinct_users
us-east	42	180	512	1,204,880
eu-west	45	190	528	843,207
ap-south	51	205	560	512,006

Rule of thumb. Use percentile_approx / approx_count_distinct for one-shot approximate answers, but persist a mergeable DataSketches column when you need rolling windows or cross-slice rollups — storing the sketch object turns a re-scan into an O(days) merge.

Worked example — Druid quantiles sketch at ingestion

Detailed explanation. Configure a Druid ingestion spec to build a quantilesDoublesSketch (KLL) on latency at rollup time, then query p99 for a dimension slice by merging segment sketches. The raw latencies are never stored — only the sketch per segment.

Ingest. A quantilesDoublesSketch metric on latency_ms with a chosen k (accuracy/size).
Store. Each segment holds the sketch per dimension combination.
Query. A quantilesDoublesSketch post-aggregator merges segment sketches and reads p99.

Question. Write the Druid ingestion metricsSpec and the query that returns p99 latency by region.

Input.

Field	Role
region	dimension
latency_ms	sketched metric
k	sketch accuracy (e.g. 256)

Code.

// Ingestion metricsSpec — build a KLL quantiles sketch on latency at rollup
{
  "metricsSpec": [
    { "type": "count", "name": "events" },
    {
      "type": "quantilesDoublesSketch",
      "name": "latency_sketch",
      "fieldName": "latency_ms",
      "k": 256
    }
  ],
  "granularitySpec": { "queryGranularity": "minute", "rollup": true }
}

// Query — merge segment sketches and read p99 by region
{
  "queryType": "groupBy",
  "dataSource": "requests",
  "granularity": "all",
  "dimensions": ["region"],
  "intervals": ["2026-09-05/2026-09-06"],
  "aggregations": [
    { "type": "quantilesDoublesSketch", "name": "merged_latency", "fieldName": "latency_sketch" }
  ],
  "postAggregations": [
    {
      "type": "quantilesDoublesSketchToQuantile",
      "name": "p99_latency",
      "field": { "type": "fieldAccess", "fieldName": "merged_latency" },
      "fraction": 0.99
    }
  ]
}

Step-by-step explanation.

The quantilesDoublesSketch metric in the ingestion spec tells Druid to fold latency_ms into a KLL sketch during rollup, with k = 256 setting the accuracy/size. Each segment ends up storing the sketch per dimension combination rather than the raw latencies.
Because rollup is on, many raw rows collapse into one sketch per (minute, region, …) bucket — bounded storage that grows with dimension cardinality and time, not with raw event volume.
At query time, the quantilesDoublesSketch aggregator merges the sketches from every segment matching the interval and dimension filter. This merge is the KLL merge — associative and error-bounded — so the result is a single sketch for the whole slice.
The quantilesDoublesSketchToQuantile post-aggregator reads p99 off the merged sketch. Changing fraction to 0.95 or 0.5 answers other percentiles from the same merged sketch — no re-query of raw data.
The net effect mirrors Spark's persistence trick: sketch at ingest, merge at query. Adding a region filter or widening the interval simply changes which segment sketches are merged, and the cost scales with segment count, not raw event count.

Output.

region	merged sketch (retained values)	p99_latency (ms)
us-east	KLL, k=256	512
eu-west	KLL, k=256	528
ap-south	KLL, k=256	560

Rule of thumb. In Druid, sketch the metric at ingestion and let queries merge segment sketches. It bounds storage (no raw rows), keeps all percentiles queryable from one merged sketch, and makes any dimension/time slice an O(segments) merge.

Worked example — Prometheus histogram vs summary, and DDSketch

Detailed explanation. The most common monitoring mistake is picking a Prometheus summary and then discovering you cannot compute a fleet-wide p99. Contrast histogram (mergeable) with summary (not), and show why DDSketch/t-digest backends exist. This is a design decision, not a coding one, so the "code" is the metric definitions and the query.

Histogram. Cumulative buckets per series; mergeable across instances; histogram_quantile estimates percentiles from merged buckets.
Summary. Client-side quantiles per instance; not mergeable; averaging p99s is statistically wrong.
DDSketch. Relative-error, mergeable quantile sketch — fleet p99 with a guaranteed bound.

Question. Show a histogram vs summary metric and explain why only the histogram (or DDSketch) yields a correct fleet-wide p99.

Input.

Metric type	Mergeable?	Fleet p99 correct?
Histogram	yes (sum buckets)	yes (bucket-bounded)
Summary	no	no (cannot merge quantiles)
DDSketch	yes	yes (relative-error bound)

Code.

# Prometheus HISTOGRAM — buckets are mergeable across instances
request_latency_seconds_bucket{le="0.05"}  ...
request_latency_seconds_bucket{le="0.1"}   ...
request_latency_seconds_bucket{le="0.5"}   ...
request_latency_seconds_bucket{le="1.0"}   ...
request_latency_seconds_bucket{le="+Inf"}  ...

# Fleet-wide p99 = histogram_quantile over the SUM of buckets across instances
histogram_quantile(0.99, sum by (le) (rate(request_latency_seconds_bucket[5m])))

# Prometheus SUMMARY — quantiles computed CLIENT-SIDE, per instance
request_latency_seconds{quantile="0.5"}   ...
request_latency_seconds{quantile="0.9"}   ...
request_latency_seconds{quantile="0.99"}  ...

# There is NO correct way to combine these across instances:
#   avg(request_latency_seconds{quantile="0.99"})   <-- STATISTICALLY WRONG
# The average of per-host p99s is not the fleet p99.

Step-by-step explanation.

A histogram exposes cumulative bucket counts (le="0.1" = "requests ≤ 100 ms"). Bucket counts are additive, so summing them across instances gives the fleet's bucket counts — a valid merge. histogram_quantile then interpolates p99 from the merged buckets.
The histogram's accuracy is bounded by bucket layout: too-coarse buckets around the tail give a fuzzy p99. You must choose bucket edges up front, which is hard for latency spanning several orders of magnitude — the same weakness equal-width histograms had in section 3.
A summary computes quantiles on each instance and exports the numbers. There is no bucket structure to merge — only pre-computed p50/p90/p99 per host. Averaging per-host p99s does not yield the fleet p99 (the p99 of a union is not the mean of the parts' p99s), so cross-instance percentiles are simply unavailable.
This is the trap: a summary looks convenient (exact-looking quantiles per host) but silently forecloses fleet aggregation. Teams discover it only when they try to build a global dashboard and find no correct query.
DDSketch (and t-digest backends) resolve this by exporting a mergeable, relative-error sketch instead of pre-computed quantiles. Fleet p99 is a merge of per-host DDSketches with a guaranteed relative-error bound — the histogram's mergeability plus tail accuracy the fixed buckets lack.

Output.

Approach	Fleet p99 query	Correct?	Tail accuracy
Histogram	`histogram_quantile` over summed buckets	yes	bucket-bounded
Summary	none exists	no	n/a (not mergeable)
DDSketch / t-digest	merge sketches, read p99	yes	relative-error, tail-tight

Rule of thumb. For any metric you will aggregate across instances, never use a Prometheus summary — its quantiles are not mergeable. Use a histogram (mergeable, bucket-bounded) or a DDSketch/t-digest backend (mergeable, relative-error) so fleet-wide p99 is a correct merge.

Systems interview question on production sketches

A senior interviewer might ask: "Design fleet-wide latency monitoring for 2,000 hosts where on-call must see p50/p95/p99/p999 sliceable by service, region, and endpoint, refreshed every 15 seconds, with a stated accuracy bound at p99 — and explain why a Prometheus summary would sink the design. Give me the per-host structure, the merge, the storage, and the accuracy guarantee."

Solution Using per-host mergeable DDSketch merged per slice at the collector

# ddsketch_monitoring.py — per-host relative-error sketch, merged per slice
# DDSketch: buckets on a logarithmic value scale -> guaranteed relative error.
import math

class DDSketch:
    def __init__(self, relative_accuracy: float = 0.01):
        self.alpha = relative_accuracy
        self.gamma = (1 + self.alpha) / (1 - self.alpha)   # log base
        self.buckets: dict[int, int] = {}                   # index -> count
        self.count = 0

    def _index(self, value: float) -> int:
        return math.ceil(math.log(value, self.gamma))       # log-scale bucket

    def add(self, value: float) -> None:
        if value <= 0:
            return
        self.buckets[self._index(value)] = self.buckets.get(self._index(value), 0) + 1
        self.count += 1

    def merge(self, other: "DDSketch") -> "DDSketch":
        assert abs(self.alpha - other.alpha) < 1e-12, "accuracy must match"
        out = DDSketch(self.alpha)
        out.buckets = dict(self.buckets)
        for idx, c in other.buckets.items():
            out.buckets[idx] = out.buckets.get(idx, 0) + c   # add bucket counts
        out.count = self.count + other.count
        return out

    def quantile(self, q: float) -> float:
        target, cum = q * self.count, 0
        for idx in sorted(self.buckets):
            cum += self.buckets[idx]
            if cum >= target:
                return 2 * self.gamma ** idx / (self.gamma + 1)  # bucket representative
        return 0.0

# Collector — group per-host DDSketches by (service, region, endpoint) and merge
from functools import reduce
from collections import defaultdict

def fleet_quantiles(host_sketches: list[tuple]) -> dict:
    groups: dict[tuple, list[DDSketch]] = defaultdict(list)
    for slice_key, sk in host_sketches:          # slice_key = (service, region, endpoint)
        groups[slice_key].append(sk)
    out = {}
    for slice_key, sketches in groups.items():
        merged = reduce(lambda a, b: a.merge(b), sketches)
        out[slice_key] = {p: merged.quantile(v) for p, v in
                          {"p50": .5, "p95": .95, "p99": .99, "p999": .999}.items()}
    return out

Step-by-step trace.

Layer	Component	Purpose
Host	DDSketch(alpha=0.01)	log-scale buckets; guaranteed 1% relative error
Wire	bucket map	small, mergeable; no raw latencies
Collector	merge per (service, region, endpoint)	add bucket counts -> one sketch per slice
Rollup	merge slices sharing a service	any union of slices = another merge
Dashboard	quantile(q) per slice	p50/p95/p99/p999 within 1% relative

Each of the 2,000 hosts keeps a DDSketch with alpha = 0.01, so every reported quantile is within 1% relative error by construction — the log-scale buckets guarantee it. The collector merges by adding bucket counts, so any slice (service × region × endpoint) or rollup is a fold of the relevant sketches, refreshed every 15 seconds. A Prometheus summary would have exported per-host quantiles that cannot be merged, making the sliceable fleet p99 impossible — which is exactly why the design uses a mergeable sketch.

Output:

Slice	p50 (ms)	p95 (ms)	p99 (ms)	p999 (ms)	Guarantee
checkout / us-east / POST	42	180	512	1170	±1% relative
checkout / eu-west / POST	45	190	528	1205	±1% relative
checkout / all (rollup)	43	185	519	1188	±1% relative
Raw latencies shipped	0	—	—	—	—

Why this works — concept by concept:

DDSketch log-scale buckets — buckets sized geometrically (gamma = (1+α)/(1−α)) give a guaranteed relative-error bound at every quantile, so p999 carries the same 1% promise as p50 — unlike a fixed-bucket histogram.
Additive bucket merge — merging two DDSketches adds their bucket counts, an associative and exact operation, so fleet aggregation across 2,000 hosts is a clean fold with no accuracy loss from merging.
Slice by any dimension — because merge is associative, "p99 for checkout across all regions" is a fold of the matching per-host sketches, computed at query time from stored bucket maps.
Summary would sink it — a Prometheus summary exports pre-computed per-host quantiles with no mergeable structure; averaging them is statistically wrong, so the sliceable fleet p99 the design requires would be uncomputable.
Cost — O(1) per observation (a log and a dict bump), O(distinct buckets) bytes per host (tiny, grows logarithmically with value range), O(hosts per slice) merges at query. The eliminated cost is storing and shipping raw latencies to compute exact percentiles across the fleet.

Data Structures
Topic — data-structures
Mergeable-summary and streaming problems

Practice →

Statistics
Topic — statistics
Percentile-monitoring and accuracy problems

Practice →

Cheat sheet — data-sketch recipes

Sketch by the question. Frequency of a key or top-K → Count-Min (+ heap) or Frequent-Items; distinct count → HyperLogLog; quantile/percentile → t-digest or KLL; membership → Bloom filter. The family is fixed by the question; the wrong family cannot be fixed with more memory. Say the question, then name the sketch.
Count-Min sizing. w = ⌈e/ε⌉ (width sets the additive error), d = ⌈ln(1/δ)⌉ (depth sets the confidence), memory = w·d counters. Guarantee: f(x) ≤ f̂(x) ≤ f(x) + ε·N with probability 1 − δ, one-sided (never under-counts). Size depth from δ first (cheap, logarithmic), then spend the rest on width.
Heavy hitters recipe. Count-Min + a size-K min-heap keyed by estimate; on each event update the sketch, take the post-update estimate, and offer it to the heap (O(log K)). Keep ε 5–10× below your φ threshold so the additive band can never lift a tail key over a genuine heavy hitter.
Conservative update vs merge. Conservative update (raise only the minimum counters) tightens single-node estimates on skewed streams but breaks exact mergeability. Standard update (increment all d) is exactly mergeable (add grids). Pick one: accuracy on a single node, or mergeability across shards — never both.
t-digest recipe. Buffer values, sort, merge into centroids under the scale function k(q) so centroids stay tiny at the tails and grow in the middle; compression sets the centroid budget (≈ O(compression)) and thus tail accuracy. Query by accumulating counts to q·N and interpolating between centroid means. Merge = union centroids + re-compress (order-insensitive).
Quantile sketch choice. t-digest for excellent empirical tail accuracy, tiny and mergeable, but no closed-form worst-case bound. KLL for a provable (ε, rank) guarantee (Apache DataSketches). Greenwald-Khanna for a classic deterministic ε-rank summary. Use KLL when an SLA needs a stated quantile error.
Additive vs relative error. Additive (± ε·N, Count-Min): negligible for heavy keys, fatal for rare keys — a heavy-hitters tool. Relative (± ε·true, HyperLogLog, t-digest tails): error scales with the answer — right for cardinality and tail quantiles. The error floor of Count-Min is ε·N; any answer below it is noise.
Mergeability contract. Fix the sketch config centrally (same w/d/seeds for Count-Min, same compression for t-digest, same alpha for DDSketch) and ship it to every worker. Merge is associative — add grids, max HLL registers, union+re-compress centroids, add DDSketch buckets — so any fold order and any dimensional slice work.
Spark. approx_count_distinct(col, rsd) (HLL++), percentile_approx(col, p, accuracy) for one-shot answers; persist a mergeable Apache DataSketches KLL/Theta/Frequent-Items column when you need rolling windows or cross-slice rollups — a stored sketch turns a re-scan into an O(days) merge.
Druid. Sketch metrics at ingestion (quantilesDoublesSketch = KLL, thetaSketch/HLLSketch = cardinality) so segments store sketches, not raw rows; queries merge segment sketches. All percentiles come from one merged sketch; any slice is O(segments).
Monitoring anti-pattern. Never use a Prometheus summary for anything you aggregate across instances — its client-side quantiles are not mergeable and averaging per-host p99s is wrong. Use a histogram (mergeable, bucket-bounded) or a DDSketch/t-digest backend (mergeable, guaranteed relative error) for fleet-wide p99.
The universal pattern. Sketch at ingest (per partition / segment / host), store the sketch object (discard or cold-store raw data), merge at query for any slice or rollup. One O(N) pass over raw data; every subsequent question is an O(sketch size) merge.

Frequently asked questions

What is a data sketch in one sentence?

A data sketch is a compact, probabilistic summary of a data stream that is built in a single pass, occupies space sublinear in the number of items (often kilobytes for billions of events), answers a specific class of query — frequency, quantile, cardinality, or membership — with a mathematically bounded error, and is mergeable so the summary of two streams can be combined from their summaries alone. The whole point is to trade an exactness you almost never need for a memory footprint that does not grow with the data. That combination — one pass, sublinear, bounded error, mergeable — is exactly what distributed aggregation at scale requires, which is why sketches, not exact aggregates, back Spark, Druid, and modern metrics systems.

Count-Min sketch vs HyperLogLog — what does each answer?

They answer different questions and are not interchangeable. A count-min sketch estimates the frequency of a given key (how many times did X appear) and, with a companion heap, the top-K heavy hitters; its error is additive (± ε·N), which makes it excellent for heavy keys and useless for rare ones. HyperLogLog estimates the number of distinct keys (cardinality) with a relative error of about 1.04/√m for m registers; it has no per-key counter at all, so it cannot tell you a frequency. Picking between them is a category decision driven by the question — "how many times" is Count-Min, "how many distinct" is HyperLogLog — and no amount of extra memory lets one answer the other's question.

How does t-digest get accurate p99s?

t-digest clusters streaming values into centroids — (mean, count) pairs — whose allowed size is governed by a scale function k(q) that keeps clusters small near the tails (q close to 0 or 1) and lets them grow in the middle. Because the tail centroids are tiny and numerous, the digest keeps fine-grained structure exactly where percentiles like p99 and p999 live, and a quantile query interpolates between those small centroids to land close to the true value. An equal-width histogram, by contrast, hard-codes its resolution by value range, so a single outlier stretches every bucket and blurs the tail. t-digest gives excellent relative accuracy at the tails for a couple of kilobytes; if you need a provable worst-case bound instead, use a KLL sketch.

Are sketches mergeable across shards?

Yes — mergeability is a defining property, and it is what makes sketches distributed-native. A count-min sketch merges by adding its counter grids element-wise (exact, provided identical width, depth, and hash seeds); HyperLogLog merges by taking the register-wise maximum; a t-digest merges by unioning centroids and re-compressing; DDSketch merges by adding bucket counts. All of these merges are associative, so a reducer can combine per-shard sketches in any order or in a tree, and any dimensional slice (region, service, time bucket) is just a different set of sketches to fold. The one hard requirement is identical configuration across shards — you fix it centrally and ship it to every worker. Note that Count-Min's conservative update optimisation forfeits exact mergeability, so distributed pipelines use standard update.

What error does a Count-Min sketch guarantee?

For width w = ⌈e/ε⌉ and depth d = ⌈ln(1/δ)⌉, a count-min sketch guarantees that the estimate f̂(x) satisfies f(x) ≤ f̂(x) ≤ f(x) + ε·N with probability at least 1 − δ, where N is the total stream weight. The error is one-sided (it never under-counts), additive (bounded by ε times the whole stream's weight, not the key's own count), and probabilistic (the bound can be exceeded with probability up to δ). The practical consequence is that Count-Min is a heavy-hitters tool: for a key whose true count is far above the ε·N noise floor the relative error is negligible, but for a rare key whose count is below that floor the estimate is dominated by collision noise. Size ε well below your heavy-hitter threshold so genuine heavy hitters stay clear of the floor.

Why not just use a Prometheus summary for percentiles?

Because a Prometheus summary computes its quantiles client-side on each instance and exports the finished numbers, which are not mergeable — you cannot combine per-host p99s into a fleet-wide p99 (the p99 of a union is not the average of the parts' p99s). The moment you need a percentile sliceable across instances, a summary silently makes it impossible, and teams usually discover this only when they try to build a global dashboard. Use a Prometheus histogram instead — its cumulative buckets are mergeable across instances, so histogram_quantile over summed buckets gives a correct (bucket-bounded) fleet p99 — or a DDSketch/t-digest backend, which exports a mergeable sketch with a guaranteed relative-error bound and tail accuracy the fixed buckets lack. mergeable summaries are the whole reason percentile monitoring scales.

Practice on PipeCode

Drill the data-structures practice library → for the Count-Min, Bloom filter, min-heap top-K, and streaming-summary problems senior interviewers love.
Sharpen the estimation axis on the statistics practice library → for quantiles, percentiles, error bounds, and confidence questions.
Stress the distinct-count axis on the cardinality practice library → for HyperLogLog, frequency estimation, and sketch-sizing scenarios.
Stack these against PipeCode's broader 450+ data-engineering catalogue to anchor the sketch-by-question decision matrix against real graded inputs.

Lock in data-sketch muscle memory

Docs explain the structures. PipeCode drills explain the decision — when a Count-Min sketch is blind to rare keys, when a t-digest's tail centroids earn their memory, when additive error disqualifies a sketch, when a Prometheus summary quietly makes fleet p99 impossible. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice data-structure problems →
Practice statistics problems →

Consistent Hashing: How Distributed Systems Partition Data

Gowtham Potureddi — Mon, 07 Sep 2026 15:59:51 +0000

consistent hashing is the algorithm that decides which machine holds which piece of your data in every distributed cache, key-value store, and sharded database you will ever operate — and it is the answer to the question that breaks the naive design: "what happens to all your data when you add or remove a server?" The moment you spread data across more than one node you have to answer where each key lives, and the obvious answer — take the hash of the key, divide by the number of servers, keep the remainder — works beautifully right up until the day you change the number of servers, at which point almost every key you own suddenly belongs to a different machine. That single failure is the reason a whole family of partitioning schemes exists, and understanding it is the difference between a cache tier that scales smoothly and one that melts down the instant you touch it.

This guide is the walkthrough you wished existed the first time an interviewer drew a circle on the whiteboard and asked "how would you shard this so growing the cluster doesn't move everything?" It builds the idea in layers: first the modulo scheme and the precise reason resizing it triggers a rebalancing storm, then the hash ring that limits key movement to a single arc, then virtual nodes that turn a lumpy ring into an even key distribution, then the replication walk that places copies on the next distinct nodes and the bounded-load variant that tames hot partitions, and finally how the real systems — Amazon's Dynamo, Apache Cassandra, and Redis Cluster — actually implement partitioning and sharding at scale. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the hash table practice library →, sharpen the fundamentals on the data structures practice library →, and connect it to storage on the database practice library →.

On this page

Why modulo hashing fails on resize
The hash ring
Virtual nodes and load balancing
Replication and bounded loads
In practice — Dynamo, Cassandra, Redis Cluster
Cheat sheet — consistent hashing recipes
Frequently asked questions
Practice on PipeCode

1. Why modulo hashing fails on resize

`node = hash(key) % N` is the partitioning scheme everyone reaches for first — and it moves almost every key the instant N changes

The one-sentence invariant: modulo hashing assigns a key to a node by taking the remainder of the key's hash divided by the node count, which distributes keys evenly while the count is fixed but reshuffles a fraction of roughly (N-1)/N of all keys the moment the count changes — turning a routine capacity change into a cluster-wide data migration and, for a cache, a near-total miss storm. The scheme is not wrong because it distributes badly; with a decent hash function hash(key) % N spreads keys almost perfectly across N buckets. It is wrong because the bucket a key lands in depends on N itself, so the assignment function changes shape whenever you scale — and scaling is exactly the operation a distributed system exists to support.

The three properties any partitioning scheme must have.

Balance. Keys should spread evenly across nodes so no single machine holds a disproportionate share of the data or the traffic. Modulo hashing nails this while N is constant — a good hash makes every remainder equally likely.
Determinism. Any client, given only the key and the current membership, must compute the same owner without coordination. Modulo is perfectly deterministic — hash(key) % N needs no shared state beyond N.
Stability under membership change. When a node joins or leaves, the number of keys that change owner should be proportional to the capacity that changed — ideally about K/N keys for one node out of N. This is the property modulo hashing catastrophically lacks, and it is the entire reason consistent hashing was invented.

The resize catastrophe — why N → N+1 is a rebalancing storm.

The math. A key keeps its owner across a resize only if hash(key) % N == hash(key) % N+1. For independent, uniform hashes those two remainders agree for only a small set of residues — going from 4 nodes to 5, exactly the hashes whose value mod 20 falls in {0,1,2,3} stay put, which is 4 of 20, or 20%. The other 80% of keys move.
The general case. For a jump from N to N+1, the expected fraction of keys that stay is roughly 1/(N+1) in the worst intuition and never better than a small constant; in practice you should assume that any change to N re-partitions the vast majority of your keyspace.
The blast radius for a cache. Every moved key is a guaranteed cache miss on the new owner. Add one node to a hot Memcached tier and you convert ~80% of reads into origin-database hits simultaneously — a thundering herd that can take down the very database the cache was protecting.
The blast radius for a store. For a stateful store (a sharded SQL tier, a key-value database), a moved key is not just a miss — it is data on the wrong machine. Resizing means physically copying most of your dataset between nodes while serving traffic, which is slow, risky, and often requires a maintenance window.

The 2026 reality — modulo still ships, but only where membership never changes.

Where it survives. Fixed-size partition maps — "always exactly 256 partitions" — where you shard by hash(key) % 256 and then map partitions to physical nodes through a separate lookup table. Here N (the partition count) is constant forever; you scale by reassigning partitions to machines, not by changing the modulus. Redis Cluster's 16384 slots are exactly this trick.
Where it kills you. Any design where the modulus is the live server count. "We hash by % number_of_cache_boxes" is the anti-pattern; the first autoscaling event re-partitions the fleet.
What interviewers listen for. Can you quantify the remap fraction (~80% for 4→5) rather than hand-wave "a lot move"? Do you name the failure mode (cache stampede / migration storm)? Do you immediately reach for consistent hashing or a fixed partition map as the fix? Naming the number is the senior signal.

Worked example — measuring the modulo remap fraction

Detailed explanation. The fastest way to internalise why modulo hashing fails is to measure it. Hash a million keys into N buckets, then re-hash the same keys into N+1 buckets, and count how many changed bucket. The number is shockingly large and it is the single most persuasive artifact you can put on a whiteboard.

Setup. One million synthetic keys, a stable hash (MD5 → integer), buckets N = 4 then N = 5.
Measurement. Count keys whose bucket differs between the two runs.
Expectation. Around 80% of keys move — matching the 4/20 residue argument above.

Question. Empirically measure the fraction of keys that change owner when a modulo-hashed cluster grows from 4 nodes to 5.

Input.

Parameter	Value
Keys	1,000,000 synthetic (`user:0 … user:999999`)
Hash	MD5 hex → int
Before	`hash % 4`
After	`hash % 5`
Metric	fraction where before != after

Code.

import hashlib

def node_for(key: str, n: int) -> int:
    """Modulo hashing: which of n nodes owns this key."""
    h = int(hashlib.md5(key.encode()).hexdigest(), 16)
    return h % n

keys = [f"user:{i}" for i in range(1_000_000)]

before = {k: node_for(k, 4) for k in keys}     # 4-node cluster
after  = {k: node_for(k, 5) for k in keys}     # add one node -> 5

moved = sum(1 for k in keys if before[k] != after[k])
print(f"keys moved: {moved:,} ({moved / len(keys):.1%})")
# keys moved: 800,3xx (80.0%)

Step-by-step explanation.

node_for is the entire modulo scheme: hash the key to a big integer, take it modulo the node count. With MD5 the hashes are uniform, so each of the n remainders is equally likely — balance is fine.
before records every key's owner in the 4-node world; after records the owner after a single node is added, making it a 5-node world. Nothing about the keys changed — only the modulus.
The comparison counts keys whose owner differs. Because hash % 4 and hash % 5 are effectively independent for a uniform hash, agreement happens only for the residues where they coincide — 4 out of every 20, or 20%.
The printed result is ~80%. Adding one node to a four-node cluster relocates four fifths of the keyspace. The fraction does not improve much at larger N: 100 → 101 still moves ~99% of keys, because hash % 100 and hash % 101 share almost no structure.
For a cache, those 800,000 moved keys are 800,000 guaranteed misses the instant you deploy the new node — the mechanical origin of the cache-stampede outage.

Output.

Transition	Keys moved	Fraction	Cache effect
4 → 5 nodes	~800,000 / 1,000,000	~80%	80% miss storm
5 → 4 nodes	~800,000 / 1,000,000	~80%	80% miss storm
100 → 101 nodes	~990,000 / 1,000,000	~99%	near-total miss storm

Rule of thumb. Never let the number of live servers be the modulus. If you can whiteboard the "~80% of keys move on 4→5" number in ten seconds, you have already justified reaching for consistent hashing or a fixed partition map.

Worked example — the cache-stampede failure that follows a resize

Detailed explanation. The remap fraction is abstract until you trace what it does to a production cache. Walk through the sequence of events when an on-call engineer adds a Memcached node to a % N tier during a traffic spike — the exact incident consistent hashing was designed to prevent.

The tier. 4 Memcached nodes, owner = hash(key) % 4, serving 200k reads/sec at a 98% hit rate. The 2% of misses (4k/sec) hit the origin database, which is comfortably sized for that.
The change. Add a fifth node to relieve memory pressure. The modulus becomes 5.
The consequence. 80% of keys now route to a different node than the one holding their cached value — and those nodes have never seen the key.

Question. Trace the origin-database load in the 60 seconds after a fifth node is added to a modulo-hashed cache tier.

Input.

Parameter	Value
Read rate	200,000 reads/sec
Steady-state hit rate	98%
Steady-state origin load	4,000 reads/sec
Keys remapped on 4→5	~80%

Code.

# Illustrative model of origin load right after the resize
READS_PER_SEC       = 200_000
STEADY_HIT_RATE     = 0.98
REMAPPED_FRACTION   = 0.80     # 4 -> 5 nodes

# Right after resize: remapped keys are guaranteed misses on their new owner.
# Non-remapped keys still hit at the steady rate.
miss_rate_after = REMAPPED_FRACTION * 1.0 + (1 - REMAPPED_FRACTION) * (1 - STEADY_HIT_RATE)
origin_after    = READS_PER_SEC * miss_rate_after

print(f"steady origin load : {READS_PER_SEC * (1 - STEADY_HIT_RATE):,.0f} reads/sec")
print(f"post-resize origin : {origin_after:,.0f} reads/sec")
print(f"spike factor       : {origin_after / (READS_PER_SEC * (1 - STEADY_HIT_RATE)):.0f}x")
# steady origin load : 4,000 reads/sec
# post-resize origin : 160,400 reads/sec
# spike factor       : 40x

Step-by-step explanation.

In steady state only 2% of reads miss the cache, so the origin database sees 4,000 reads/sec — a load it was provisioned for with headroom.
The instant the modulus changes from 4 to 5, ~80% of keys hash to a node that has never cached them. Every read for those keys misses, regardless of how hot the key is.
The blended miss rate jumps to 0.80 * 100% + 0.20 * 2% ≈ 80.4%, so origin load leaps to ~160,000 reads/sec — a 40× spike landing all at once.
The origin database, sized for 4k/sec, saturates; latency climbs; the application's own retries add more load; the cache nodes fill slowly because the database can't answer fast enough to repopulate them. This is the thundering herd.
The fix is not "add capacity to the origin" — it is to stop remapping 80% of keys on every resize. Consistent hashing caps the remap at ~1/5 of keys for a 4→5 change, turning a 40× spike into a bounded, survivable bump.

Output.

Phase	Miss rate	Origin load	Status
Steady state	2%	4,000/sec	healthy
Immediately post-resize (modulo)	~80%	~160,000/sec	40× spike; likely outage
Immediately post-resize (consistent hashing)	~22%	~44,000/sec	bounded; recoverable

Rule of thumb. A cache's blast radius on resize is directly proportional to the remap fraction. Choose a partitioning scheme by asking "how many keys move when membership changes?" before you ask anything about raw throughput.

Worked example — SQL sharding by modulo and the migration it forces

Detailed explanation. Modulo hashing is just as common — and just as dangerous — in stateful stores. A team shards a users table across 8 Postgres instances by user_id % 8. It works for a year, then they need a ninth shard. Walk through why that is a full-dataset migration, not a config change.

The scheme. shard = user_id % 8; the application routes each query to the computed shard.
The growth. Storage on the 8 shards is full; the team provisions a ninth.
The problem. Changing the modulus to 9 relocates ~89% of rows — they must be physically moved between databases while the system stays online.

Question. Quantify the data movement when a modulo-sharded 8-instance SQL tier grows to 9 instances, and describe the safe migration.

Input.

Parameter	Value
Rows	800,000,000 users
Before	`user_id % 8`
After	`user_id % 9`
Row size	~1 KB

Code.

-- The routing rule the application hard-codes (the anti-pattern)
-- shard_id = user_id % 8   ->  changes to user_id % 9 on resize

-- Estimate rows that change shard on 8 -> 9 (run per current shard)
WITH sampled AS (
    SELECT user_id,
           (user_id % 8) AS old_shard,
           (user_id % 9) AS new_shard
    FROM   users
)
SELECT count(*)                                              AS total_rows,
       count(*) FILTER (WHERE old_shard <> new_shard)        AS rows_moving,
       round(100.0 * count(*) FILTER (WHERE old_shard <> new_shard)
             / count(*), 1)                                  AS pct_moving
FROM   sampled;
-- total_rows | rows_moving  | pct_moving
-- 800000000  | 711111111    |      88.9

Step-by-step explanation.

The routing rule user_id % 8 lives in application code, so every read and write already assumes 8 shards. Changing to % 9 is a code change and a data change — both must land atomically or queries route to the wrong shard.
The SQL counts how many rows would compute a different shard under the new modulus. As with the cache case, x % 8 and x % 9 rarely agree, so ~88.9% of rows must relocate.
Moving 711 million rows of ~1 KB each is roughly 711 GB of cross-instance transfer, executed while the tier serves live traffic. During the move, a row may exist on both its old and new shard, so the application needs dual-read / dual-write logic to stay correct.
The migration typically runs as: stand up shard 9, dual-write new data under the % 9 rule, backfill the 711M relocating rows in batches, verify, then cut reads over. It is weeks of work and a standing outage risk — all to add one shard.
Had the tier used consistent hashing (or a fixed 256-partition map decoupled from instance count), adding a ninth instance would move only ~1/9 of rows (~89 GB) and require no change to the routing rule — just a membership update.

Output.

Approach	Rows moved (8→9)	Bytes moved	Routing-code change?
Modulo (`% instance_count`)	~711M (88.9%)	~711 GB	yes
Consistent hashing	~89M (~11%)	~89 GB	no
Fixed 256-partition map	0 keys re-hashed; ~1/9 partitions reassigned	~89 GB	no

Rule of thumb. In a stateful store, the remap fraction is data you have to physically copy. Decouple the hash modulus from the live instance count — use consistent hashing or a fixed partition map — so scaling copies a proportional slice, not the whole dataset.

System design interview question on partitioning

A senior interviewer often opens with: "You are running a 12-node Memcached tier sharded by hash(key) % 12, serving 500k reads/sec at a 97% hit rate. Traffic is growing, so you need to add 4 nodes. Walk me through exactly what happens to your keys and your origin database the moment you deploy, quantify it, and tell me what you would do instead."

Solution Using consistent hashing instead of modulo

# Baseline: modulo hashing. Show the damage, then contrast with a ring.
import hashlib

def h(s: str) -> int:
    return int(hashlib.md5(s.encode()).hexdigest(), 16)

def modulo_owner(key: str, n: int) -> int:
    return h(key) % n

keys = [f"key:{i}" for i in range(1_000_000)]

# --- Modulo: 12 -> 16 nodes ---
mod_before = {k: modulo_owner(k, 12) for k in keys}
mod_after  = {k: modulo_owner(k, 16) for k in keys}
mod_moved  = sum(1 for k in keys if mod_before[k] != mod_after[k])

# --- Consistent hashing preview: a ring of node tokens ---
import bisect

class Ring:
    def __init__(self, nodes):
        self.ring = []          # sorted (position, node) by position
        self._pos = []          # parallel sorted positions for bisect
        for node in nodes:
            self.add(node)

    def add(self, node):
        p = h(node) % (2**32)
        bisect.insort(self.ring, (p, node))
        self._pos = [p for p, _ in self.ring]

    def owner(self, key):
        p = h(key) % (2**32)
        i = bisect.bisect(self._pos, p) % len(self.ring)   # first node clockwise
        return self.ring[i][1]

ring_before = Ring([f"node-{i}" for i in range(12)])
own_before  = {k: ring_before.owner(k) for k in keys}
ring_after  = Ring([f"node-{i}" for i in range(16)])       # add 4 nodes
own_after   = {k: ring_after.owner(k) for k in keys}
ring_moved  = sum(1 for k in keys if own_before[k] != own_after[k])

print(f"modulo 12->16 moved: {mod_moved/len(keys):.1%}")     # ~92.x%
print(f"ring   12->16 moved: {ring_moved/len(keys):.1%}")    # ~25%

Step-by-step trace.

Step	Modulo path	Consistent-hashing path
Owner formula	`hash(key) % N`	first node clockwise on a `2^32` ring
Add 4 nodes (12→16)	modulus 12 → 16	drop 4 new tokens onto the ring
Keys that move	~92% (almost all)	~25% (the four new arcs only)
Origin miss spike	~92% of reads	~25% of reads
Data migration (if stateful)	~92% of dataset	~25% of dataset
Routing change	new modulus everywhere	membership update only

After the change, the modulo tier remaps roughly 92% of the million keys — a near-total miss storm — while the ring remaps only about 25%, the keys that fall on the four arcs the new tokens carved out. The origin database sees a 4× bump instead of a 30× one, and it recovers.

Output:

Metric	Modulo 12→16	Consistent hashing 12→16
Fraction of keys moved	~92%	~25%
Origin load multiplier	~30×	~4×
Survivable without an outage?	usually no	usually yes
Ideal fraction for +4 of 12	—	~4/16 = 25%

Why this works — concept by concept:

Modulo couples ownership to N — because the owner is hash(key) % N, every key's owner is a function of the total node count. Change the count and you change the function for essentially every key. The scheme has no memory of where a key used to live.
The ring decouples ownership from N — each key hashes to a fixed point on a 2^32 circle and is owned by the next node clockwise. Adding a node inserts one new point; it can only steal the keys in the arc between it and its predecessor. Every other key's owner is unchanged.
Bounded remap = bounded blast radius — moving ~k/(N+k) of keys when you add k nodes to N means the cache-miss spike and the data migration are proportional to the capacity you added, not to the whole cluster. That is the property that makes scaling routine.
Cost — modulo lookup is O(1) but resize is O(all keys). Ring lookup is O(log N) per key via binary search over the sorted token positions, and resize is O(K/N) keys moved plus O(N) to rebuild the position index. You trade a hair of lookup cost for a scheme that survives membership change — always the right trade in a system that scales.

Hash Table
Topic — hash-table
Hash-table and hashing problems

Practice →

Data Structures Topic — data-structures Data-structure design problems

Practice →

2. The hash ring

Map keys and nodes onto one circle, and ownership becomes "the next node clockwise" — so adding a node only disturbs a single arc

The one-sentence invariant: the hash ring places both keys and nodes onto the same circular hash space — usually the integers [0, 2^32) or [0, 2^160) wrapped end-to-end — and defines a key's owner as the first node encountered walking clockwise from the key's position, so inserting or removing a node changes ownership only for the keys in the arc between that node and its clockwise predecessor, bounding the movement to about K/N keys. This is the core idea introduced by Karger and colleagues in 1997 for web caching, and it is the mechanism underneath every system in section 5. Once you see the circle, the modulo catastrophe looks obviously avoidable: the whole problem was that ownership depended on N; on the ring it depends only on positions, and adding a node adds exactly one position.

The construction — three steps to a working ring.

Hash the nodes onto the circle. For each node, compute hash(node_id) mod 2^32 and record that integer as the node's token — its position on the ring. Keep the tokens in a sorted structure.
Hash the keys onto the same circle. A key's position is hash(key) mod 2^32, in the identical space. Keys and nodes now live on one number line that wraps from 2^32 - 1 back to 0.
Own by successor. The owner of a key is the first node token at or after the key's position, wrapping past the top of the ring back to the smallest token if necessary. This "first node clockwise" rule is a successor search — a binary search over the sorted tokens.

Why membership change is cheap.

Adding a node. A new token drops in at some position. The only keys that change owner are those between the new token and the previous token clockwise-behind it — that arc used to belong to the next node clockwise and now belongs to the newcomer. Every other arc is untouched.
Removing a node. The departing node's arc merges into its clockwise successor; only that node's former keys move, and they all move to exactly one place. No other key is affected.
The expected fraction. With N roughly-evenly-spaced tokens, one node owns about 1/N of the circle, so adding or removing a node moves about K/N keys — proportional to the one unit of capacity you changed, which is the whole point.

The data structure — a sorted token array plus binary search.

Store. A sorted list of (position, node) pairs, or two parallel arrays (positions and nodes) so you can binary-search positions directly.
Lookup. bisect (binary search) for the key's position, take the index modulo the token count to wrap, return that node. O(log N) per lookup.
Mutation. Insert or delete a token and keep the array sorted — O(N) to rebuild a parallel index, or O(log N) amortised with a balanced tree / sorted container. Membership changes are rare compared to lookups, so an array is usually fine.

The catch a bare ring still has.

Uneven arcs. With only a handful of randomly-placed tokens, the arcs between them are far from equal — one node can easily own twice its fair share of the circle, and therefore twice the data and traffic. Randomness at small N is lumpy.
Removal doubles a neighbor. When a node leaves, all of its keys pile onto a single successor, momentarily doubling that node's load rather than spreading the departed load across the survivors.
The fix. Both problems are solved by virtual nodes — giving each physical node many tokens so the law of large numbers smooths the arcs. That is section 3. A bare ring is the right mental model but not the production configuration.

Worked example — build a minimal hash ring

Detailed explanation. A production-shaped ring is about forty lines of Python: hash nodes to tokens, keep them sorted, and resolve a key by binary-searching for its successor token. Build it from scratch and confirm the successor rule with a hand-checkable example.

Space. 2^32 positions.
Store. Parallel sorted arrays: _keys (positions) and _nodes (owners).
Lookup. bisect for the successor, wrap with modulo.

Question. Implement a hash ring supporting add_node, remove_node, and get_node(key) with O(log N) lookup.

Input.

Operation	Argument
add_node	"A", "B", "C"
get_node	"photo:42"
Ring space	2^32

Code.

import bisect
import hashlib

class HashRing:
    def __init__(self):
        self._pos: list[int] = []      # sorted token positions
        self._node: list[str] = []     # node at the same index

    @staticmethod
    def _hash(s: str) -> int:
        return int(hashlib.md5(s.encode()).hexdigest(), 16) % (2**32)

    def add_node(self, node: str) -> None:
        p = self._hash(node)
        i = bisect.bisect(self._pos, p)
        self._pos.insert(i, p)
        self._node.insert(i, node)

    def remove_node(self, node: str) -> None:
        p = self._hash(node)
        i = bisect.bisect_left(self._pos, p)
        if i < len(self._pos) and self._node[i] == node:
            self._pos.pop(i)
            self._node.pop(i)

    def get_node(self, key: str) -> str:
        if not self._pos:
            raise ValueError("empty ring")
        p = self._hash(key)
        i = bisect.bisect(self._pos, p) % len(self._pos)   # first token clockwise
        return self._node[i]

ring = HashRing()
for n in ("A", "B", "C"):
    ring.add_node(n)

print(ring.get_node("photo:42"))     # -> whichever node is first clockwise

Step-by-step explanation.

_hash folds any string into [0, 2^32). Nodes and keys use the same function into the same space, which is what makes "first node clockwise" meaningful.
add_node finds the insertion point with bisect and splices the token into both parallel arrays, keeping them sorted so lookups stay O(log N).
remove_node locates the exact token by position and confirms the node identity before removing — guarding against a hash collision removing the wrong entry.
get_node binary-searches for the first token strictly after the key's position. If the key hashes past the largest token, bisect returns len(self._pos), and % len wraps the index back to token 0 — that wrap is the "circle" made concrete.
Because ownership is a pure function of token positions, calling get_node again after adding a fourth node changes the answer only for keys that fall on the new node's arc — the property section 1 was missing.

Output.

Key	Position (mod 2^32)	First token clockwise	Owner
photo:42	0x4c…	token for "C"	C
photo:7	0x9a…	token for "A"	A
photo:99	0xf1… (past all)	wraps to smallest	(wrapped node)

Rule of thumb. Keep nodes and keys in one hash space and resolve ownership with a single binary search for the successor token. The wrap-around modulo on the index is the one line beginners forget — and the one that makes it a ring instead of a line.

Worked example — prove only K/N keys move when a node joins

Detailed explanation. The value of the ring is entirely in its stability, so measure it. Put 100,000 keys on a 4-node ring, record owners, add a fifth node, and count movers. The number should hover near 1/5 — the arc the newcomer steals — not the 80% modulo inflicted.

Before. 4 nodes, 100,000 keys, record owners.
After. Add node "E", re-resolve, count changes.
Expectation. ~20% move, and every mover goes to the new node.

Question. Empirically confirm that adding one node to a 4-node ring moves about 1/5 of keys, all onto the new node.

Input.

Parameter	Value
Nodes before	A, B, C, D
Node added	E
Keys	100,000

Code.

ring = HashRing()
for n in ("A", "B", "C", "D"):
    ring.add_node(n)

keys = [f"obj:{i}" for i in range(100_000)]
before = {k: ring.get_node(k) for k in keys}

ring.add_node("E")
after = {k: ring.get_node(k) for k in keys}

moved     = [k for k in keys if before[k] != after[k]]
to_new    = [k for k in moved if after[k] == "E"]
print(f"moved   : {len(moved)/len(keys):.1%}")     # ~20%
print(f"to node E: {len(to_new)}/{len(moved)}")    # all of them

Step-by-step explanation.

The 4-node ring gives each node roughly a quarter of the circle (subject to token randomness), so each owns ~25,000 of the 100,000 keys.
before snapshots every key's owner. Adding "E" inserts exactly one new token; the ring's other three tokens do not move.
"E" captures the arc between its token and the previous token clockwise-behind it — an arc that previously belonged to whichever node was next clockwise. That arc is about 1/5 of the circle once there are five tokens.
Re-resolving shows ~20% of keys changed owner, and the check confirms every single mover now resolves to "E". No key moves between the old nodes — the ring never disturbs an arc it did not touch.
Contrast with modulo, where adding a node reshuffles ~80%. The ring's movement is bounded, one-directional (onto the newcomer), and proportional to the capacity added.

Output.

Metric	Value
Keys moved	~20,000 (~20%)
Movers going to E	100%
Keys moved between old nodes	0
Ideal fraction (1 of 5)	20%

Rule of thumb. A correct ring moves only keys onto (or off) the node whose membership changed — never between untouched nodes. If your measurements show cross-node churn on a simple add, your successor logic or wrap-around is wrong.

Worked example — the lumpy-arc problem on a bare ring

Detailed explanation. Before celebrating, measure the balance of a bare ring. With one token per node, arc sizes are random and uneven, so some nodes own far more than their fair share. Quantifying the imbalance is what motivates virtual nodes in the next section.

Setup. 8 nodes, one token each, 1,000,000 keys.
Measure. Per-node key count; compare max to the fair share (1/8 = 12.5%).
Expectation. The busiest node holds well over its share — often 1.5–2× the fair amount.

Question. Measure the load imbalance of an 8-node bare ring and express the hottest node as a multiple of its fair share.

Input.

Parameter	Value
Nodes	8, one token each
Keys	1,000,000
Fair share	125,000 (12.5%)

Code.

from collections import Counter

ring = HashRing()
for i in range(8):
    ring.add_node(f"node-{i}")

keys  = [f"k:{i}" for i in range(1_000_000)]
load  = Counter(ring.get_node(k) for k in keys)

fair  = 1_000_000 / 8
hi    = max(load.values())
lo    = min(load.values())
print(f"hottest: {hi:,} ({hi/fair:.2f}x fair)")
print(f"coldest: {lo:,} ({lo/fair:.2f}x fair)")
# hottest: ~2xx,xxx (1.6-2.0x fair)
# coldest: ~ xx,xxx (0.3-0.6x fair)

Step-by-step explanation.

Eight random tokens carve the circle into eight arcs of random length. Arc length maps directly to key count, so uneven arcs mean uneven load.
Counting owners across a million keys reveals the spread. It is common for the hottest node to own 1.6–2.0× its fair share and the coldest to own half — a 3–4× ratio between busiest and idlest.
That imbalance is a real cost: the hot node needs more memory and CPU, hits capacity first, and becomes the tail-latency bottleneck for the whole tier.
Worse, removing a node dumps its entire arc onto one successor, transiently doubling that node — the ring gives you cheap movement but not, by itself, even distribution.
The cure is many small tokens per node: with V tokens each, the arcs a node owns average out and the imbalance shrinks like 1/sqrt(V). That is exactly virtual nodes, next.

Output.

Configuration	Hottest node	Coldest node	Max/min ratio
8 nodes, 1 token each	~1.8× fair	~0.45× fair	~4×
8 nodes, 150 tokens each	~1.05× fair	~0.95× fair	~1.1×

Rule of thumb. A bare ring solves movement but not balance. Never ship one token per node in production — measure the max-to-fair ratio, and if it exceeds ~1.1×, add virtual nodes until it doesn't.

System design interview question on the hash ring

A senior interviewer might ask: "Implement consistent hashing for a distributed cache. Support adding and removing nodes and looking up the owner of a key. Then tell me the time complexity of each operation, prove that adding a node moves only about 1/N of keys, and identify the one problem your basic ring still has."

Solution Using a sorted-token ring with binary-search successor lookup

import bisect
import hashlib

class ConsistentHashRing:
    """Bare ring: one token per node. O(log N) lookup, O(K/N) keys move on resize."""

    def __init__(self, nodes: list[str] | None = None):
        self._pos: list[int] = []
        self._node: list[str] = []
        for n in nodes or []:
            self.add_node(n)

    @staticmethod
    def _hash(s: str) -> int:
        return int(hashlib.sha1(s.encode()).hexdigest(), 16) % (2**32)

    def add_node(self, node: str) -> None:
        p = self._hash(node)
        i = bisect.bisect(self._pos, p)
        self._pos.insert(i, p)          # keep positions sorted
        self._node.insert(i, node)

    def remove_node(self, node: str) -> None:
        p = self._hash(node)
        i = bisect.bisect_left(self._pos, p)
        if i < len(self._pos) and self._node[i] == node:
            self._pos.pop(i)
            self._node.pop(i)

    def get_node(self, key: str) -> str:
        if not self._pos:
            raise ValueError("ring is empty")
        p = self._hash(key)
        i = bisect.bisect(self._pos, p) % len(self._pos)   # successor, wrapped
        return self._node[i]

Step-by-step trace.

Input	Step	Result
`add_node("A")`	hash A → 0x21…, bisect-insert	ring = [A]
`add_node("B")`	hash B → 0xC4…, bisect-insert	ring = A, B
`add_node("C")`	hash C → 0x88…, bisect-insert	ring = [A, C, B]
`get_node("user:5")`	pos 0x53…, bisect → index of C	owner = C
`get_node("user:9")`	pos 0xF0… past all, `% len` wraps	owner = A
`remove_node("C")`	pop C's token	C's arc merges into B

Walking the trace: three nodes hash to three positions and sort by position (not insertion order). A key resolves by binary-searching for the first token at or after it; user:9 hashes past the largest token so the index wraps to the smallest, node A — the circle in action. Removing C hands its whole arc to B, the tell-tale sign that a bare ring needs virtual nodes for balance.

Output:

Operation	Complexity	Keys moved
`get_node`	O(log N)	—
`add_node`	O(N) index shift	~K/N onto the new node
`remove_node`	O(N) index shift	departed node's ~K/N onto one successor
Rebalance on +1 of N	—	~1/(N+1) of keys

Why this works — concept by concept:

Successor lookup on a circle — defining ownership as "first token clockwise" means a key's owner is determined by position, not by node count. bisect finds that successor in O(log N); the modulo on the index turns the sorted line into a wrap-around ring.
Sorted token array — keeping positions sorted lets lookups binary-search and lets inserts/deletes touch only the local neighborhood conceptually (an array pays O(N) to shift, a tree pays O(log N)). Lookups vastly outnumber membership changes, so the simple array is usually the right call.
Bounded movement — a new token can only capture the arc between itself and its predecessor, so adding a node moves ~1/(N+1) of keys and touches no other node's arc. This is the exact property modulo lacked and the reason the ring scales.
One remaining flaw — one token per node gives random, uneven arcs (load imbalance) and dumps a removed node's whole arc onto a single successor. The fix is virtual nodes, which the bare ring sets up but does not provide.
Cost — O(log N) lookup, O(N) membership mutation for the array form, and O(K/N) keys relocated per node change. Compared with modulo's O(all keys) resize, the ring converts scaling from a full migration into a proportional, survivable one.

Data Structures
Topic — data-structures
Design a data structure with binary search

Practice →

Hash Table Topic — hash-table Hashing and key-lookup problems

Practice →

3. Virtual nodes and load balancing

Give each machine many small tokens instead of one, and random lumpiness averages out into an even key distribution

The one-sentence invariant: virtual nodes (vnodes) place each physical node on the ring at many positions — typically dozens to a few hundred tokens per node — so that the arcs a node owns are the sum of many independent random pieces, which by the law of large numbers evens out the key distribution (load variance shrinks roughly like 1/sqrt(V) for V tokens per node) and spreads a departed node's keys across many successors instead of dumping them all on one. Virtual nodes are not a different algorithm; they are the ring from section 2 with each node hashed under several distinct labels. This one change turns the lumpy, 4×-imbalanced bare ring into the smooth, near-uniform distribution real systems depend on — and it is why Dynamo, Cassandra, and every serious library default to many tokens per node.

How virtual nodes are constructed.

Many labels per node. For physical node A and V virtual nodes, hash A#0, A#1, … A#(V-1) onto the ring — V separate tokens that all map back to A. A lookup that lands on any of them resolves to A.
A token-to-node map. Alongside the sorted positions, keep a map from each token position to its owning physical node. Successor lookup is unchanged; you just dereference the token to its physical node at the end.
Interleaving. Because the V tokens per node are scattered independently, tokens from different physical nodes interleave all around the ring. Every node ends up owning many small arcs distributed across the circle rather than one big contiguous slice.

Why this balances load.

The averaging argument. With one token, a node's load is a single random arc — high variance. With V tokens, its load is the sum of V independent arcs; the sum concentrates around the mean, and the relative standard deviation falls like 1/sqrt(V). A hundred-ish tokens per node typically brings the busiest node within a few percent of fair share.
Graceful removal. When a node leaves, each of its V arcs merges into a different successor, so its load is redistributed across many nodes — no single successor doubles. Symmetrically, a joining node steals many small arcs from many nodes, so no single donor is drained.
Graceful addition. A new node with V tokens picks up V small arcs from around the ring, immediately taking its fair share without a lopsided handoff.

Heterogeneous capacity — weighting by token count.

The lever. Token count is the capacity knob. A machine with twice the memory or CPU gets twice the tokens and therefore owns about twice the keyspace. Consistent hashing supports weighted nodes for free — just scale V per node.
The practice. Cassandra exposes this as num_tokens; give beefier hosts a larger value. Client libraries like ketama let you assign weights that translate into proportional token counts.
The caution. Weighting is coarse at low token counts. To make a node reliably own 1.5× its neighbors, you need enough tokens that the arcs average out — another reason V is in the hundreds, not the single digits.

Choosing V — the trade-off.

Higher V. Smoother load, gentler rebalancing, finer capacity weighting — but a larger sorted token array (N × V entries), slightly slower membership updates, and more metadata to gossip around the cluster.
Lower V. Less memory and metadata, faster updates — but lumpier load and coarser weighting.
Typical values. Libraries use 100–200 replicas (tokens) per node; Cassandra historically defaulted num_tokens to 256, later lowering the recommendation to 16 with an improved allocation algorithm. The right number balances the load-smoothness you need against the token-array size you can afford.

Worked example — add virtual nodes and measure the smoothing

Detailed explanation. Extend the ring so each physical node contributes V tokens, then measure how imbalance falls as V grows. This is the single most convincing demonstration that vnodes matter — the max-to-fair ratio drops visibly with every increase in V.

Change. Hash node#0 … node#(V-1) for each node; map tokens back to the physical node.
Measure. Hottest node as a multiple of fair share, for V in {1, 10, 50, 200}.
Expectation. Imbalance shrinks roughly like 1/sqrt(V).

Question. Implement a virtual-node ring and show how the hottest node's load approaches fair share as V increases.

Input.

Parameter	Value
Physical nodes	8
Keys	1,000,000
V values	1, 10, 50, 200

Code.

import bisect
import hashlib
from collections import Counter

class VNodeRing:
    def __init__(self, vnodes: int = 150):
        self.v = vnodes
        self._pos: list[int] = []
        self._node: list[str] = []

    @staticmethod
    def _hash(s: str) -> int:
        return int(hashlib.md5(s.encode()).hexdigest(), 16) % (2**32)

    def add_node(self, node: str) -> None:
        for r in range(self.v):
            p = self._hash(f"{node}#{r}")      # V distinct tokens per node
            i = bisect.bisect(self._pos, p)
            self._pos.insert(i, p)
            self._node.insert(i, node)         # token -> physical node

    def get_node(self, key: str) -> str:
        p = self._hash(key)
        i = bisect.bisect(self._pos, p) % len(self._pos)
        return self._node[i]

keys = [f"k:{i}" for i in range(1_000_000)]
for v in (1, 10, 50, 200):
    ring = VNodeRing(vnodes=v)
    for i in range(8):
        ring.add_node(f"node-{i}")
    load = Counter(ring.get_node(k) for k in keys)
    fair = 1_000_000 / 8
    print(f"V={v:>3}: hottest {max(load.values())/fair:.2f}x fair")
# V=  1: hottest ~1.80x fair
# V= 10: hottest ~1.28x fair
# V= 50: hottest ~1.12x fair
# V=200: hottest ~1.05x fair

Step-by-step explanation.

add_node now inserts V tokens per physical node, each hashed from a distinct label node#r. All V tokens map back to the same physical node via the parallel _node array.
get_node is unchanged — it finds the successor token and returns its physical node. The vnode machinery is invisible at lookup time; only the token density changed.
At V=1 the ring is the bare ring from section 2, and the hottest node sits near 1.8× fair — the lumpiness we measured before.
As V rises, each node's load is the sum of more independent arcs, so the distribution tightens. By V=50 the hottest node is within ~12% of fair; by V=200 within ~5%.
The improvement tracks 1/sqrt(V): quadrupling V roughly halves the excess imbalance. Beyond a couple hundred tokens the returns diminish and the token array's memory cost starts to matter.

Output.

V (tokens/node)	Hottest node	Excess over fair	Token array size
1	~1.80×	~80%	8
10	~1.28×	~28%	80
50	~1.12×	~12%	400
200	~1.05×	~5%	1,600

Rule of thumb. Pick V so the hottest node stays within ~5–10% of fair share — usually 100–200 tokens per node. Remember imbalance falls like 1/sqrt(V), so doubling smoothness costs 4× the tokens.

Worked example — graceful rebalancing on node removal

Detailed explanation. The second gift of vnodes is that removing a node spreads its load across many survivors instead of one. Measure where a removed node's keys go, with and without virtual nodes, to see the difference.

Setup. 6 nodes, compare V=1 and V=150.
Action. Remove one node; track which survivors absorb its keys.
Expectation. V=1 dumps everything on one successor; V=150 spreads it near-evenly across all five.

Question. When a node is removed, how is its load redistributed with one token per node versus 150 tokens per node?

Input.

Parameter	Value
Nodes	6
Removed	node-3
Keys	600,000
V compared	1 vs 150

Code.

def redistribution(v: int) -> dict[str, float]:
    ring = VNodeRing(vnodes=v)
    for i in range(6):
        ring.add_node(f"node-{i}")
    keys   = [f"k:{i}" for i in range(600_000)]
    before = {k: ring.get_node(k) for k in keys}

    # remove node-3 by rebuilding without it
    ring2 = VNodeRing(vnodes=v)
    for i in range(6):
        if i != 3:
            ring2.add_node(f"node-{i}")
    after = {k: ring2.get_node(k) for k in keys}

    orphans = [k for k in keys if before[k] == "node-3"]
    dest    = Counter(after[k] for k in orphans)
    total   = len(orphans)
    return {n: c / total for n, c in dest.items()}

print("V=1  :", {k: round(v, 2) for k, v in redistribution(1).items()})
# V=1  : {'node-4': 1.0}                      # all keys to a single successor
print("V=150:", {k: round(v, 2) for k, v in redistribution(150).items()})
# V=150: {'node-0': 0.21, 'node-1': 0.19, 'node-2': 0.20, 'node-4': 0.20, 'node-5': 0.20}

Step-by-step explanation.

orphans are the keys that node-3 owned before removal — the load that must go somewhere.
With V=1, node-3 had one token owning one contiguous arc; removing it merges that entire arc into the single next token clockwise. One survivor inherits 100% of node-3's load and momentarily runs at ~2× capacity.
With V=150, node-3 had 150 small tokens scattered around the ring; each arc merges into a different neighbor. The 150 handoffs land on all five survivors roughly equally (~20% each).
Even redistribution means no survivor spikes — the cluster degrades from 6 nodes to 5 smoothly, exactly the behavior you want during a failure or a planned decommission.
The same mechanism runs in reverse on addition: a joining node's 150 tokens each steal a small arc from a different neighbor, so it reaches fair share without starving any single donor.

Output.

V	Survivors absorbing load	Max single survivor share	Failure behavior
1	1 of 5	100%	one node doubles; hotspot
150	5 of 5	~21%	smooth; no hotspot

Rule of thumb. Virtual nodes are what make failures boring. If losing one node overloads exactly one neighbor, you shipped a bare ring — add tokens so a departed node's load fans out across all survivors.

Worked example — weighting a heterogeneous cluster

Detailed explanation. Real fleets mix instance sizes. Consistent hashing handles this by giving bigger machines more tokens. Show a 3-node cluster where one node has double the capacity and confirm it earns double the keys.

Cluster. node-big (weight 2), node-a (weight 1), node-b (weight 1).
Tokens. V × weight tokens per node.
Expectation. node-big owns ~50% of keys; the others ~25% each.

Question. Configure a weighted ring so a double-capacity node owns twice the keyspace, and verify the split.

Input.

Node	Weight	Base V	Tokens
node-big	2	150	300
node-a	1	150	150
node-b	1	150	150

Code.

class WeightedRing(VNodeRing):
    def add_weighted(self, node: str, weight: int) -> None:
        for r in range(self.v * weight):        # more tokens = more keyspace
            p = self._hash(f"{node}#{r}")
            i = bisect.bisect(self._pos, p)
            self._pos.insert(i, p)
            self._node.insert(i, node)

ring = WeightedRing(vnodes=150)
ring.add_weighted("node-big", weight=2)
ring.add_weighted("node-a",   weight=1)
ring.add_weighted("node-b",   weight=1)

keys = [f"k:{i}" for i in range(1_000_000)]
load = Counter(ring.get_node(k) for k in keys)
for n, c in sorted(load.items()):
    print(f"{n:>9}: {c/1_000_000:.1%}")
# node-big : 50.0%
# node-a   : 25.0%
# node-b   : 25.0%

Step-by-step explanation.

add_weighted multiplies the token count by the node's weight, so node-big drops 300 tokens versus 150 for the others.
Since keyspace ownership is proportional to token count once V is large enough to average out, node-big's 300 of 600 total tokens claim ~50% of the ring.
node-a and node-b split the remaining half, ~25% each — exactly proportional to their weight-1 share.
This makes capacity planning a single integer per node. Replace a node with a bigger box? Raise its weight. Consistent hashing routes proportionally more keys to it with no other change.
The averaging caveat applies: at low base V the weighted split is noisy. Keep enough tokens that the realized split lands within a couple percent of the intended weights.

Output.

Node	Intended share	Realized share
node-big (weight 2)	50%	~50.0%
node-a (weight 1)	25%	~25.0%
node-b (weight 1)	25%	~25.0%

Rule of thumb. Express capacity as token count. Weighting a node by giving it proportionally more virtual nodes is the cleanest way to run a heterogeneous fleet — but keep base V high enough that the realized split matches the intended weights.

System design interview question on load balancing with virtual nodes

A senior interviewer might ask: "Your consistent-hashing cache tier is showing one node at 90% CPU while the others idle at 40%, even though the ring should balance them. Diagnose the likely cause, explain how virtual nodes fix it, tell me how many tokens per node you would use and why, and describe what happens to load distribution when that hot node eventually fails."

Solution Using virtual nodes to smooth load and rebalancing

import bisect
import hashlib
from collections import Counter

class BalancedRing:
    """Virtual-node ring. V tokens per node smooths load and rebalancing."""

    def __init__(self, vnodes: int = 150):
        self.v = vnodes
        self._pos: list[int] = []
        self._tok: list[str] = []          # physical node for each token

    @staticmethod
    def _hash(s: str) -> int:
        return int(hashlib.sha1(s.encode()).hexdigest(), 16) % (2**32)

    def add_node(self, node: str, weight: int = 1) -> None:
        for r in range(self.v * weight):
            p = self._hash(f"{node}#{r}")
            i = bisect.bisect(self._pos, p)
            self._pos.insert(i, p)
            self._tok.insert(i, node)

    def get_node(self, key: str) -> str:
        p = self._hash(key)
        i = bisect.bisect(self._pos, p) % len(self._pos)
        return self._tok[i]

    def load_report(self, keys: list[str]) -> dict[str, float]:
        c = Counter(self.get_node(k) for k in keys)
        total = sum(c.values())
        return {n: v / total for n, v in c.items()}

Step-by-step trace.

Step	Action	Effect
Diagnose	one token per node	random arcs → one node owns ~1.8× share
Fix	set V = 150 tokens/node	each node = sum of 150 arcs → ~1.05× share
Verify	`load_report` on 1M keys	hottest node within ~5% of fair
Weight	`add_node(big, weight=2)`	big node owns proportionally 2×
Fail	remove the (former) hot node	150 arcs fan out across all survivors

Tracing the fix: the imbalance was almost certainly one-token-per-node (or too few tokens), so a single node drew a lopsided arc. Raising V to 150 makes every node's load the average of 150 independent arcs, collapsing the hottest node from ~1.8× to ~1.05× fair. When that node later fails, its 150 tokens each merge into a different neighbor, so the load redistributes evenly instead of doubling one survivor.

Output:

Metric	Before (V=1)	After (V=150)
Hottest node	~1.8× fair	~1.05× fair
Load std-dev (relative)	high	~1/sqrt(150) of the V=1 spread
Removal behavior	1 successor doubles	5+ survivors share evenly
Capacity weighting	coarse / noisy	proportional to token count

Why this works — concept by concept:

Many tokens per node — hashing each physical node under V distinct labels scatters it across the ring, so a node owns many small arcs instead of one big random one. Ownership resolution is unchanged; only token density grows.
Variance falls like 1/sqrt(V) — a node's load becomes the sum of V independent arcs, and sums of independent random variables concentrate. That is the statistical reason ~150 tokens brings the hottest node within a few percent of fair.
Fan-out on membership change — because a node's tokens are spread out, adding or removing it touches many neighbors with small handoffs rather than one neighbor with a huge one. Failures and scale events become smooth.
Token count = capacity knob — weighting a node by its token count gives free, proportional support for heterogeneous hardware, exposed in real systems as num_tokens or per-node weights.
Cost — the token array grows to N × V entries and gossip/metadata scales with it, so lookups stay O(log(N·V)) and memory and update cost rise linearly in V. The trade — a few thousand extra sorted integers for near-perfect balance and boring failures — is almost always worth it.

Data Structures
Topic — data-structures
Load-balancing and distribution problems

Practice →

Hash Table Topic — hash-table Hashing and bucket-distribution problems

Practice →

4. Replication and bounded loads

Store each key on the next several distinct nodes clockwise, and cap any node's share so a hot key can't melt one machine

The one-sentence invariant: replication on a ring stores each key not just on its owner but on the next N-1 distinct physical nodes walking clockwise — the "preference list" — so the data survives node loss, while consistent hashing with bounded loads adds a per-node capacity cap of (1+ε)·average and spills any key that would exceed it to the next node clockwise, guaranteeing no node exceeds its cap even under a skewed, adversarial key distribution. Replication is what makes the ring a durable store rather than a routing table, and bounded loads is the refinement that closes consistent hashing's last real weakness: a single scorching-hot key or a burst of correlated keys can still overwhelm one node even with perfect vnode balance, because balance is about count of keys, not their traffic.

The preference list — replication done right on a ring.

The walk. For replication factor RF, a key's replica set is its owner plus the next RF-1 nodes clockwise. This ordered list is Dynamo's "preference list": the first node is the primary, the rest are backups in ring order.
Distinct physical nodes. With virtual nodes, the next few tokens clockwise may belong to the same physical machine. The preference-list walk must skip tokens whose physical node is already in the set, so RF=3 means three different machines, not three tokens.
Rack / zone awareness. Production systems extend the skip rule: don't just require distinct nodes, require distinct failure domains (racks, availability zones). A key with RF=3 should land in three zones so one zone outage can't take all copies.
Read/write quorums. With RF replicas you tune consistency by requiring W acknowledged writes and R reads such that W + R > RF guarantees a read sees the latest write. This is the Dynamo-style quorum built directly on the preference list.

Why balance-by-count is not enough — the hot-key problem.

The gap. Virtual nodes balance the number of keys per node, assuming every key gets similar traffic. Real workloads are skewed: a celebrity's profile, a viral post, or a trending product can make one key carry a thousand times the traffic of an average key.
The failure. That hot key lives on one owner (and its replicas). Perfect count-balance does nothing for it — the owner saturates while the rest of the ring idles. Vnodes solved lumpy arcs, not lumpy traffic.
The mitigations. Options include key-splitting (append a random suffix to fan a hot key across nodes, at the cost of scatter-gather reads), caching the hot key in front of the store, and — the systematic answer — bounding per-node load so overflow spills to neighbors.

Consistent hashing with bounded loads (CHBL).

The cap. Define average load avg = total_load / N and a per-node capacity cap = ceil((1+ε)·avg) for a small ε (say 0.25). No node may hold more than cap items.
The assignment. Place a key at its ring owner if that node is under cap; otherwise walk clockwise to the next node with spare capacity. The key "spills forward" until it finds room.
The guarantee. Introduced by Mirrokni, Thorup, and Zadimoghaddam, CHBL proves every node stays at or below (1+ε)·avg while the movement on insertion/deletion remains bounded — you get the ring's cheap rebalancing and a hard load ceiling. Google and others use it for load balancing where a backend can be genuinely overloaded.

The trade-offs of bounding.

Lookup cost. A bounded lookup may probe several nodes clockwise before finding capacity, so worst-case lookup is longer than the plain successor search — usually still small for a modest ε.
Locality. Spilled keys are no longer on their "natural" owner, so the mapping is load-dependent, not purely positional. Clients must consult the same capacity view or a coordinator to resolve a spilled key.
Choosing ε. Small ε gives tight balance but more spilling and probing; large ε allows more imbalance but less movement. ε around 0.25 is a common middle ground.

Worked example — the preference-list replica walk

Detailed explanation. Implement replica resolution: from a key's owner, walk clockwise collecting distinct physical nodes until you have RF of them. This is the exact routine a Dynamo-style coordinator runs to find where to write and read a key.

Input. A vnode ring, a key, RF=3.
Rule. Owner first, then next distinct physical nodes clockwise.
Output. An ordered list of 3 distinct machines.

Question. Given a virtual-node ring, return the ordered replica set (preference list) of a key for RF=3, skipping duplicate physical nodes.

Input.

Parameter	Value
Physical nodes	node-0 … node-5
V	150
RF	3
Key	"cart:8821"

Code.

def preference_list(ring: BalancedRing, key: str, rf: int) -> list[str]:
    """Owner + next distinct physical nodes clockwise, length rf."""
    p = ring._hash(key)
    start = bisect.bisect(ring._pos, p) % len(ring._pos)

    result: list[str] = []
    n = len(ring._pos)
    i = start
    while len(result) < rf and len(result) < len(set(ring._tok)):
        node = ring._tok[i]
        if node not in result:          # skip tokens of an already-chosen node
            result.append(node)
        i = (i + 1) % n                 # step clockwise, wrapping the ring
    return result

ring = BalancedRing(vnodes=150)
for i in range(6):
    ring.add_node(f"node-{i}")

print(preference_list(ring, "cart:8821", rf=3))
# e.g. ['node-2', 'node-5', 'node-0']   # 3 distinct machines, ring order

Step-by-step explanation.

Resolve the key's start index exactly like a normal lookup — the successor token position on the ring.
Walk clockwise from there, reading the physical node of each token. Because vnodes interleave, consecutive tokens often belong to different machines, but not always.
The if node not in result guard is the crucial correctness check: it skips any token whose physical node is already in the replica set, so we collect RF distinct machines, not RF tokens.
Stepping with (i + 1) % n wraps around the top of the ring, so the walk continues past the largest token back to the smallest — the preference list is a circular scan.
The loop also stops if the ring has fewer than RF physical nodes (the len(set(...)) guard), avoiding an infinite loop on a tiny cluster.

Output.

Position	Role	Node
1	primary (owner)	node-2
2	replica	node-5
3	replica	node-0

Rule of thumb. The preference-list walk must count distinct physical nodes, not tokens — forgetting the skip means RF=3 can put two or three "replicas" on the same machine, and one failure loses multiple copies. Extend the skip to distinct racks/zones for real durability.

Worked example — bounded-load assignment with a capacity cap

Detailed explanation. Implement CHBL's spill rule: compute a per-node cap, and when a key's owner is full, walk clockwise to the next node with room. Show that no node exceeds (1+ε)·avg even when many keys target the same arc.

Cap. cap = ceil((1+ε)·(total_keys / N)), ε=0.25.
Assign. Owner if under cap, else next node clockwise with spare capacity.
Check. Max node load stays at or below cap.

Question. Assign keys with a bounded-load cap and confirm no node exceeds (1+ε)·avg, even under skew.

Input.

Parameter	Value
Nodes	5
Keys	10,000 (some correlated to one arc)
ε	0.25
Cap	ceil(1.25 × 2000) = 2500

Code.

import math

def assign_bounded(ring: BalancedRing, keys: list[str], n_nodes: int,
                   eps: float = 0.25) -> dict[str, str]:
    cap = math.ceil((1 + eps) * (len(keys) / n_nodes))
    load: Counter = Counter()
    placement: dict[str, str] = {}
    ntok = len(ring._pos)

    for key in keys:
        i = bisect.bisect(ring._pos, ring._hash(key)) % ntok
        # walk clockwise until we find a node under cap
        for _ in range(ntok):
            node = ring._tok[i]
            if load[node] < cap:
                load[node] += 1
                placement[key] = node
                break
            i = (i + 1) % ntok
        else:
            raise RuntimeError("cluster over capacity")
    print("cap:", cap, "| max node load:", max(load.values()))
    return placement

ring = BalancedRing(vnodes=150)
for i in range(5):
    ring.add_node(f"node-{i}")

keys = [f"k:{i}" for i in range(10_000)]
assign_bounded(ring, keys, n_nodes=5, eps=0.25)
# cap: 2500 | max node load: 2500      # no node exceeds the cap

Step-by-step explanation.

cap is the hard ceiling: 25% above the average of 2,000 keys/node, so 2,500. No node may hold more.
Each key resolves to its natural ring owner first. If that node is below cap, the key lands there — the common, fast path identical to plain consistent hashing.
If the owner is already at cap, the loop walks clockwise, checking each subsequent node's live load, until it finds one with room. The key "spills forward."
Because total keys (10,000) fit under total capacity (5 × 2,500 = 12,500), the walk always finds room; the else on the for guards the impossible-over-capacity case.
The printed max load equals the cap exactly for the busy arcs and less elsewhere — the guarantee holds: every node is at or below (1+ε)·avg regardless of how skewed the key targeting was.

Output.

Node	Load	Under cap (2500)?
node-0	2,500	yes (at cap)
node-1	2,100	yes
node-2	2,500	yes (at cap)
node-3	1,400	yes
node-4	1,500	yes

Rule of thumb. Bounded loads trade a little locality and a few extra probes for a hard guarantee that no node exceeds (1+ε)·avg. Reach for it when your workload is skewed enough that count-balance (vnodes) isn't sufficient — hot keys, correlated bursts, or backends that genuinely fall over when overloaded.

Worked example — mitigating a single scorching-hot key

Detailed explanation. Sometimes one key is so hot that even its owner can't serve it. Bounded loads don't help a single key (a key can't be split across nodes without changing its identity). The systematic fix is key-splitting: fan the hot key into S sub-keys across the ring and scatter-gather. Walk through it.

Detect. Per-key request counters flag celebrity:1 as carrying 40% of traffic.
Split. Route reads to celebrity:1#{0..S-1}, each on a different node; writes fan out to all S.
Trade-off. Reads become scatter-gather across S nodes; writes cost S×.

Question. Design a mitigation for a single hot key that saturates its owner despite balanced vnodes, and state the cost.

Input.

Parameter	Value
Hot key	celebrity:1 (40% of read traffic)
Ring	6 nodes, V=150, RF=3
Split factor S	6

Code.

import random

def hot_read(ring: BalancedRing, key: str, split: int) -> str:
    """Read one of S shards of a hot key; each shard lives on a different node."""
    shard = random.randrange(split)          # spread reads across shards
    return ring.get_node(f"{key}#{shard}")

def hot_write_targets(ring: BalancedRing, key: str, split: int) -> set[str]:
    """A write must update every shard so all reads are consistent."""
    return {ring.get_node(f"{key}#{s}") for s in range(split)}

ring = BalancedRing(vnodes=150)
for i in range(6):
    ring.add_node(f"node-{i}")

reads_to  = Counter(hot_read(ring, "celebrity:1", split=6) for _ in range(60_000))
print("read fan-out:", dict(reads_to))
print("write hits :", hot_write_targets(ring, "celebrity:1", split=6))
# read fan-out: spread across ~6 nodes, ~10k each
# write hits : {'node-0','node-1','node-2','node-3','node-4','node-5'}

Step-by-step explanation.

A single key celebrity:1 maps to exactly one owner; no amount of vnode balancing or load-bounding relocates the traffic to that one key, because splitting it would change the key.
Key-splitting creates S derived keys celebrity:1#0 … celebrity:1#5, which hash to (typically) S different owners — spreading the read load across many nodes.
hot_read picks a random shard per read, so 60,000 reads fan out ~10,000 to each of the six shards' owners — the hot spot is gone on the read path.
hot_write_targets shows the cost: a write must update all S shards to keep reads consistent, so writes are S× more expensive and must be applied to every shard atomically or with a versioning scheme.
Key-splitting is the tool of last resort for genuine single-key hotspots; for merely skewed-but-many-keys workloads, bounded loads (previous example) is cheaper because it needs no application-level fan-out.

Output.

Path	Behavior	Cost
Read (split S=6)	random shard → 6 owners	~1/6 load each
Write (split S=6)	update all 6 shards	6× write amplification
No split	all traffic to 1 owner	owner saturates

Rule of thumb. Bounded loads fix many skewed keys; key-splitting fixes one scorching key. Split only the keys you must, because every split turns a cheap point read into a scatter-gather and multiplies write cost by the split factor.

System design interview question on replication and hot keys

A senior interviewer might ask: "Design the replication scheme for a consistent-hashing key-value store with RF=3 across three availability zones. Show how you pick the three replicas on the ring, how you guarantee they land in distinct zones, and how you keep one viral hot key from saturating a single replica. Then explain how bounded loads change the picture and what they cost you."

Solution Using preference-list replication with zone awareness and bounded loads

import bisect, hashlib, math
from collections import Counter

class ReplicatedRing:
    """Vnode ring with zone-aware RF replication and bounded-load spill."""

    def __init__(self, vnodes: int = 150):
        self.v = vnodes
        self._pos: list[int] = []
        self._tok: list[str] = []          # physical node per token
        self.zone: dict[str, str] = {}     # node -> availability zone

    @staticmethod
    def _hash(s: str) -> int:
        return int(hashlib.sha1(s.encode()).hexdigest(), 16) % (2**32)

    def add_node(self, node: str, zone: str) -> None:
        self.zone[node] = zone
        for r in range(self.v):
            p = self._hash(f"{node}#{r}")
            i = bisect.bisect(self._pos, p)
            self._pos.insert(i, p); self._tok.insert(i, node)

    def replicas(self, key: str, rf: int) -> list[str]:
        """RF distinct nodes in distinct zones, clockwise from the owner."""
        n = len(self._pos)
        i = bisect.bisect(self._pos, self._hash(key)) % n
        chosen: list[str] = []
        seen_zone: set[str] = set()
        for _ in range(n):
            node = self._tok[i]
            z = self.zone[node]
            if node not in chosen and z not in seen_zone:
                chosen.append(node); seen_zone.add(z)
                if len(chosen) == rf:
                    break
            i = (i + 1) % n
        return chosen

ring = ReplicatedRing(vnodes=150)
for i in range(6):
    ring.add_node(f"node-{i}", zone=f"zone-{i % 3}")   # 3 zones, 2 nodes each

print(ring.replicas("order:5501", rf=3))
# e.g. ['node-4', 'node-2', 'node-0']  -> zone-1, zone-2, zone-0 (all distinct)

Step-by-step trace.

Step	Action	Result
Resolve owner	bisect key → successor token	start at node-4 (zone-1)
Replica 2	walk clockwise, need new zone	node-2 (zone-2) accepted
Skip	next token node-5 (zone-1)	rejected: zone-1 already used
Replica 3	continue clockwise	node-0 (zone-0) accepted
Stop	3 distinct nodes, 3 distinct zones	preference list complete
Hot key	split celebrity key S ways / bound loads	traffic fans out; no node over cap

Tracing it: the owner is the first token clockwise; the walk then accepts nodes only if both the node and its zone are new, guaranteeing RF=3 copies in three distinct availability zones so one zone outage leaves two live replicas. For a viral key, key-splitting fans reads across shards; for broadly skewed load, bounded loads cap each node at (1+ε)·avg and spill the overflow forward.

Output:

Concern	Mechanism	Guarantee
Durability	preference list, RF=3	survives 2 node losses
Zone failure	distinct-zone skip	survives 1 full zone outage
Skewed load	bounded loads, ε=0.25	no node over 1.25× avg
Single hot key	key-splitting, factor S	~1/S load per shard
Consistency	W + R > RF quorum	reads see latest write

Why this works — concept by concept:

Preference list = owner + successors — replicating to the next RF-1 distinct nodes clockwise reuses the ring's ordering for free; the same walk that finds the owner finds the backups, and it stays cheap under membership change.
Distinct-zone skip — requiring each replica in a new failure domain converts RF=3 into "survives one zone outage." Without it, three replicas can share a rack and a single power event loses all copies.
Quorum W + R > RF — tuning acknowledged writes and reads so their sizes overlap guarantees a read intersects the latest write, giving Dynamo-style tunable consistency on top of the preference list.
Bounded loads — a (1+ε)·avg cap with clockwise spill closes the count-vs-traffic gap that vnodes leave open, guaranteeing a hard ceiling per node even under adversarial skew, at the cost of a few extra probes and some lost locality.
Cost — replication multiplies storage by RF and write cost by the write-quorum size; bounded loads add probing and a shared capacity view; key-splitting multiplies a hot key's write cost by S. Each mechanism buys a specific guarantee — durability, zone survival, a load ceiling, hotspot relief — and you pay only for the ones your workload needs.

Database
Topic — database
Replication and partitioning design problems

Practice →

Data Structures Topic — data-structures Ring and interval design problems

Practice →

5. In practice — Dynamo, Cassandra, Redis Cluster

The same ring shows up as Dynamo preference lists, Cassandra token ranges, and Redis Cluster hash slots — with revealing differences

The one-sentence invariant: every production partitioning system is a variation on the ring — Amazon Dynamo uses a token ring with vnodes and preference-list replication; Apache Cassandra assigns each node many token ranges (num_tokens) and replicates by NetworkTopologyStrategy; and Redis Cluster deliberately does not use a live ring but instead maps keys to one of 16384 fixed hash slots via CRC16(key) mod 16384, then assigns slots to nodes — a fixed partition map that gets consistent-hashing-like rebalancing by moving slots, not by re-hashing keys. Knowing which system uses which shape, and why Redis chose fixed slots over a classic ring, is exactly the depth a senior systems interview probes.

Amazon Dynamo — the paper that popularised the pattern.

Token ring + vnodes. Dynamo hashes keys with MD5 onto a ring and gives each physical node many virtual nodes for balance and smooth rebalancing — the exact construction from sections 2 and 3.
Preference list. Each key's replicas are the next RF distinct physical nodes clockwise, skipping duplicates and spanning data centers — section 4's walk.
Tunable quorum. Dynamo exposes N (replicas), W (write quorum), R (read quorum); W + R > N gives read-your-writes. This is the origin of the "NWR" model many stores copied.
Anti-entropy. Because writes can be accepted during partitions (AP under CAP), Dynamo reconciles divergent replicas with vector clocks and Merkle-tree-based repair — the price of high availability.

Apache Cassandra — a Dynamo-shaped open-source store.

Token ranges. Each node owns a set of token ranges on the ring. With vnodes enabled (num_tokens > 1), a node owns many small ranges scattered around the ring; the partitioner (default Murmur3Partitioner) hashes the partition key to a token.
num_tokens. The vnode count per node. Older Cassandra defaulted to 256; modern versions recommend 16 paired with an allocation algorithm (allocate_tokens_for_local_replication_factor) that keeps ranges even without needing hundreds of tokens.
Replication strategy. SimpleStrategy walks the ring for RF nodes; NetworkTopologyStrategy places replicas in distinct racks across each data center — the zone-aware preference list from section 4, configured per keyspace.
Consistency levels. Per-query ONE, QUORUM, LOCAL_QUORUM, ALL implement the read/write quorum knobs on top of the ring.

Redis Cluster — fixed slots, not a live ring.

16384 hash slots. A key maps to a slot by CRC16(key) mod 16384. Slots — not keys — are assigned to master nodes. The slot count is fixed forever, so the "modulus" never changes even as nodes come and go.
Why fixed slots. Redis wanted resharding to be a bounded, explicit operation: to move data you migrate whole slots between nodes, and clients learn the new slot→node map via MOVED/ASK redirections. This sidesteps live-ring token math and makes the cluster state a compact 16384-entry map every node gossips.
Hash tags. {user1000}.followers and {user1000}.profile hash by only the {...} substring, forcing related keys into the same slot so multi-key operations and transactions work. This is the escape hatch for co-locating keys the plain scheme would scatter.
The distinction to state in an interview. Redis Cluster is not classic consistent hashing — it is a fixed partition map. It achieves the same goal (bounded key movement on resize) by decoupling the modulus (16384) from the node count and reassigning slots, which is the "fixed partition map" pattern from section 1.

Client-side rings — ketama and Maglev.

Ketama. The de-facto consistent-hashing algorithm for Memcached client libraries: 160 tokens per node on a 2^32 ring, MD5-based, so any client computes the same owner without a coordinator. This is consistent hashing living entirely in the client.
Maglev. Google's load balancer hashing: builds a fixed-size lookup table (a permutation-based scheme) that gives near-perfect balance and minimal disruption on backend change, optimised for the packet-routing hot path where even O(log N) per lookup is too slow.
When client-side wins. Caches and L4/L7 load balancers, where you want every client/proxy to agree on placement with zero coordination and sub-microsecond lookups.

Worked example — Cassandra vnodes and replication config

Detailed explanation. Configure a Cassandra keyspace for even distribution and zone-aware replication across two data centers, and explain what each setting does on the ring.

Node. num_tokens sets vnode count per node.
Keyspace. NetworkTopologyStrategy with per-DC replication factor.
Query. LOCAL_QUORUM for low-latency, zone-tolerant reads/writes.

Question. Write the Cassandra node and keyspace configuration for a 2-DC cluster with RF=3 per DC and even token distribution, and justify each choice.

Input.

Setting	Value
num_tokens	16
Partitioner	Murmur3Partitioner
Strategy	NetworkTopologyStrategy
RF per DC	3

Code.

# cassandra.yaml (per node)
num_tokens: 16                         # 16 vnodes/node: even ranges, modern default
allocate_tokens_for_local_replication_factor: 3
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
endpoint_snitch: GossipingPropertyFileSnitch   # teaches the ring about racks/DCs

-- Keyspace: zone-aware replication, 3 copies in each of two data centers
CREATE KEYSPACE shop
WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'dc-east': 3,
  'dc-west': 3
};

-- Table partitioned by a high-cardinality key for even token spread
CREATE TABLE shop.orders (
    customer_id  bigint,
    order_id     timeuuid,
    total_cents  bigint,
    PRIMARY KEY ((customer_id), order_id)   -- partition key -> token
);

-- Read/write at LOCAL_QUORUM: 2 of 3 in the local DC, no cross-DC latency
-- (set per session/statement, e.g. in the driver)

Step-by-step explanation.

num_tokens: 16 gives each node 16 token ranges — enough vnodes, with the allocation algorithm, to keep ranges even without the metadata cost of the old 256 default.
allocate_tokens_for_local_replication_factor: 3 tells Cassandra to choose token positions that balance ownership for RF=3 specifically, tightening distribution beyond random token placement.
Murmur3Partitioner hashes the partition key (customer_id) to a 64-bit token; the node owning that token's range is the primary, and NetworkTopologyStrategy walks the ring for replicas in distinct racks.
NetworkTopologyStrategy with dc-east: 3, dc-west: 3 places three replicas in each data center, each in a distinct rack where possible — the zone-aware preference list, so a rack or even a DC can fail without data loss.
LOCAL_QUORUM requires 2 of the 3 local-DC replicas to ack, giving strong-enough consistency and durability while avoiding cross-DC round-trips on the hot path.

Output.

Setting	Effect on the ring
num_tokens=16	16 vnodes/node → even ranges
Murmur3Partitioner	partition key → 64-bit token
NetworkTopologyStrategy	RF=3 per DC in distinct racks
LOCAL_QUORUM	2/3 local acks; no cross-DC latency

Rule of thumb. In Cassandra, pick a high-cardinality partition key (so tokens spread), set num_tokens with the allocation algorithm for even ranges, and use NetworkTopologyStrategy + LOCAL_QUORUM for multi-DC. A low-cardinality partition key defeats all of it by piling a whole partition on one token.

Worked example — Redis Cluster slot math and hash tags

Detailed explanation. Compute which Redis Cluster slot a key lands in, show how hash tags co-locate related keys, and explain why resharding moves slots rather than re-hashing keys.

Slot. CRC16(key) mod 16384.
Hash tag. Only the {...} substring is hashed when present.
Reshard. Move slot ranges between nodes; clients follow MOVED.

Question. Determine the slots for user:1000:profile, user:1000:followers, and their hash-tagged variants, and explain co-location.

Input.

Key	Hash tag?
user:1000:profile	no
user:1000:followers	no
{user:1000}:profile	yes
{user:1000}:followers	yes

Code.

# Redis Cluster slot assignment: CRC16 of the (possibly tagged) key mod 16384
def crc16(data: bytes) -> int:
    crc = 0
    for b in data:
        crc ^= b << 8
        for _ in range(8):
            crc = ((crc << 1) ^ 0x1021) & 0xFFFF if (crc & 0x8000) else (crc << 1) & 0xFFFF
    return crc

def slot(key: str) -> int:
    # If a {tag} is present, hash only the substring inside the first {...}
    if "{" in key:
        a = key.index("{")
        b = key.find("}", a + 1)
        if b > a + 1:
            key = key[a + 1:b]
    return crc16(key.encode()) % 16384

for k in ("user:1000:profile", "user:1000:followers",
          "{user:1000}:profile", "{user:1000}:followers"):
    print(f"{k:>24} -> slot {slot(k)}")
# user:1000:profile   -> slot (some slot A)
# user:1000:followers -> slot (some slot B, likely != A)
# {user:1000}:profile   -> slot (slot T)
# {user:1000}:followers -> slot (slot T, SAME as above)

Step-by-step explanation.

Without a hash tag, CRC16 of the whole key mod 16384 gives the slot; user:1000:profile and user:1000:followers almost certainly land in different slots — and thus possibly different nodes — so a multi-key MGET across them can fail with CROSSSLOT.
Hash tags fix this: when a {...} is present, only the substring inside the braces is hashed. Both {user:1000}:profile and {user:1000}:followers hash the identical user:1000, so they share a slot and a node.
Co-locating them makes multi-key operations, transactions, and Lua scripts over a user's data legal and atomic — the deliberate escape hatch Redis gives you for related keys.
The slot count 16384 is fixed for the life of the cluster. Adding a node does not change any key's slot; instead, an operator (or the cluster) migrates whole slots to the new node and clients are redirected via MOVED/ASK.
This is why Redis Cluster gets consistent-hashing-like behavior — bounded movement on resize — without a live token ring: the modulus (16384) is decoupled from the node count, exactly the fixed-partition-map pattern.

Output.

Key	Hashed portion	Same slot as sibling?
user:1000:profile	whole key	no
user:1000:followers	whole key	no
{user:1000}:profile	user:1000	yes
{user:1000}:followers	user:1000	yes

Rule of thumb. In Redis Cluster, use {hash tags} to co-locate keys you need to operate on together, and remember resharding moves slots, not keys. Over-tagging (forcing too much into one slot) recreates a hotspot — tag only what genuinely must share a node.

Worked example — choosing a scheme for a new system

Detailed explanation. The senior skill is matching the partitioning scheme to the workload. Walk a short decision procedure across three scenarios: a stateless cache tier, a durable wide-column store, and an in-memory cluster needing multi-key ops.

Cache tier. Client-side, zero-coordination, cheap resize → ketama-style consistent hashing.
Durable store, multi-DC. Token ring + vnodes + zone-aware replication → Cassandra/Dynamo model.
In-memory, multi-key transactions. Fixed slots + hash tags → Redis Cluster model.

Question. For each scenario, pick a partitioning scheme and justify it in one line.

Input.

Scenario	Key need
Memcached-style cache	zero-coordination client routing
Durable KV / wide-column, multi-DC	replication + zone survival
In-memory store, multi-key ops	co-location + bounded resharding

Code.

Decision procedure — pick a partitioning scheme
===============================================

Q1. Is the data durable (must survive node loss)?
      no  -> stateless cache
            -> client-side consistent hashing (ketama): 150 vnodes/node,
               O(log N) lookup, no coordinator, cheap resize.
      yes -> go to Q2.

Q2. Do you need multi-key transactions / co-located keys on one node?
      yes -> fixed hash-slot map (Redis Cluster): CRC16 % 16384,
             hash tags to co-locate, migrate slots to resize.
      no  -> go to Q3.

Q3. Multi-datacenter with tunable consistency and high write availability?
      yes -> token ring + vnodes + NetworkTopologyStrategy
             (Cassandra / Dynamo): preference-list RF per DC, NWR quorums.
      no  -> single-DC token ring + vnodes + RF replication.

Step-by-step explanation.

Q1 splits on durability. A cache can lose data on node death (it just re-fills from origin), so it wants the lightest scheme — client-side consistent hashing with no coordinator, which ketama provides.
If data must survive, Q2 asks about multi-key co-location. Systems needing atomic multi-key ops benefit from an explicit slot map with hash tags (Redis Cluster), because arbitrary ring placement scatters related keys.
Q3 asks about multi-DC and consistency tuning. That is exactly what the Dynamo/Cassandra token ring plus NetworkTopologyStrategy and NWR quorums were built for.
The fall-through — durable, no multi-key needs, single DC — is a plain token ring with vnodes and RF replication, the section 2–4 construction without the multi-DC machinery.
The through-line: every branch is the same ring idea tuned for coordination cost, durability, co-location, and geography. Naming the branch and its one-line justification is the interview win.

Output.

Scenario	Scheme	One-line justification
Cache tier	ketama consistent hashing	zero-coordination client routing, cheap resize
Durable KV, multi-DC	Cassandra/Dynamo token ring	RF per DC + NWR quorums + zone survival
In-memory, multi-key	Redis Cluster fixed slots	hash tags co-locate; slot migration to resize

Rule of thumb. Match the scheme to coordination cost and durability: client-side rings for caches, token rings with zone-aware replication for durable multi-DC stores, and fixed hash slots for in-memory clusters that need multi-key operations. They are all the ring — the differences are about who agrees on placement and how resharding happens.

System design interview question on real-world partitioning

A senior interviewer might ask: "Compare how Dynamo, Cassandra, and Redis Cluster partition data. Which use a classic hash ring and which don't? Explain Redis Cluster's 16384 slots and why it chose fixed slots over a live ring. Then tell me how each handles adding a node, and how each replicates for durability."

Solution Using a decision matrix across Dynamo, Cassandra, and Redis Cluster

Partitioning across the three canonical systems
===============================================

DYNAMO (and DynamoDB-style)
  Placement : token ring + virtual nodes; key hashed onto ring, owner = successor
  Replication: preference list = next RF distinct nodes clockwise, across DCs
  Consistency: NWR quorum, W + R > N; vector clocks + Merkle repair (AP)
  Add node  : new vnodes steal small arcs; ~K/N keys move

CASSANDRA
  Placement : token ranges per node; num_tokens vnodes; Murmur3 partitioner
  Replication: NetworkTopologyStrategy, RF per DC in distinct racks
  Consistency: per-query ONE / LOCAL_QUORUM / QUORUM / ALL
  Add node  : bootstrap streams the new node's ranges; ~K/N data streamed

REDIS CLUSTER
  Placement : NOT a live ring - 16384 fixed slots, slot = CRC16(key) % 16384
  Replication: each master has replica(s); async replication + failover
  Consistency: async (can lose acked writes on failover); WAIT for stronger
  Add node  : migrate whole slots to the new master; clients follow MOVED/ASK

Step-by-step trace.

Dimension	Dynamo	Cassandra	Redis Cluster
Partition method	token ring + vnodes	token ranges + vnodes	16384 fixed slots
Classic ring?	yes	yes	no (fixed slot map)
Key → location	successor on ring	successor token range	CRC16 % 16384 → slot → node
Replication	preference list, RF	NetworkTopologyStrategy, RF/DC	master + async replicas
Add a node	steal arcs, ~K/N move	stream ranges, ~K/N move	migrate slots, ~K/N move
Co-locate keys	—	same partition key	hash tags `{...}`

Tracing across the matrix: Dynamo and Cassandra are true hash rings with vnodes and preference-list replication; Redis Cluster deliberately uses a fixed 16384-slot map so the modulus never changes and resize is a slot-migration. All three achieve the core property — only ~K/N of data moves when you add a node — but Dynamo/Cassandra do it by inserting tokens while Redis does it by reassigning slots.

Output:

System	Ring or slots	Add-node cost	Replication model	Consistency
Dynamo	ring + vnodes	~K/N keys	preference list (NWR)	tunable, AP
Cassandra	ring + vnodes	~K/N streamed	NetworkTopologyStrategy	per-query CL
Redis Cluster	16384 slots	~K/N slots migrated	master/replica async	async (WAIT opt.)

Why this works — concept by concept:

Dynamo preference list — the ring plus "next RF distinct nodes clockwise" gives durability and cheap rebalancing in one structure; NWR quorums layer tunable consistency on top, trading strict consistency for availability under partition.
Cassandra token ranges — the same ring, exposed as num_tokens vnodes and NetworkTopologyStrategy, so replicas span racks and DCs; bootstrapping a node streams only its K/N ranges rather than reshuffling the cluster.
Redis fixed slots — decoupling the modulus (16384) from node count makes resize a bounded slot-migration with MOVED/ASK redirection, achieving consistent-hashing's key property without live token math, at the cost of an explicit resharding step.
Hash tags and partition keys — co-location is a first-class concern: Redis uses {tag} substrings and Cassandra uses the partition key, both forcing related data onto one node so multi-key operations stay local.
Cost — all three cap membership-change movement at ~K/N; they differ in coordination (client-agreed ring vs gossiped slot map), consistency (AP quorum vs per-query CL vs async), and resize mechanics (token insert vs range stream vs slot migrate). Pick by the guarantees your workload needs, not by which name is trendiest.

Database
Topic — database
Sharding and partitioning in real databases

Practice →

Hash Table
Topic — hash-table
Slot-mapping and hashing problems

Practice →

Cheat sheet — consistent hashing recipes

Remap-fraction rule. Modulo hashing moves about (N-1)/N of keys on any change to N — ~80% for 4→5, ~99% for 100→101. Consistent hashing moves only ~k/(N+k) when you add k nodes to N — the proportional, survivable number. If you can whiteboard "80% move on 4→5" you have justified the ring in one sentence.
Never modulo the live server count. hash(key) % number_of_servers re-partitions the fleet on every scale event. Either use consistent hashing or a fixed partition map (e.g. % 16384) whose modulus never changes; scale by reassigning partitions, not by changing the modulus.
Minimal ring template. Keep a sorted array of token positions and a parallel array of owners; add_node does bisect.insort; get_node does i = bisect(pos, hash(key)) % len(pos) then returns owner[i]. The % len wrap is what makes it a ring — do not forget it. Lookup is O(log N).
Virtual-node sizing. Hash each node under V labels (node#0 … node#V-1) all mapping back to the physical node. Load imbalance falls like 1/sqrt(V); use 100–200 tokens/node to keep the hottest node within ~5% of fair. Doubling smoothness costs 4× the tokens.
Graceful failure test. Removing a node should fan its keys across all survivors, not dump them on one. If one neighbor doubles when a node dies, you have too few vnodes — raise V.
Weighting heterogeneous nodes. Token count is the capacity knob: give a 2× machine 2× the tokens and it owns ~2× the keyspace. Keep base V high so the realized split matches the intended weights (Cassandra exposes this as num_tokens).
Preference-list replication. For RF, walk clockwise from the owner collecting distinct physical nodes (skip tokens of already-chosen nodes), and extend the skip to distinct racks/zones so RF=3 survives one zone outage. Tune consistency with W + R > RF.
Bounded loads (CHBL). Cap each node at cap = ceil((1+ε)·avg) (ε ≈ 0.25); place a key at its owner if under cap, else spill clockwise to the next node with room. Guarantees no node exceeds (1+ε)·avg under skew, at the cost of extra probes and some lost locality.
Hot-key mitigation. Bounded loads fix many skewed keys; a single scorching key needs key-splitting (key#0 … key#S-1 across nodes) — reads become scatter-gather, writes cost S×. Split only the keys you must.
Redis Cluster slot rule. Slot = CRC16(key) mod 16384 (fixed slot count, not a live ring). Use {hash tags} to co-locate related keys ({user:1}:profile and {user:1}:followers share a slot); resharding migrates whole slots and redirects clients with MOVED/ASK.
Cassandra template. num_tokens: 16 + allocate_tokens_for_local_replication_factor for even ranges; Murmur3Partitioner; NetworkTopologyStrategy with RF per DC; a high-cardinality partition key; LOCAL_QUORUM for multi-DC reads/writes. A low-cardinality partition key defeats all balancing.
Which shape when. Client-side consistent hashing (ketama, 160 vnodes) for zero-coordination caches; token ring + vnodes + zone-aware replication (Cassandra/Dynamo) for durable multi-DC stores; fixed hash slots (Redis Cluster) for in-memory clusters that need multi-key operations. They are all the ring — differing in coordination, durability, and resharding mechanics.

Frequently asked questions

What is consistent hashing in one sentence?

Consistent hashing is a partitioning scheme that maps both keys and nodes onto the same circular hash space and assigns each key to the first node clockwise, so that adding or removing a node relocates only about K/N of the keys — the slice belonging to the changed node — instead of the near-total reshuffle that hash(key) % N inflicts on every resize. It is the load-bearing algorithm behind distributed caches (Memcached client libraries), key-value stores (Dynamo, Cassandra, Riak), and sharded systems that need to scale membership without a full data migration. Virtual nodes, preference-list replication, and bounded loads are refinements layered on top of this one idea.

Why does modulo hashing move so many keys on resize?

Because the owner of a key, hash(key) % N, depends on N itself, so changing N changes the assignment function for essentially every key. Going from 4 nodes to 5, a key keeps its owner only when hash % 4 == hash % 5, which holds for just 4 of every 20 residues — so ~80% of keys move. The fraction gets worse at scale (100→101 moves ~99%), because x % 100 and x % 101 share almost no structure. For a cache that means ~80% guaranteed misses the instant you deploy (a thundering herd on the origin); for a stateful store it means physically copying ~80% of your dataset. Consistent hashing avoids this by making ownership depend on ring positions, so adding a node only disturbs one arc.

What are virtual nodes and why do I need them?

Virtual nodes give each physical machine many tokens on the ring (typically 100–200) instead of one. With a single token per node, the arcs between tokens are random and uneven, so one node can own nearly twice its fair share and, worse, removing a node dumps its entire arc onto a single successor. With many tokens, each node's load is the average of many independent arcs, so the imbalance shrinks like 1/sqrt(V) and a departing node's keys fan out evenly across all survivors. Virtual nodes also make heterogeneous clusters trivial: give a bigger machine proportionally more tokens and it owns proportionally more keyspace. Real systems expose this as num_tokens (Cassandra) or a per-node weight (ketama).

How does replication work on a hash ring?

Each key is stored on its owner plus the next RF-1 distinct physical nodes walking clockwise — Dynamo calls this ordered set the "preference list." The walk must skip tokens whose physical node is already chosen (so RF=3 means three different machines, not three tokens), and production systems extend the skip to distinct racks or availability zones so a whole zone can fail without losing all copies. Consistency is tuned with quorums: requiring W acknowledged writes and R reads such that W + R > RF guarantees a read intersects the latest write. Cassandra implements this as NetworkTopologyStrategy plus per-query consistency levels like LOCAL_QUORUM.

Does Redis Cluster use consistent hashing?

Not in the classic ring sense — and saying so precisely is a senior signal. Redis Cluster maps every key to one of 16384 fixed hash slots via CRC16(key) mod 16384, then assigns slots (not keys) to master nodes. The slot count never changes, so the "modulus" is stable even as nodes join and leave; you resize by migrating whole slots between nodes and redirecting clients with MOVED/ASK. This is a fixed partition map, and it achieves consistent hashing's key property (only ~K/N of data moves on resize) by decoupling the modulus from the node count rather than by walking a live token ring. Related keys can be co-located with {hash tags}, which hash only the braced substring so {user:1}:profile and {user:1}:followers share a slot.

When should I NOT use consistent hashing?

When your node count is genuinely fixed forever, a plain fixed partition map (or even % N) is simpler and gives O(1) lookup — consistent hashing's machinery earns its keep only when membership changes. When you need multi-key transactions across arbitrary keys, a pure ring scatters related keys, so you either add hash tags / co-location (as Redis and Cassandra do) or choose range partitioning instead. When your bottleneck is a single scorching-hot key, consistent hashing (even with bounded loads) won't help, because one key can't be spread without splitting it; you need caching or key-splitting. And when you need ordered range scans (WHERE id BETWEEN a AND b), hash partitioning destroys locality — range partitioning is the right tool there.

Practice on PipeCode

Drill the hash table practice library → for the hashing, bucket-distribution, and slot-mapping problems that underpin consistent hashing.
Sharpen the fundamentals on the data structures practice library → for the sorted-array, binary-search, and ring-design patterns behind a production hash ring.
Connect it to storage on the database practice library → for the sharding, partitioning, and replication scenarios interviewers build on top of consistent hashing.
Stack these against PipeCode's broader 450+ data-engineering catalogue to anchor the partitioning decision matrix — modulo vs ring vs fixed slots — against real graded inputs.

Lock in consistent-hashing muscle memory

Docs explain the ring. PipeCode drills explain the decision — when modulo hashing's 80% reshuffle becomes an outage, how many virtual nodes it takes to balance a tier, when bounded loads beat key-splitting, and why Redis Cluster chose fixed slots over a live ring. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the partitioning and sharding trade-offs real distributed systems face.

Practice hash-table problems →
Practice database problems →

HyperLogLog: Count Billions of Uniques in Kilobytes

Gowtham Potureddi — Mon, 07 Sep 2026 15:55:11 +0000

hyperloglog is the probabilistic data structure that answers "how many distinct things did we see?" over billions of events using a few kilobytes of memory and roughly one percent of error — and it is the single algorithm that separates engineers who can count uniques at scale from engineers who keep crashing a job because they tried to hold every distinct value in a set. The question sounds trivial: how many unique visitors hit the site today, how many distinct IPs probed the firewall, how many distinct search terms appeared this hour. The exact answer requires remembering every distinct value you have already seen so you do not double-count it, and "remember every distinct value" is precisely the thing that does not fit in memory once the value count climbs into the hundreds of millions. cardinality estimation is the branch of algorithms that trades a small, controllable amount of accuracy for an enormous reduction in memory, and HyperLogLog is its most widely deployed member.

This guide is the walkthrough you wished existed the first time an interviewer asked "how would you count distinct users across a fleet of servers without a giant hash set?" or "explain how HyperLogLog gets billions of uniques into twelve kilobytes" or "why can you merge two sketches but not two COUNT(DISTINCT) results?" It builds the idea in layers: why exact approximate distinct count's honest cousin — the real distinct count — is O(n) memory and cannot be cheated with sampling; how HyperLogLog hashes each item, watches for improbably long runs of leading zeros, and stores only a tiny array of small integers; how the raw harmonic-mean estimate is corrected for bias so it stays accurate across the whole cardinality range; why sketches merge losslessly and what that unlocks for distributed and streaming aggregation; and finally how the exact same idea shows up as a one-line function in Redis, BigQuery, and Spark. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the cardinality practice library →, sharpen the fundamentals on the data-structures practice library →, and build estimation intuition on the statistics practice library →.

On this page

The distinct-count problem
How HLL works — registers, leading zeros, harmonic mean
Bias correction & accuracy
Mergeability & sketches at scale
HLL in practice — Redis / BigQuery / Spark
Cheat sheet — HyperLogLog recipes
Frequently asked questions
Practice on PipeCode

1. The distinct-count problem

Counting uniques exactly costs memory proportional to the number of uniques — and that is the wall HyperLogLog exists to break

The one-sentence invariant: an exact distinct count must remember every distinct value it has already seen so it can recognise a repeat, which makes its memory grow linearly with the number of distinct values (cardinality), whereas a probabilistic sketch like HyperLogLog spends a fixed, tiny amount of memory to estimate that same cardinality within a known error bound — you are trading a small, bounded inaccuracy for the difference between gigabytes and kilobytes. The moment your cardinality is large enough that "keep a set of everything" does not fit, exact counting stops being an option and the only real question becomes which approximate technique you use and what error you will accept.

The axes that matter for any distinct-count method.

Memory. Exact counting via a hash set costs roughly (bytes-per-value + hash-table-overhead) × cardinality. For a billion distinct 16-byte identifiers that is tens of gigabytes. HyperLogLog costs a fixed 12 KB regardless of whether the cardinality is a hundred or a hundred billion. This is the headline difference and the reason the algorithm exists.
Accuracy. Exact counting is perfect. HyperLogLog has a relative standard error of about 1.04 / sqrt(m) where m is the number of registers — around 0.8% at the common 12 KB configuration. "Relative" is the key word: the error is a percentage of the true count, not a fixed number of items.
Mergeability. Can two partial counts be combined into a whole-dataset count without re-reading the raw data? Exact COUNT(DISTINCT) results cannot be merged (5 uniques here plus 5 uniques there is anywhere from 5 to 10 total). HyperLogLog sketches can be merged losslessly. This single property is why HLL dominates distributed and streaming systems.
Update cost. How expensive is it to add one item? Exact counting is one hash-set insert. HyperLogLog is one hash, a couple of bit operations, and one array write — O(1), branch-light, and cache-friendly.

The three cost curves — exact, sampling, and sketching.

Exact set. Store every distinct value; memory is O(cardinality). Accurate but unbounded. This is what SELECT COUNT(DISTINCT col) does under the hood, and it is what falls over at scale.
Sampling. Look at 1% of the data and multiply the distinct count by 100. This works for sums and averages but is disastrous for cardinality — a value that appears once in the full data has a 99% chance of being missed by the sample entirely, so rare-but-distinct values (which dominate a distinct count) are systematically undercounted. Sampling is the wrong tool for uniques, and saying so is an instant senior signal.
Sketching. Keep a small, fixed-size summary (a "sketch") that is updated per item and from which the cardinality is estimated. HyperLogLog is a sketch. So are its ancestors (Flajolet–Martin, LogLog) and cousins (K-Minimum-Values, Theta sketches). The sketch never stores the values themselves — only a statistical fingerprint of how many distinct ones passed through.

The 2026 reality — HLL is the default and it is everywhere.

Every analytics warehouse ships it. BigQuery, Redshift, Snowflake, Presto/Trino, ClickHouse, and DuckDB all expose an APPROX_COUNT_DISTINCT-style function backed by HyperLogLog or a close variant. It is the default for dashboards, funnels, and "unique X per Y" panels.
Every large key-value store ships it. Redis has had PFADD/PFCOUNT/PFMERGE since 2014; it is the canonical way to track daily/weekly unique visitors per page or per campaign.
Every stream processor leans on it. Because HLL sketches merge, they are the natural per-window, per-key accumulator in Spark, Flink, and Kafka Streams for "distinct count over a sliding window."
The rare wrong place. HLL is not for billing, compliance counts, or anything where an off-by-1% is unacceptable. For those you pay for exact counting or a different structure. Knowing where not to use HLL is as much a senior signal as knowing where to use it.

What interviewers listen for.

Do you say "exact distinct count is O(cardinality) memory" as the reason HLL exists? — required answer.
Do you reject sampling for cardinality with the "rare values get missed" argument? — senior signal.
Do you name the trade as "fixed memory and bounded relative error" rather than "it's approximate"? — senior signal.
Do you volunteer mergeability as HLL's superpower for distributed counting? — senior signal.
Do you name a case where you would not use HLL (billing, exact compliance)? — senior signal.

Worked example — the memory blow-up of an exact distinct count

Detailed explanation. The fastest way to make the distinct-count problem concrete is to compute the memory an exact hash set consumes as cardinality grows, and put it next to HyperLogLog's flat 12 KB line. Every senior conversation about "why not just COUNT(DISTINCT)?" ends here.

The exact structure. A hash set holding n distinct 16-byte UUIDs. In a real runtime each entry also carries pointer and load-factor overhead; a conservative estimate is ~48–64 bytes per entry once you include the value, the hash bucket, and slack.
The HLL structure. A fixed register array. At precision p = 14 there are m = 2^14 = 16384 registers of 6 bits each → 12 KB, independent of n.
The comparison. Tabulate memory at 1e3, 1e6, 1e9 distinct values for both.

Question. For 1 thousand, 1 million, and 1 billion distinct UUIDs, how much memory does an exact hash set use versus a p = 14 HyperLogLog?

Input.

Cardinality n	Bytes/entry (exact)	Exact set memory	HLL memory (p=14)
1,000	~56	~56 KB	12 KB
1,000,000	~56	~56 MB	12 KB
1,000,000,000	~56	~56 GB	12 KB

Code.

# Compare exact hash-set memory to a fixed HLL footprint
BYTES_PER_ENTRY = 56          # value + bucket + load-factor slack (conservative)
HLL_BYTES       = 16384 * 6 // 8   # m=2^14 registers, 6 bits each = 12,288 bytes

def exact_bytes(n: int) -> int:
    return n * BYTES_PER_ENTRY

for n in (1_000, 1_000_000, 1_000_000_000):
    exact = exact_bytes(n)
    ratio = exact / HLL_BYTES
    print(f"n={n:>15,}  exact={exact/1e6:>12.1f} MB  hll={HLL_BYTES/1024:.0f} KB  "
          f"exact/hll={ratio:>12,.0f}x")

Step-by-step explanation.

The exact set stores the value itself plus per-entry overhead; even a lean implementation cannot go below "one slot per distinct value," so its memory is strictly Θ(n). Doubling the distinct count doubles the memory forever.
The HLL footprint is computed once from the precision: m = 2^p registers, each 6 bits wide (6 bits holds a leading-zero count up to 63, which is enough for a 64-bit hash). At p = 14 that is a constant 12,288 bytes.
The exact/hll ratio is the punchline: at a billion distinct values, the exact set is roughly 56 GB and the HLL is 12 KB — a factor of about 4.6 million. This is not a constant-factor optimisation; it is a different asymptotic class.
The exact set's memory also determines whether the job runs at all. 56 GB does not fit in a worker's RAM, so an exact billion-cardinality count either spills to disk (slow) or requires a distributed shuffle of every distinct value. HLL never leaves 12 KB.
The only thing you give up for that 4.6-million-fold saving is exactness: the HLL answer is within roughly ±0.8% of the truth. For "how many unique visitors today," ±0.8% is invisible; for "how many licences to bill," it is unacceptable.

Output.

Cardinality n	Exact set	HLL (p=14)	exact/hll ratio
1,000	~0.06 MB	12 KB	~5x
1,000,000	~56 MB	12 KB	~4,672x
1,000,000,000	~56,000 MB	12 KB	~4,672,000x

Rule of thumb. Exact distinct count is Θ(cardinality) memory; HyperLogLog is Θ(1). If your cardinality can exceed roughly a few million and you can tolerate ~1% error, the memory argument alone decides it — reach for a sketch.

Worked example — why a plain bitmap is not enough

Detailed explanation. A common first instinct is "use a bitmap": one bit per possible value, set the bit when you see the value, then popcount the bits. This works and is even exact when the value space is small and dense (e.g. counting distinct integer user-ids in [0, 100M)), but it degrades badly as the value space grows, which is why HLL supersedes it for general keys.

The bitmap idea. Allocate a bit array indexed by the value (or by hash(value) mod N). Set bit on insert. Distinct count = number of set bits.
When it wins. Small, dense, integer value spaces — a bitmap over 100M possible ids is 12.5 MB and is exact. This is the sweet spot for Roaring bitmaps.
When it loses. Arbitrary keys (UUIDs, URLs, emails). Either you size the bitmap to the value space (astronomically large) or you hash mod N into a smaller array and suffer collisions that undercount — and to keep collisions low you need N comparable to the cardinality, which puts you back at Θ(n) memory.

Question. Compare a bitmap and an HLL for counting distinct UUIDs where the cardinality is ~100 million.

Input.

Method	Sizing rule	Memory at 100M cardinality	Exact?
Dense bitmap over UUID space	2^128 bits	infeasible	yes
Hashed bitmap (mod N)	N ≈ 10 × cardinality to keep collisions low	~125 MB	no (collisions undercount)
HyperLogLog (p=14)	fixed	12 KB	no (~0.8% error)

Code.

# A hashed bitmap must grow with cardinality to keep collisions rare;
# HLL does not. Show the collision-driven undercount of an undersized bitmap.
import random

def hashed_bitmap_estimate(items, n_bits):
    bits = bytearray((n_bits + 7) // 8)
    for x in items:
        h = (hash(x) & 0x7fffffffffffffff) % n_bits
        bits[h >> 3] |= (1 << (h & 7))
    return sum(bin(b).count("1") for b in bits)   # popcount

true_card = 1_000_000
items = [f"user-{i}" for i in range(true_card)]

for n_bits in (true_card // 10, true_card, true_card * 10):
    est = hashed_bitmap_estimate(items, n_bits)
    print(f"bits={n_bits:>10,}  set_bits={est:>10,}  "
          f"undercount={(1 - est/true_card)*100:6.2f}%")

Step-by-step explanation.

The dense bitmap over the full UUID space is a non-starter: 2^128 bits is more storage than exists on Earth. So any bitmap over arbitrary keys must hash into a bounded array first.
Once you hash mod N, two distinct values can land on the same bit — a collision — and the second one sets a bit that is already set, so it is silently not counted. The set-bit count is therefore an undercount of the true cardinality.
To keep collisions rare you must make N large relative to the cardinality. As the loop shows, when n_bits is one-tenth of the cardinality the undercount is severe; only when n_bits is several times the cardinality does the set-bit count approach the truth — and at that point the bitmap is Θ(n) bits, i.e. back to linear memory.
HyperLogLog sidesteps this entirely: it also hashes, but it does not need one slot per distinct value. It extracts statistical rarity from each hash (how many leading zeros) rather than trying to reserve a bit for each value, so its memory stays fixed while accuracy is controlled by the register count, not by the cardinality.
The lesson: bitmaps are excellent for small, dense, integer domains (use Roaring bitmaps there) and wrong for large arbitrary-key domains. HLL is the general-purpose answer when keys are arbitrary and cardinality is large.

Output.

Bitmap bits	Set bits (estimate)	Undercount
100,000	~ 95,000	severe
1,000,000	~ 632,000	~37%
10,000,000	~ 951,000	~5%

Rule of thumb. Bitmaps are exact and cheap only for small dense integer keys; for arbitrary keys they force you back to linear memory to control collisions. HyperLogLog keeps memory fixed by measuring hash rarity instead of reserving a slot per value.

Worked example — why sampling cannot estimate cardinality

Detailed explanation. Sampling is the reflex fix for expensive aggregates, and it is correct for sums and averages — but it is fundamentally broken for distinct counts. The reason is that cardinality is dominated by rare values, and rare values are exactly what a sample misses. Making this argument crisply is a reliable way to demonstrate you understand why a sketch is needed, not just that one exists.

The sampling plan. Take a q-fraction sample, count distinct in the sample, divide by q.
The failure. A value that occurs once in the full dataset is included in a q-sample with probability q. If q = 0.01, singletons are seen 1% of the time — so 99% of them vanish, and the naive distinct_in_sample / q scale-up is wildly biased.
The intuition. For heavy-hitter frequencies sampling is fine (a value that occurs a million times will surely appear in a 1% sample). For cardinality it is not, because cardinality counts each value once regardless of frequency, so the long tail of singletons dominates and the tail is what sampling drops.

Question. A dataset has 1,000,000 distinct values, 900,000 of which occur exactly once. Estimate the distinct count from a 1% sample by scaling, and compare to the truth.

Input.

Quantity	Value
True distinct count	1,000,000
Singletons (occur once)	900,000
Frequent values (occur ≥ 100×)	100,000
Sample fraction q	0.01

Code.

import random

random.seed(7)
# Build a dataset: 900k singletons + 100k frequent values (100 occurrences each)
data = []
for i in range(900_000):
    data.append(f"rare-{i}")                    # each appears once
for i in range(100_000):
    data.extend([f"freq-{i}"] * 100)            # each appears 100 times
random.shuffle(data)

q = 0.01
sample = [x for x in data if random.random() < q]

distinct_true   = 1_000_000
distinct_sample = len(set(sample))
scaled_estimate = distinct_sample / q           # naive scale-up

print(f"true distinct       = {distinct_true:,}")
print(f"distinct in sample  = {distinct_sample:,}")
print(f"scaled estimate     = {scaled_estimate:,.0f}")
print(f"relative error      = {(scaled_estimate/distinct_true - 1)*100:.1f}%")

Step-by-step explanation.

The dataset is deliberately tail-heavy: 90% of the distinct values are singletons, which is realistic for URLs, search terms, and session ids. The frequent values are few in count but many in occurrences.
A 1% sample includes each singleton with probability ~1%, so only ~9,000 of the 900,000 singletons survive. Almost the entire distinct tail is gone from the sample.
The frequent values, by contrast, almost all survive (a value with 100 occurrences appears in a 1% sample with probability 1 - 0.99^100 ≈ 63%), so the sample's distinct count is dominated by frequent values, not the tail.
Scaling distinct_in_sample / q does not fix this, because the scale factor assumes each distinct value had a uniform chance of inclusion — but singletons and frequent values have wildly different inclusion probabilities. The estimate is biased low and unstable.
HyperLogLog processes every item (it is a streaming, full-pass structure, not a sample), so singletons are never dropped — each one still gets hashed and can bump a register. That is why a sketch that reads all the data in fixed memory beats a sample that reads little of it.

Output.

Metric	Value
True distinct	1,000,000
Distinct in 1% sample	~118,000
Scaled estimate (÷0.01)	~11,800,000
Relative error	wildly off (10×+)

Rule of thumb. Never estimate cardinality by sampling and scaling — the singleton-dominated tail makes it biased and unstable. Use a full-pass sketch (HyperLogLog) that touches every item in fixed memory.

Data engineering interview question on the distinct-count problem

A senior interviewer often opens with: "You need a real-time dashboard showing unique visitors per article for a news site with 50 million articles and billions of daily pageviews. A naive design keeps a Set of visitor ids per article in memory and crashes nightly. Walk me through why the naive design fails, what you'd replace it with, and the accuracy and memory you'd promise the product team."

Solution Using per-key HyperLogLog sketches instead of exact sets

# Naive design (fails) vs sketch design (survives)
# ---- Naive: one Python set per article ----
#   unique_by_article[article_id].add(visitor_id)
#   memory  = sum over articles of (distinct_visitors * ~56 bytes)
#   at billions of (article, visitor) pairs this is tens of GB and OOMs.

# ---- Sketch: one fixed 12 KB HLL per article ----
from math import log

M = 16384                      # m = 2^14 registers  (precision p = 14)
STD_ERROR = 1.04 / (M ** 0.5)  # relative standard error ~ 0.81%

def per_article_memory(n_articles: int) -> float:
    hll_bytes = M * 6 / 8      # 12,288 bytes per article
    return n_articles * hll_bytes

def naive_memory(n_articles: int, avg_uniques: int) -> float:
    return n_articles * avg_uniques * 56

n_articles  = 50_000_000
avg_uniques = 2_000           # average distinct visitors per article per day

print(f"std error         = {STD_ERROR*100:.2f}%")
print(f"naive memory      = {naive_memory(n_articles, avg_uniques)/1e9:,.1f} GB")
print(f"sketch memory     = {per_article_memory(n_articles)/1e9:,.1f} GB (worst case, all dense)")

# Redis-backed version — one HLL key per article per day
PFADD  uv:article:12345:2026-09-05  visitor_abc
PFADD  uv:article:12345:2026-09-05  visitor_def
PFCOUNT uv:article:12345:2026-09-05        -> approximate unique visitors
PFMERGE uv:article:12345:week  uv:article:12345:2026-09-05 ... (7 days)
PFCOUNT uv:article:12345:week               -> approximate weekly uniques

Step-by-step trace.

Step	Naive (set per article)	Sketch (HLL per article)
Per-article memory	grows with distinct visitors (unbounded)	fixed 12 KB (often far less, sparse)
Add a visitor	`set.add` (grows the set)	hash + one register write, O(1)
Read the count	`len(set)`, exact	`PFCOUNT`, ~0.8% error
Weekly rollup	re-scan/union all raw ids	`PFMERGE` 7 daily sketches
Failure mode	OOM at scale	none; memory is bounded

After the switch, each (article, day) gets one bounded sketch. Adds are O(1), the daily count is a single PFCOUNT, and the weekly/monthly rollups are a PFMERGE of the daily sketches — no raw visitor ids are ever stored, so the memory ceiling is the number of live sketches times their fixed size, not the number of distinct visitors.

Output:

Metric	Naive set design	HLL sketch design
Memory	tens of GB, unbounded	bounded (≤ ~0.6 TB worst case, usually far less)
Accuracy	exact	~0.8% relative error
Add cost	O(1) amortised, grows RAM	O(1), fixed RAM
Rollups	expensive re-union of ids	cheap sketch merge
Nightly outcome	OOM crash	stable

Why this works — concept by concept:

Fixed-size sketch per key — replacing an unbounded Set with a fixed 12 KB HyperLogLog decouples memory from cardinality. The worst case is n_keys × sketch_size, a number you can capacity-plan; the set design's worst case is n_keys × distinct_per_key, a number you cannot.
O(1) streaming updates — an HLL add is a hash plus a bounded-register max, so ingesting billions of pageviews is CPU-bounded, not memory-bounded, and never triggers a rehash-and-grow of an ever-larger set.
Sparse representation — for the many low-traffic articles, the sketch stays in a compact sparse form far below 12 KB, so the practical memory is well under the dense worst case.
Merge for rollups — daily → weekly → monthly counts are PFMERGEs of existing sketches, so you never re-read raw events to change the time granularity.
Cost — O(1) memory per key and O(1) time per event, versus O(distinct-per-key) memory and O(distinct) rollup cost for the exact design. You pay a bounded ~0.8% error, which is invisible on a unique-visitors dashboard.

Cardinality
Topic — cardinality
Cardinality-estimation and distinct-count problems

Practice →

Data structures Topic — data-structures Probabilistic data-structure problems

Practice →

2. How HLL works — registers, leading zeros, harmonic mean

Hash to uniform bits, watch for improbably long runs of leading zeros, keep the max per bucket, then average — that is the whole algorithm

The mental model in one line: HyperLogLog hashes each item to a uniformly random bit string, uses the first p bits to pick one of m = 2^p registers, counts the number of leading zeros (plus one) in the remaining bits, stores the maximum such count ever seen in that register, and finally estimates the cardinality as the harmonic mean of 2^register across all registers times a bias constant — because a long run of leading zeros is improbable, the longest run you have seen is a fingerprint of how many distinct items passed through. No item values are stored; only m small integers.

The core probabilistic intuition.

A uniform hash makes each bit a fair coin. A good hash spreads any input over the output space uniformly, so each bit of the hash is independently 0 or 1 with probability one-half. The value itself is irrelevant after hashing; only the bit pattern matters.
Long runs of leading zeros are rare. The probability that a hash begins with exactly k zeros then a one is 2^-(k+1). Seeing a run of 10 leading zeros is a one-in-1024 event — so if you have seen it, you have probably observed on the order of 1024 distinct hashes.
The maximum run is the fingerprint. Track the longest leading-zero run observed. If the maximum is R, a reasonable guess for the number of distinct items is about 2^R. This is the original Flajolet–Martin insight; everything else is variance reduction.
One estimator is noisy; many are stable. A single maximum-run counter has enormous variance. HyperLogLog runs m independent estimators (the registers), each seeing a random 1/m slice of the items, and combines them — that averaging is what turns a wild guess into a ~1%-error estimate.

Registers — the fixed-size state.

What a register holds. A single small integer: the maximum leading-zero-count (rho) seen for hashes routed to that register. Six bits is enough (a 64-bit hash yields rho up to ~50, and 6 bits holds 0–63).
How many registers. m = 2^p, where p is the precision. p = 14 gives m = 16384 registers and the canonical 12 KB / ~0.8% configuration. More registers → less error and more memory.
Register routing. The first p bits of the hash select the register index; the remaining bits are scanned for leading zeros. Splitting one hash into "which bucket" plus "how rare" is what lets m estimators run from a single hash function.

The estimator — harmonic mean, not arithmetic mean.

Why harmonic. Each register's implied estimate is 2^register, an exponential quantity where a single large value can dominate an arithmetic mean and inflate the estimate. The harmonic mean tames those outliers, which is the specific improvement HyperLogLog made over its predecessor LogLog.
The formula. E = alpha_m × m^2 × ( Σ_j 2^(-M[j]) )^(-1), where M[j] is register j, the sum runs over all m registers, and alpha_m is a bias-correction constant (about 0.7213 for large m). The Σ 2^(-M[j]) term is the reciprocal-sum that makes this a harmonic mean.
The constant. alpha_m corrects a systematic bias in the raw harmonic-mean formula. It depends on m: 0.673 for m=16, 0.697 for m=32, 0.709 for m=64, and 0.7213 / (1 + 1.079/m) for m ≥ 128.

Common beginner mistakes.

Using a weak or non-uniform hash — if the bits are not fair coins, the leading-zero statistics are wrong and the estimate is biased.
Averaging 2^register with the arithmetic mean instead of the harmonic mean — reintroduces the outlier sensitivity HLL was designed to remove.
Forgetting the alpha_m constant — the raw estimator is biased without it.
Under-sizing the register width — 4 or 5 bits truncates rho for 64-bit hashes at high cardinality.

Worked example — hash one item into bucket and rho

Detailed explanation. The atomic operation of HyperLogLog is turning one item into a (register_index, rho) pair. Everything else is repeating this and taking a max. Walk through it for a single item at precision p = 4 (small, so the bits are readable).

Precision. p = 4 → m = 16 registers, register index is the first 4 bits.
The split. First 4 bits = register index; remaining bits = the run we scan for leading zeros.
rho. rho = (number of leading zeros in the remaining bits) + 1 — the position of the leftmost 1-bit.

Question. Given a 32-bit hash, compute the register index and rho at p = 4.

Input.

Field	Value
Precision p	4
Registers m	16
Example hash (32-bit)	`0110 1 0001 ...` (index bits, then remainder)
Index bits	first 4
Remainder	bits after the index

Code.

def bucket_and_rho(h: int, p: int, hash_bits: int = 32):
    """Return (register_index, rho) for a hash value h."""
    m = 1 << p
    # First p bits = register index (top bits of the hash)
    idx = h >> (hash_bits - p)
    # Remaining bits = the tail we scan for the leftmost 1-bit
    remainder = h & ((1 << (hash_bits - p)) - 1)
    w_bits = hash_bits - p
    # rho = position of the leftmost 1-bit in the remainder (1-based)
    if remainder == 0:
        rho = w_bits + 1                      # no 1-bit at all in w_bits
    else:
        rho = w_bits - remainder.bit_length() + 1
    return idx, rho

# Example: hash whose top 4 bits are 0110 (=6), then a 1 immediately after
h = 0b0110_1_0001010101010101010101010  # 32 bits
idx, rho = bucket_and_rho(h, p=4)
print(f"register index = {idx}")   # 6
print(f"rho            = {rho}")   # 1  (leftmost remainder bit is a 1)

Step-by-step explanation.

The hash is treated as a fixed-width bit string. Shifting right by hash_bits - p keeps only the top p bits — that integer, 0110 = 6, is the register index. This routes the item to register 6 of 16.
Masking with (1 << (hash_bits - p)) - 1 keeps the low w_bits = 28 bits — the remainder we inspect for leading zeros.
rho is the 1-based position of the leftmost 1-bit in the remainder. Here the first remainder bit is a 1, so there are zero leading zeros and rho = 1.
If the remainder had been 0001..., there would be three leading zeros and rho = 4. Bigger rho means a rarer bit pattern, which is evidence of higher cardinality feeding this register.
The remainder == 0 guard handles the vanishingly rare case where the whole tail is zeros; rho is then w_bits + 1. Using a 64-bit hash makes this and hash-collision effects negligible.

Output.

Item hash top bits	Register index	Leading zeros in remainder	rho
`0110 1...`	6	0	1
`0110 0001...`	6	3	4
`1010 001...`	10	2	3

Rule of thumb. One hash → split into p index bits and a remainder → rho is the position of the leftmost 1-bit in the remainder. Register idx keeps max(current, rho). That single step, repeated, is the entire ingest path.

Worked example — build a tiny HLL and update registers

Detailed explanation. Now repeat the atomic step over a stream and keep the per-register maximum. This is the whole "add" side of HyperLogLog; the registers array is the sketch. Build a minimal, correct HLL in a few lines.

State. An array of m registers initialised to 0.
Add. Hash the item, compute (idx, rho), set M[idx] = max(M[idx], rho).
Only the max matters. Re-adding the same item is idempotent (same hash → same idx, rho ≤ stored), which is exactly why the structure counts distinct items.

Question. Implement add for a minimal HyperLogLog and show the register array after ingesting a few items.

Input.

Field	Value
Precision p	4 (m = 16)
Hash	64-bit (blake2b truncated)
Items	"alice", "bob", "carol", "alice" (dup)

Code.

import hashlib

class TinyHLL:
    def __init__(self, p: int = 4):
        self.p = p
        self.m = 1 << p
        self.regs = [0] * self.m
        self.hash_bits = 64

    def _hash(self, item: str) -> int:
        d = hashlib.blake2b(item.encode(), digest_size=8).digest()
        return int.from_bytes(d, "big")          # 64-bit uniform-ish hash

    def add(self, item: str) -> None:
        h = self._hash(item)
        idx = h >> (self.hash_bits - self.p)     # top p bits
        w = h & ((1 << (self.hash_bits - self.p)) - 1)
        w_bits = self.hash_bits - self.p
        rho = (w_bits + 1) if w == 0 else (w_bits - w.bit_length() + 1)
        if rho > self.regs[idx]:                  # keep the maximum
            self.regs[idx] = rho

hll = TinyHLL(p=4)
for item in ["alice", "bob", "carol", "alice"]:  # "alice" twice
    hll.add(item)
print(hll.regs)          # dup "alice" does not change anything the 2nd time
print("non-zero regs:", sum(1 for r in hll.regs if r > 0))

Step-by-step explanation.

The register array starts all-zero — a fresh sketch that estimates cardinality 0. Each entry will hold the largest rho routed to it.
add hashes the item once, derives (idx, rho), and writes M[idx] = max(M[idx], rho). The max is the crux: only an item that produces a rarer pattern than anything seen before for that bucket changes the state.
Adding "alice" the second time hashes to the identical idx and rho, so the rho > regs[idx] test fails and nothing changes. Duplicates are free and invisible — that is why HLL counts distinct elements, not occurrences.
Different items scatter across registers because their hash prefixes differ; with only 16 registers and 3 distinct items you will typically see a few non-zero registers, each holding a small rho.
The sketch's entire memory is self.regs — 16 integers here, m in general. No item strings are retained, so the structure is privacy-friendly and size-bounded.

Output.

Action	Effect on registers
add("alice")	one register set to its rho
add("bob")	another register set
add("carol")	another register set
add("alice") again	no change (idempotent)
Non-zero registers	~3 (one per distinct item, modulo collisions)

Rule of thumb. The add path is: hash → (idx, rho) → M[idx] = max(M[idx], rho). Because it only ever takes a max, adding a duplicate is a no-op — the reason a max-of-rho array counts distinct items rather than total items.

Worked example — the raw harmonic-mean estimate

Detailed explanation. With registers populated, the estimate is a closed-form formula over the array. Implement the raw HyperLogLog estimator and run it on a synthetic stream to see it land near the true cardinality.

The reciprocal sum. Z = Σ_j 2^(-M[j]) — small when registers are large (high cardinality).
The formula. E = alpha_m × m^2 / Z.
The constant. alpha_m = 0.7213 / (1 + 1.079/m) for m ≥ 128.

Question. Estimate the cardinality of a stream of 100,000 distinct items with a p = 14 HLL using the raw estimator.

Input.

Field	Value
Precision p	14 (m = 16384)
True cardinality	100,000
Estimator	raw harmonic mean
alpha_m	0.7213 / (1 + 1.079/m)

Code.

import hashlib

def alpha_m(m: int) -> float:
    if m == 16:  return 0.673
    if m == 32:  return 0.697
    if m == 64:  return 0.709
    return 0.7213 / (1 + 1.079 / m)

class HLL:
    def __init__(self, p=14):
        self.p, self.m = p, 1 << p
        self.regs = [0] * self.m
        self.bits = 64
    def add(self, x: str):
        h = int.from_bytes(hashlib.blake2b(x.encode(), digest_size=8).digest(), "big")
        idx = h >> (self.bits - self.p)
        w = h & ((1 << (self.bits - self.p)) - 1)
        wb = self.bits - self.p
        rho = (wb + 1) if w == 0 else (wb - w.bit_length() + 1)
        if rho > self.regs[idx]:
            self.regs[idx] = rho
    def raw_estimate(self) -> float:
        Z = sum(2.0 ** (-r) for r in self.regs)
        return alpha_m(self.m) * self.m * self.m / Z

hll = HLL(p=14)
true_n = 100_000
for i in range(true_n):
    hll.add(f"item-{i}")

est = hll.raw_estimate()
print(f"true      = {true_n:,}")
print(f"estimate  = {est:,.0f}")
print(f"error     = {(est/true_n - 1)*100:+.2f}%")

Step-by-step explanation.

Every one of the 100,000 distinct items is hashed and routed; because m = 16384, each register sees about 100000 / 16384 ≈ 6 items on average and stores the largest rho among them.
raw_estimate computes Z = Σ 2^(-M[j]). A register at value 6 contributes 2^-6; larger registers contribute exponentially less. Z is therefore small when cardinality is high.
E = alpha_m × m^2 / Z inverts that relationship: small Z → large E. The m^2 factor scales the harmonic mean of the per-register estimates up to a whole-sketch cardinality.
alpha_m ≈ 0.7213 for m = 16384 corrects the systematic bias that the un-normalised harmonic mean carries. Without it the estimate is consistently a few percent high.
The result lands within roughly ±0.8% of 100,000 — the ~1% accuracy from 12 KB of registers, no item values stored. At very low or very high cardinality the raw estimate drifts, which is exactly the bias the next section corrects.

Output.

Metric	Value
True cardinality	100,000
Raw HLL estimate	~99,200–100,800
Relative error	within ~±0.8%
Registers used	16,384 (12 KB)
Item values stored	0

Rule of thumb. The estimate is alpha_m × m^2 / Σ 2^(-register) — a harmonic mean of the per-register guesses. It is accurate in the mid-range and biased at the extremes, which is why production HLLs bolt on the corrections in the next section.

Data engineering interview question on the HLL mechanism

A senior interviewer might ask: "Implement the core of HyperLogLog from scratch — the add path and the raw estimator — and explain, as you go, why you use the maximum leading-zero count per register and the harmonic mean across registers rather than the arithmetic mean. Then tell me what breaks if the hash function is not uniform."

Solution Using a from-scratch register array with max-rho updates and a harmonic-mean estimator

import hashlib

def alpha_m(m: int) -> float:
    return {16: 0.673, 32: 0.697, 64: 0.709}.get(m, 0.7213 / (1 + 1.079 / m))

class HyperLogLog:
    def __init__(self, p: int = 14):
        assert 4 <= p <= 18
        self.p, self.m, self.bits = p, 1 << p, 64
        self.regs = bytearray(self.m)              # 1 byte per register (>=6 bits)

    def _hp(self, x: str):
        h = int.from_bytes(hashlib.blake2b(x.encode(), digest_size=8).digest(), "big")
        idx = h >> (self.bits - self.p)
        w = h & ((1 << (self.bits - self.p)) - 1)
        wb = self.bits - self.p
        rho = (wb + 1) if w == 0 else (wb - w.bit_length() + 1)
        return idx, rho

    def add(self, x: str) -> None:
        idx, rho = self._hp(x)
        if rho > self.regs[idx]:                    # MAX, not sum
            self.regs[idx] = rho

    def count(self) -> float:
        Z = sum(2.0 ** (-r) for r in self.regs)
        return alpha_m(self.m) * self.m * self.m / Z

hll = HyperLogLog(p=14)
for i in range(250_000):
    hll.add(f"user-{i}")
print(f"estimate = {hll.count():,.0f}   (true = 250,000)")

Step-by-step trace.

Concern	Choice	Reasoning
Per-register value	`max(rho)`	duplicates and smaller runs are no-ops → counts distinct
Combine registers	harmonic mean	tames the exponential `2^rho` outliers LogLog suffered
Bias constant	`alpha_m`	removes systematic over-estimate of the raw formula
Hash width	64-bit	avoids collisions / large-range issues near billions
Register width	1 byte (≥6 bits)	holds rho up to 63 for a 64-bit hash

Adding an item takes one hash and a single conditional register write; the estimate is a linear scan over m registers computing alpha_m × m^2 / Σ 2^(-register). The maximum-per-register rule makes re-adding an item free, so the array measures how many distinct items were needed to produce the observed rarest patterns, and the harmonic mean keeps a single lucky long run from blowing up the estimate.

Output:

Metric	Value
True cardinality	250,000
Estimate	~248,000–252,000
Memory	16,384 bytes (register array)
Add complexity	O(1)
Count complexity	O(m)

Why this works — concept by concept:

Maximum-per-register — storing max(rho) makes the structure a set fingerprint: adding a duplicate can never raise the max, so it is idempotent, and the array reflects distinct elements rather than event volume.
Leading-zero rarity — the longest leading-zero run is an unbiased signal of how many distinct hashes were drawn, because a run of k zeros occurs with probability 2^-k; the longest run seen scales like log2(cardinality).
Harmonic mean over registers — averaging 2^register harmonically (a reciprocal sum) suppresses the outlier registers that made the earlier LogLog estimator noisy, cutting the standard error to 1.04/sqrt(m).
alpha_m bias constant — the raw harmonic-mean formula over-estimates by a fixed factor; alpha_m normalises it. A non-uniform hash breaks the whole chain, because leading-zero probabilities are no longer 2^-k and every downstream constant is wrong.
Cost — O(1) time and O(1) memory per add, O(m) to read the count, m × 6 bits total. Accuracy is set by m (standard error 1.04/sqrt(m)), completely independent of the cardinality — the property that makes billions-in-kilobytes possible.

Data structures
Topic — data-structures
Implement-the-structure problems (HLL, bloom, sketches)

Practice →

Cardinality Topic — cardinality Cardinality-estimation mechanics problems

Practice →

3. Bias correction & accuracy

The raw estimator drifts at small and large cardinalities — corrections and HLL++ pin it to ~1% across the whole range

The mental model in one line: the raw harmonic-mean estimator is accurate in the middle of the cardinality range but biased at the extremes — at low cardinality many registers are still zero and the formula over-counts, at very high cardinality (with a 32-bit hash) it saturates — so production HyperLogLog swaps in linear counting at the low end, a large-range correction at the top of a 32-bit space, and, in Google's HLL++, a 64-bit hash plus an empirically-measured bias table and a sparse encoding to be accurate and compact across the entire range. The accuracy you can promise is the relative standard error 1.04/sqrt(m), and it is set entirely by the register count.

Where the raw estimator goes wrong.

Small range (low cardinality). When the true count is below about 2.5 × m, many registers are still zero. The harmonic-mean formula systematically over-estimates here, and the fix is to switch to linear counting, which is very accurate when there are empty registers.
Large range (32-bit hash only). As the estimate approaches 2^32, hash collisions make the raw estimator under-count; the original paper adds a correction E = -2^32 × ln(1 - E/2^32). With a 64-bit hash this range is effectively unreachable and the correction is unnecessary.
Mid range. The raw estimator is already good; no correction is applied.
The seam. The original algorithm chooses which branch to use by comparing the raw estimate to thresholds, which creates a small discontinuity at the boundaries — one of the specific things HLL++ smooths out.

Linear counting — the low-cardinality rescue.

The idea. If V of the m registers are still zero, the cardinality is well estimated by E = m × ln(m / V). This is the classic "balls in bins, count empty bins" estimator and it is far more accurate than harmonic mean when the sketch is sparsely populated.
The switch. Use linear counting when the raw estimate is ≤ 2.5m and V > 0. Otherwise use the harmonic-mean estimate (with any large-range correction).
Why it matters. Without it, counting a few hundred distinct items in a 16384-register sketch can be off by several percent — embarrassing on a low-traffic dashboard.

HLL++ — Google's production upgrade (2013).

64-bit hash. Eliminates the large-range correction and pushes collision effects out past any realistic cardinality (~1.8e19). This alone removes a whole class of high-end error.
Empirical bias correction. Instead of the analytic thresholds, HLL++ ships a measured bias table for the intermediate range and interpolates, removing the discontinuities of the original branch-select.
Sparse representation. At low cardinality it stores an explicit, compressed list of (index, rho) pairs instead of the full dense register array — often kilobytes down to bytes — and only converts to dense when the sparse form grows past a threshold. This makes low-cardinality sketches both smaller and more accurate.

The accuracy contract.

Standard error. Relative standard error ≈ 1.04 / sqrt(m). This is a standard deviation, so ~68% of estimates fall within ±1 SE and ~95% within ±2 SE of the truth.
Precision knob. p sets m = 2^p sets both memory (m × 6 bits) and error. Double m (one more bit of p) and the error shrinks by sqrt(2) ≈ 1.41× while memory doubles — diminishing returns, so most systems settle at p = 14.
It is relative. The error is a percentage of the true count, so absolute error grows with cardinality but the percentage stays fixed — the opposite of a fixed-width counter.

Worked example — precision, memory, and standard error

Detailed explanation. The single most useful table for an HLL interview maps precision p to register count, memory, and standard error. Memorising the shape of it lets you answer "how accurate / how big?" instantly. Build it from the two formulas.

Memory. m × 6 bits = 2^p × 6 / 8 bytes.
Error. 1.04 / sqrt(m) = 1.04 / sqrt(2^p).
The sweet spot. p = 14 → 12 KB, ~0.81% — the Redis / common default.

Question. Tabulate memory and standard error for p from 10 to 16.

Input.

Precision p	Registers m	Formula for memory	Formula for error
10	1,024	2^10 × 6 / 8	1.04 / sqrt(2^10)
12	4,096	2^12 × 6 / 8	1.04 / sqrt(2^12)
14	16,384	2^14 × 6 / 8	1.04 / sqrt(2^14)
16	65,536	2^16 × 6 / 8	1.04 / sqrt(2^16)

Code.

import math

def hll_profile(p: int):
    m = 1 << p
    mem_bytes = m * 6 / 8
    std_err   = 1.04 / math.sqrt(m)
    return m, mem_bytes, std_err

print(f"{'p':>3} {'m':>8} {'memory':>10} {'std error':>10}")
for p in (10, 11, 12, 13, 14, 15, 16):
    m, mem, err = hll_profile(p)
    mem_str = f"{mem/1024:.1f} KB"
    print(f"{p:>3} {m:>8,} {mem_str:>10} {err*100:>9.2f}%")

Step-by-step explanation.

m = 2^p is the number of registers; each is 6 bits, so memory is m × 6 / 8 bytes. Every increment of p doubles both m and the memory.
Standard error is 1.04 / sqrt(m). Because it depends on sqrt(m), doubling m only improves error by 1.41× — you pay double memory for a ~29% error reduction. That diminishing return is why nobody uses p = 20.
p = 14 is the industry default: 16,384 registers, 12 KB, 0.81% error. Redis hard-codes this; BigQuery's default precision (15) is one step finer.
At p = 10 you get a 768-byte sketch at ~3.25% error — useful when you have millions of low-cardinality keys and can tolerate coarser counts (e.g. per-user distinct actions).
At p = 16 you get 48 KB at ~0.4% error — for a small number of high-value aggregate counters where accuracy matters more than footprint.

Output.

p	m (registers)	Memory	Standard error
10	1,024	0.75 KB	~3.25%
12	4,096	3.0 KB	~1.63%
14	16,384	12.0 KB	~0.81%
16	65,536	48.0 KB	~0.41%

Rule of thumb. Error is 1.04/sqrt(m) and memory is m × 6 bits, so accuracy improves only with the square root of memory. p = 14 (12 KB, ~0.8%) is the default; go coarser for many keys, finer for a few high-stakes counters.

Worked example — linear counting at low cardinality

Detailed explanation. Show the raw estimator drifting at low cardinality and the linear-counting correction fixing it. This is the correction that matters most in practice, because most real keys (per-user, per-page) have low cardinality.

The symptom. With m = 16384 and only 500 distinct items, the raw harmonic-mean estimate reads noticeably high.
The fix. Count zero registers V; if the raw estimate ≤ 2.5m and V > 0, use E = m × ln(m/V).
The result. Linear counting lands within a fraction of a percent where the raw formula was several percent off.

Question. Estimate a 500-distinct stream with a p = 14 sketch using both the raw and the linear-counting estimators.

Input.

Field	Value
Precision p	14 (m = 16384)
True cardinality	500
Raw threshold	E ≤ 2.5m → use linear counting
Linear formula	m × ln(m / V), V = zero registers

Code.

import hashlib, math

class HLL:
    def __init__(self, p=14):
        self.p, self.m, self.bits = p, 1 << p, 64
        self.regs = bytearray(self.m)
    def add(self, x):
        h = int.from_bytes(hashlib.blake2b(x.encode(), digest_size=8).digest(), "big")
        idx = h >> (self.bits - self.p)
        w = h & ((1 << (self.bits - self.p)) - 1)
        wb = self.bits - self.p
        rho = (wb + 1) if w == 0 else (wb - w.bit_length() + 1)
        if rho > self.regs[idx]:
            self.regs[idx] = rho
    def raw(self):
        Z = sum(2.0 ** (-r) for r in self.regs)
        a = 0.7213 / (1 + 1.079 / self.m)
        return a * self.m * self.m / Z
    def count(self):
        E = self.raw()
        if E <= 2.5 * self.m:
            V = self.regs.count(0)                 # empty registers
            if V > 0:
                return self.m * math.log(self.m / V)   # linear counting
        return E

hll = HLL(p=14)
for i in range(500):
    hll.add(f"key-{i}")

print(f"true            = 500")
print(f"raw estimate    = {hll.raw():,.0f}")
print(f"corrected count = {hll.count():,.0f}")

Step-by-step explanation.

With only 500 distinct items spread over 16,384 registers, the vast majority of registers are still zero. The harmonic-mean formula was derived assuming most registers are populated, so in this regime it over-estimates.
count() first computes the raw estimate and checks it against the 2.5m threshold. Since 500 is far below 2.5 × 16384, it enters the linear-counting branch.
Linear counting counts the empty registers V and applies E = m × ln(m/V). This is the "how many balls did it take to leave V of m bins empty" estimator, which is extremely accurate when many bins are empty.
The corrected count lands within a fraction of a percent of 500, whereas the raw estimate reads meaningfully higher. On a low-traffic page's unique-visitor counter, that is the difference between a trustworthy and an embarrassing number.
As cardinality climbs and registers fill up (V → 0), the linear branch stops firing and the harmonic-mean estimate takes over seamlessly — the two estimators agree in the crossover region around 2.5m.

Output.

Estimator	Value	Error vs 500
Raw harmonic mean	~525–560	+5% to +12%
Linear counting	~499–502	< ±0.6%
Chosen (count())	linear counting	best available

Rule of thumb. Below ~2.5m distinct items, switch to linear counting (m × ln(m/V)); it is far more accurate than the harmonic-mean formula when registers are mostly empty. This is the correction that keeps low-cardinality keys honest.

Worked example — reading the standard error as a confidence band

Detailed explanation. The standard error is a standard deviation, so it defines a confidence band, not a hard bound. Interviewers love the follow-up "so a single HLL count could be off by more than 0.8%?" — the honest answer is yes, with a known probability. Turn the SE into a band.

The band. Relative SE σ = 1.04/sqrt(m). ~68% of estimates fall within ±σ, ~95% within ±2σ, ~99.7% within ±3σ.
Absolute band. Multiply by the true count: at 10,000,000 uniques and p=14, σ ≈ 0.81% → ±81,000 (1σ), ±162,000 (2σ).
The implication. For dashboards this is fine; for anything with a hard threshold ("alert if uniques > X") you must account for the band.

Question. For a p = 14 sketch and a true count of 10,000,000, give the 1σ, 2σ, and 3σ absolute error bands.

Input.

Field	Value
Precision p	14 (m = 16384)
True cardinality	10,000,000
Relative σ	1.04 / sqrt(16384)
Bands	±1σ (68%), ±2σ (95%), ±3σ (99.7%)

Code.

import math

m = 1 << 14
sigma_rel = 1.04 / math.sqrt(m)          # ~0.0081
true_n = 10_000_000

for k in (1, 2, 3):
    band = k * sigma_rel * true_n
    pct  = k * sigma_rel * 100
    coverage = {1: "68%", 2: "95%", 3: "99.7%"}[k]
    print(f"±{k}σ ({coverage}): ±{band:,.0f} uniques  (±{pct:.2f}%)  "
          f"range [{true_n-band:,.0f} .. {true_n+band:,.0f}]")

Step-by-step explanation.

The relative standard error at p = 14 is 1.04/sqrt(16384) ≈ 0.81%. This is one standard deviation of the estimator's error distribution, which is approximately Gaussian for large m.
At a true count of 10,000,000, one relative sigma is 0.81% × 10,000,000 ≈ 81,000 uniques. So a single estimate is typically within ±81,000 but not guaranteed to be.
Two sigma (±162,000, ±1.6%) covers ~95% of estimates; three sigma (±243,000, ±2.4%) covers ~99.7%. The tail beyond that is rare but not impossible.
This is why HLL is right for dashboards and funnels (nobody notices ±0.8% on a visitor count) and wrong for billing or hard-threshold alerts (a 2σ excursion could trip or miss a threshold spuriously).
If a tighter band is required, the only lever is more registers: going to p = 16 shrinks σ to ~0.41%, halving every band at 4× the memory. There is no free accuracy — it is always paid in registers.

Output.

Band	Relative	Absolute (at 10M)	Range
±1σ (68%)	±0.81%	±81,000	9,919,000 .. 10,081,000
±2σ (95%)	±1.63%	±163,000	9,837,000 .. 10,163,000
±3σ (99.7%)	±2.44%	±244,000	9,756,000 .. 10,244,000

Rule of thumb. Treat the HLL count as true ± 1.04/sqrt(m) at one sigma — a band, not a point. Fine for dashboards; if a hard threshold depends on the number, widen your margins by 2–3σ or raise precision.

Statistics interview question on HLL accuracy

A senior interviewer might ask: "Your product team wants unique-user counts accurate to within 1% for a metric that ranges from a few hundred to tens of millions of uniques per key. Pick a precision, justify the memory, explain how you keep the low end accurate, and tell me the probability that any single reported number is off by more than 2%."

Solution Using p=14 with linear counting at the low end and a stated 2σ confidence contract

import hashlib, math

class ProdHLL:
    """p=14 HLL with linear-counting low-range correction (HLL++-style hash width)."""
    def __init__(self, p=14):
        self.p, self.m, self.bits = p, 1 << p, 64
        self.regs = bytearray(self.m)
        self.alpha = 0.7213 / (1 + 1.079 / self.m)

    def add(self, x: str):
        h = int.from_bytes(hashlib.blake2b(x.encode(), digest_size=8).digest(), "big")
        idx = h >> (self.bits - self.p)
        w = h & ((1 << (self.bits - self.p)) - 1)
        wb = self.bits - self.p
        rho = (wb + 1) if w == 0 else (wb - w.bit_length() + 1)
        if rho > self.regs[idx]:
            self.regs[idx] = rho

    def count(self) -> float:
        Z = sum(2.0 ** (-r) for r in self.regs)
        E = self.alpha * self.m * self.m / Z
        if E <= 2.5 * self.m:                       # low range
            V = self.regs.count(0)
            if V > 0:
                return self.m * math.log(self.m / V)   # linear counting
        return E

    @property
    def std_error(self) -> float:
        return 1.04 / math.sqrt(self.m)

hll = ProdHLL(p=14)
print(f"chosen precision  : p=14  (m={hll.m:,}, memory=12 KB)")
print(f"1σ relative error : {hll.std_error*100:.2f}%")
print(f"2σ relative error : {2*hll.std_error*100:.2f}%  <- ~95% of reads within this")
print(f"P(|error| > 2%)   : ~5% (2% ≈ 2.47σ) -> under 1.5%")

Step-by-step trace.

Requirement	Design choice	Result
~1% accuracy	p = 14 → σ = 0.81%	1σ inside the 1% ask
Memory budget	12 KB per key	capacity-plannable
Low-end (hundreds) accurate	linear counting below 2.5m	< ~0.6% at low cardinality
High end (tens of millions)	64-bit hash, no large-range issue	unbiased across range
"off by > 2%?"	2% ≈ 2.47σ	probability ~1.4%

The design fixes p = 14, so the standard error is 0.81% — comfortably inside the 1% target at one sigma. Linear counting handles the low end where the metric starts (a few hundred uniques), and the 64-bit hash keeps the top end (tens of millions) unbiased. Because 2% is about 2.47 standard errors, the probability that any single reported number is off by more than 2% is roughly 1.4% — small, quantified, and stated up front to the product team.

Output:

Metric	Value
Precision	p = 14
Memory per key	12 KB (dense; less when sparse)
1σ error	0.81%
2σ error	1.63%
P(error > 2%)	~1.4%

Why this works — concept by concept:

Precision picks the error — since σ = 1.04/sqrt(m) is independent of cardinality, choosing p = 14 fixes the accuracy at 0.81% for every key from hundreds to tens of millions of uniques, which is exactly the range the product team named.
Linear counting at the low end — below 2.5m the harmonic-mean estimator is biased high; m × ln(m/V) on the empty-register count is accurate to a fraction of a percent, keeping small counts honest.
64-bit hash — removes the original 32-bit large-range correction and pushes hash-collision bias past any real cardinality, so the top of the range is unbiased without special-casing.
Confidence stated as a band — reporting "±0.81% at 1σ, ~1.4% chance of exceeding 2%" turns a vague "it's approximate" into a contract the product team can reason about and sign off on.
Cost — 12 KB per key, O(1) add, O(m) count. Halving the error to 0.41% would cost 4× memory (p = 16), so p = 14 is the memory-optimal choice for a 1% target.

Statistics
Topic — statistics
Standard-error and estimator-bias problems

Practice →

Cardinality Topic — cardinality Accuracy-tuning and precision-selection problems

Practice →

4. Mergeability & sketches at scale

The union of two HyperLogLogs is the element-wise max of their registers — lossless, order-independent, and the reason HLL owns distributed counting

The mental model in one line: because each HyperLogLog register stores the maximum leading-zero count for its bucket, the union of two sketches (built at the same precision) is simply the element-wise maximum of their register arrays — this merge is exact, commutative, associative, and idempotent, so you can compute sketches independently per partition, per shard, or per time-window and combine them with zero loss, which is precisely why HyperLogLog is the default distinct-count primitive in every distributed and streaming system. COUNT(DISTINCT) results cannot be combined this way; HLL sketches can, and that single property is worth more than the memory saving in most large systems.

Why max-per-register is a correct union.

A register is a max already. Register j in sketch A holds the largest rho among items routed to bucket j in A's stream. Register j in B holds the same for B's stream. The largest rho in the combined stream is just the larger of the two — max(A[j], B[j]).
No double counting. If an item is in both A and B, it produced the same (idx, rho) in each, so max of two equal values is unchanged. Overlap is handled automatically; the merged sketch reflects the true union cardinality, not the sum.
Order-independent. max is commutative and associative, so merging in any order — A then B, B then A, or a tree of thousands of partial sketches — gives the identical result. There is no "merge order" bug to worry about.
Idempotent. Merging a sketch with itself changes nothing. Re-processing a partition (e.g. after a retry) cannot corrupt the union.

What mergeability unlocks.

Map-reduce distinct count. Each mapper builds a sketch over its shard; the reducer maxes them together and reads one count. No shuffle of raw values, no giant distributed set.
Streaming windows. Keep one sketch per minute; a "distinct over the last hour" query merges 60 minute-sketches on the fly. Sliding windows become sketch arithmetic.
Rollups. Store daily sketches; weekly = merge 7, monthly = merge ~30, quarterly = merge the months. You compute the finest granularity once and derive every coarser one by merging — never re-reading raw events.
Multi-dimensional counts. Store a sketch per (country, day); "distinct users in Europe this week" merges the relevant cells. This is how warehouses answer arbitrary-slice unique counts fast.

The one hard rule and the intersection caveat.

Same precision to merge. Two sketches can only be merged if they share the same p (same m). A p=12 and a p=14 sketch are not directly mergeable — you can down-sample the finer one to the coarser precision, but you cannot up-sample. Standardise p across a system.
Intersections are inclusion-exclusion. HLL has no direct intersection. You estimate |A ∩ B| = |A| + |B| - |A ∪ B|. The union is exact-ish; the individual counts each carry error, and subtracting two noisy numbers adds their errors — so intersections of similar-sized, small-overlap sets can be very noisy.
Jaccard / set difference. Derived the same inclusion-exclusion way, with the same error-amplification caveat. For accurate set operations, Theta sketches or MinHash are the better tools.

Common interview probes on mergeability.

"Why can you merge HLLs but not COUNT(DISTINCT) results?" — because a register is a max, and max-of-max handles overlap; two distinct counts have no overlap information.
"Does merge order matter?" — no; max is commutative and associative.
"Can you merge different precisions?" — no directly; down-sample the finer to the coarser.
"How do you intersect?" — inclusion-exclusion, and beware error amplification.

Worked example — merge two sketches by element-wise max

Detailed explanation. Implement the merge and verify it equals a fresh sketch built over the concatenated streams. This equivalence — "merge of parts == sketch of whole" — is the property everything else relies on.

Merge. merged[j] = max(A[j], B[j]) for all j.
The check. Build A over stream 1, B over stream 2, merge; separately build C over stream 1 + stream 2; the register arrays must be identical.
The payoff. Distributed counting is provably lossless, not approximately so.

Question. Merge two p = 14 sketches and confirm the merged registers equal a sketch built over both streams together.

Input.

Field	Value
Sketch A stream	users 0..599,999
Sketch B stream	users 400,000..999,999 (200k overlap)
True union cardinality	1,000,000
Merge rule	element-wise max

Code.

import hashlib

class HLL:
    def __init__(self, p=14):
        self.p, self.m, self.bits = p, 1 << p, 64
        self.regs = bytearray(self.m)
        self.alpha = 0.7213 / (1 + 1.079 / self.m)
    def add(self, x):
        h = int.from_bytes(hashlib.blake2b(x.encode(), digest_size=8).digest(), "big")
        idx = h >> (self.bits - self.p)
        w = h & ((1 << (self.bits - self.p)) - 1)
        wb = self.bits - self.p
        rho = (wb + 1) if w == 0 else (wb - w.bit_length() + 1)
        if rho > self.regs[idx]:
            self.regs[idx] = rho
    def merge(self, other):
        assert self.p == other.p, "precisions must match"
        for j in range(self.m):
            if other.regs[j] > self.regs[j]:
                self.regs[j] = other.regs[j]
    def count(self):
        Z = sum(2.0 ** (-r) for r in self.regs)
        return self.alpha * self.m * self.m / Z

A, B, C = HLL(), HLL(), HLL()
for i in range(0, 600_000):        A.add(f"user-{i}")
for i in range(400_000, 1_000_000): B.add(f"user-{i}")
for i in range(0, 1_000_000):      C.add(f"user-{i}")   # whole stream directly

A.merge(B)
print("merged == whole:", A.regs == C.regs)      # True — lossless
print(f"union estimate = {A.count():,.0f}  (true = 1,000,000)")

Step-by-step explanation.

Sketch A covers users 0–599,999 and B covers 400,000–999,999, so they overlap on 200,000 users. The true union is 1,000,000 distinct users, not 1,200,000.
merge walks the register arrays and keeps the larger value at each index. Because each register is already a per-bucket max, the merged register is the max over the combined stream for that bucket.
The assertion A.regs == C.regs passes: the merge of two partial sketches is byte-for-byte identical to a sketch built over the whole stream at once. This is the lossless property, demonstrated rather than asserted.
The overlapping 200,000 users contribute the same (idx, rho) in both A and B, so max collapses them automatically — the union estimate is ~1,000,000, not ~1,200,000. No dedup logic was needed; the structure dedups by construction.
Because A.regs == C.regs, the estimate is exactly what you would have gotten from the monolithic sketch — merging costs nothing in accuracy, only the O(m) pass over registers.

Output.

Quantity	Value
Sketch A distinct	~600,000
Sketch B distinct	~600,000
Naive A + B (wrong)	~1,200,000
Merged union estimate	~1,000,000
merged registers == whole-stream registers	True

Rule of thumb. Merge is element-wise max, and "merge of parts" is byte-identical to "sketch of the whole." Overlap is handled for free — never add two HLL counts, always merge the sketches then count once.

Worked example — map-reduce distinct count across partitions

Detailed explanation. Scale the merge to the distributed setting: many workers each sketch their shard, a reducer merges all partials, one count comes out. This is the canonical big-data distinct-count job and it moves sketches (12 KB each) across the network, not raw values.

Map. Each of k workers builds a sketch over its shard.
Reduce. Max all k sketches together (a tree-reduce works because merge is associative).
Shuffle cost. k × 12 KB, independent of the number of rows — often megabytes total where the raw data is terabytes.

Question. Sketch 8 partitions independently and reduce them into a single distinct-count.

Input.

Field	Value
Partitions	8
Rows per partition	~2,000,000 (overlapping keyspace)
True global distinct	5,000,000
Shuffle payload	8 × 12 KB = 96 KB

Code.

import hashlib
from functools import reduce

# (HLL class as before: add(), merge(), count(), p=14)
def make_partition_sketch(key_range):
    h = HLL(p=14)
    for i in key_range:
        h.add(f"user-{i}")
    return h

# 8 partitions drawn from a 5,000,000-key universe (with overlap)
import random
random.seed(1)
partitions = []
for _ in range(8):
    start = random.randint(0, 3_000_000)
    partitions.append(make_partition_sketch(range(start, start + 2_000_000)))

# Reduce: merge all partial sketches (associativity makes this safe in any order)
def merge_two(a, b):
    a.merge(b)
    return a

global_sketch = reduce(merge_two, partitions)
print(f"partitions          : {len(partitions)}")
print(f"shuffle payload      : {len(partitions) * 12} KB (sketches, not rows)")
print(f"global distinct est.: {global_sketch.count():,.0f}")

Step-by-step explanation.

Each partition builds its own 12 KB sketch over ~2,000,000 rows. No cross-partition communication happens during the map phase — the workers are fully independent, which is what makes the job embarrassingly parallel.
The partitions' key ranges overlap (drawn from a shared 5,000,000-key universe), so naively summing their eight individual counts would massively over-count. The merge handles the overlap.
reduce(merge_two, partitions) maxes the eight sketches together. Because merge is associative and commutative, a driver-side linear fold and a distributed tree-reduce yield the identical result — you can reduce in whatever topology is cheapest.
The only data crossing the network is eight 12 KB sketches — 96 KB total — no matter that the underlying data is ~16,000,000 rows. Contrast an exact distinct count, which must shuffle every distinct key to a common reducer.
The final count() on the merged sketch estimates the global distinct at ~5,000,000 within the usual ~0.8% band. One O(m) pass produces the answer for the entire dataset.

Output.

Metric	Value
Partitions sketched	8
Rows processed	~16,000,000
Shuffle payload	96 KB (8 sketches)
Global distinct estimate	~5,000,000
Exact-count shuffle (for contrast)	all distinct keys → one reducer

Rule of thumb. Distributed distinct count = "sketch per partition, merge the sketches." You move kilobytes of sketch instead of terabytes of keys, and associativity lets you reduce in any tree shape.

Worked example — daily → weekly rollups and the intersection caveat

Detailed explanation. Two everyday operations: rolling daily sketches up to weekly (a merge), and estimating an intersection (inclusion-exclusion, with its error caveat). Both come up constantly and the intersection one is a classic trap.

Rollup. Weekly distinct = merge of 7 daily sketches; monthly = merge of ~30. Compute daily once, derive the rest.
Intersection. |A ∩ B| = |A| + |B| - |A ∪ B|. Correct in expectation but noisy, because it subtracts two error-bearing estimates.
The trap. Reporting an intersection of two large, barely-overlapping sets as if it were as accurate as a union.

Question. Roll up 7 daily sketches to a weekly count, and estimate the overlap between two of the days via inclusion-exclusion.

Input.

Field	Value
Daily sketches	7 (one per day)
Weekly count	merge all 7, then count
Day-A distinct	~1,000,000
Day-B distinct	~1,000,000
True A∩B	~300,000

Code.

import hashlib
from functools import reduce
# (HLL class as before, p=14)

def day_sketch(users):
    h = HLL(p=14)
    for u in users:
        h.add(f"user-{u}")
    return h

# 7 days, each ~1,000,000 users, heavily overlapping population
days = [day_sketch(range(d * 100_000, d * 100_000 + 1_000_000)) for d in range(7)]

# Weekly rollup = merge of the 7 daily sketches
weekly = reduce(lambda a, b: (a.merge(b) or a), days, HLL(p=14))
print(f"weekly distinct  = {weekly.count():,.0f}")

# Intersection of day 0 and day 1 via inclusion-exclusion
A, B = days[0], days[1]
union = HLL(p=14); union.merge(A); union.merge(B)
inter = A.count() + B.count() - union.count()      # |A| + |B| - |A ∪ B|
print(f"|A|={A.count():,.0f}  |B|={B.count():,.0f}  |A∪B|={union.count():,.0f}")
print(f"estimated |A∩B| = {inter:,.0f}   (true ≈ 900,000 overlap here)")

Step-by-step explanation.

Each day gets its own sketch during ingestion. The weekly rollup is a single merge of the seven daily register arrays — no raw events are touched, so changing the reporting granularity is nearly free.
Because the daily populations overlap heavily, the weekly distinct is far less than the sum of the seven daily counts; the merge's automatic dedup produces the correct union.
The intersection uses inclusion-exclusion: |A ∩ B| = |A| + |B| - |A ∪ B|. Each of the three terms is an HLL estimate with its own ~0.8% error.
The subtraction is where accuracy degrades: if A and B are each ~1,000,000 with 0.8% error (~±8,000 each) and the union is ~1,100,000, the intersection is a difference of large numbers, so the absolute errors of all three terms pile onto a comparatively small result. For small true overlaps the relative error can be enormous.
The rule that follows: merges and unions are safe and accurate; intersections and set-differences derived from them are only as good as the gap between the numbers being subtracted. For accurate intersections, reach for Theta sketches or MinHash instead of HLL inclusion-exclusion.

Output.

Operation	Result	Reliability
Weekly rollup (merge 7)	~1.6M distinct	high (union)
\|A\|	~1,000,000	~0.8% error
\|B\|	~1,000,000	~0.8% error
\|A ∪ B\|	~1,100,000	~0.8% error
\|A ∩ B\| (incl-excl)	~900,000	noisy (errors add)

Rule of thumb. Rollups are merges and are accurate; intersections are inclusion-exclusion and amplify error. Trust HLL unions and rollups freely; treat HLL intersections as rough, and switch to Theta/MinHash when set overlap must be precise.

Data engineering interview question on mergeable sketches

A senior interviewer might ask: "Design a system that reports distinct active users at daily, weekly, monthly, and per-country granularity over billions of daily events, and lets analysts ask arbitrary date-range and country-combination questions without re-scanning raw logs. Explain what you store, how a query is answered, and the one rule that makes it all work."

Solution Using per-(country, day) HLL sketches merged on demand

import hashlib
from functools import reduce
# (HLL class as before, p=14; add(), merge(), count())

# Storage: one sketch per (country, day). Built once during ingest.
#   sketch_store[(country, day)] -> 12 KB HLL
def ingest_event(store, country, day, user_id):
    key = (country, day)
    if key not in store:
        store[key] = HLL(p=14)
    store[key].add(user_id)

def distinct_users(store, countries, days):
    """Answer any (set of countries) x (date range) distinct-user query by merging."""
    acc = HLL(p=14)
    for c in countries:
        for d in days:
            s = store.get((c, d))
            if s is not None:
                acc.merge(s)
    return acc.count()

# --- usage ---
store = {}
# ... ingest_event(store, country, day, user_id) for every event ...

# Queries are all merges of the stored cells:
# weekly_global   = distinct_users(store, ALL_COUNTRIES, last_7_days)
# monthly_europe  = distinct_users(store, EU_COUNTRIES,  last_30_days)
# custom_range    = distinct_users(store, ["US","CA"],   ["2026-09-01","2026-09-02"])

-- The same design in a warehouse: store a sketch column per (country, day),
-- then MERGE the relevant cells at query time (BigQuery HLL_COUNT.* shown).
CREATE TABLE uv_sketch AS
SELECT country, event_date,
       HLL_COUNT.INIT(user_id, 14) AS sketch      -- one 12 KB-ish sketch per cell
FROM   events
GROUP  BY country, event_date;

-- Arbitrary date-range + country-set distinct users: merge, don't re-scan
SELECT HLL_COUNT.MERGE(sketch) AS distinct_users
FROM   uv_sketch
WHERE  country IN ('US','CA')
  AND  event_date BETWEEN '2026-09-01' AND '2026-09-07';

Step-by-step trace.

Layer	What is stored / done	Why
Ingest	one HLL per (country, day)	finest granularity, computed once
Storage	~12 KB per cell	bounded; cells × 12 KB
Weekly query	merge 7 days' cells	no raw re-scan
Per-country query	merge that country's cells	slice by dimension
Arbitrary combo	merge the matching cells	any slice = a merge
Hard rule	all sketches share p = 14	mergeability requires equal precision

The system stores exactly one sketch per finest-grain cell (country, day). Every analyst question — any date range, any set of countries — is answered by merging the matching cells and reading one count. Raw logs are never re-scanned to change granularity or slice; the whole query surface reduces to "select the cells, merge, count."

Output:

Query	How answered	Cost
Global daily	one cell-group merge per day	O(cells) merge
Global weekly	merge 7 days × all countries	O(cells) merge
Europe monthly	merge EU countries × 30 days	O(cells) merge
Custom country+range	merge matching cells	O(cells) merge
Storage	cells × 12 KB	bounded, plannable

Why this works — concept by concept:

Finest-grain sketch cells — storing one sketch per (country, day) and deriving every coarser view by merging means you compute the expensive part once and answer all granularities from it.
Any slice is a merge — because merge is associative and commutative, an arbitrary set of cells combines to the correct union regardless of order, so any date-range × country-set query is just "merge the matching cells."
No raw re-scan — changing granularity (day → week → month) or dimension (country → region) never touches the event logs; it merges 12 KB sketches, turning terabyte scans into kilobyte merges.
Equal precision is the one rule — every sketch is built at p = 14; sketches of different precision cannot be merged, so standardising p system-wide is what makes the whole design compose.
Cost — O(number of cells) to merge and O(m) to count per query, with storage of cells × 12 KB — independent of event volume. An exact design would re-aggregate raw events per query, an O(rows) cost every time.

Cardinality
Topic — cardinality
Mergeable-sketch and rollup problems

Practice →

Data structures Topic — data-structures Set-operation and union/intersection problems

Practice →

5. HLL in practice — Redis / BigQuery / Spark

The same hash-registers-estimate idea ships as a one-line function in Redis, BigQuery, and Spark — learn the API once, use it everywhere

The mental model in one line: every major data platform exposes HyperLogLog as a small, high-level API — Redis via PFADD/PFCOUNT/PFMERGE, BigQuery via APPROX_COUNT_DISTINCT and the composable HLL_COUNT.* family, Spark via approx_count_distinct(col, rsd) — and although the surface syntax differs, they all implement the identical hash → registers → harmonic-mean estimate you now understand, so the skill transfers directly: pick precision/error, add items, and (in Redis and BigQuery) store and merge portable sketches for rollups. Knowing which knob controls accuracy and which operation is a merge lets you use any of them correctly on day one.

Redis — PFADD / PFCOUNT / PFMERGE.

The commands. PFADD key element [element ...] adds items; PFCOUNT key [key ...] returns the estimate (of one key, or the union of several); PFMERGE dest src [src ...] writes the merged sketch to a new key. The PF prefix honours Philippe Flajolet.
The footprint. A dense Redis HLL is capped at 12 KB (p = 14, ~0.81% error); low-cardinality keys use a sparse encoding that can be a few dozen bytes and auto-upgrades to dense as they grow.
The idiom. One key per (entity, time-bucket) — uv:page:42:2026-09-05. Daily uniques are PFCOUNT; weekly uniques are PFCOUNT over 7 daily keys (implicit union) or a materialised PFMERGE.
The gotcha. PFCOUNT of multiple keys computes their union on the fly; it is not the sum of their individual counts. That is a feature (correct overlap handling) that surprises people expecting addition.

BigQuery — APPROX_COUNT_DISTINCT and HLL_COUNT.*.

The one-liner. APPROX_COUNT_DISTINCT(col) returns an approximate distinct count directly — the everyday tool for dashboards.
The composable family. HLL_COUNT.INIT(col, precision) builds a sketch (a storable BYTES value); HLL_COUNT.MERGE(sketch) unions sketches and returns the count; HLL_COUNT.MERGE_PARTIAL(sketch) unions sketches and returns a new sketch (for staged rollups); HLL_COUNT.EXTRACT(sketch) reads the count from one sketch without merging.
Precision. INIT's precision runs 10–24 (default 15). Higher precision → less error, bigger sketch — the same 1.04/sqrt(m) trade.
The pattern. Materialise HLL_COUNT.INIT sketches per finest-grain cell; answer any slice with HLL_COUNT.MERGE. Portable sketches also cross engines (BigQuery, Dataflow, and Java/Go libraries share a compatible format).

Spark — approx_count_distinct(col, rsd).

The function. approx_count_distinct(col, rsd) where rsd is the target relative standard deviation (default 0.05 = 5%). Smaller rsd → more registers → more memory and time.
The mapping. rsd is the accuracy knob that indirectly picks p: rsd = 1.04/sqrt(m), so rsd = 0.02 implies m ≈ 2704 → p ≈ 12. Set rsd to the error you can tolerate; do not over-tighten it.
The distributed win. Spark builds per-partition sketches and merges them in the aggregation — the mergeability from section 4 is exactly what lets it run distributed with a tiny shuffle.
Cousins. Presto/Trino: approx_distinct(col, e) and a first-class HyperLogLog type with merge(). Postgres: the postgresql-hll extension with an hll column type and hll_union_agg. ClickHouse: uniqHLL12 and the default uniq.

Common interview probes on HLL in practice.

"What does PFCOUNT key1 key2 return?" — the union estimate, not the sum.
"How do you do rollups in BigQuery?" — store HLL_COUNT.INIT sketches; HLL_COUNT.MERGE on query.
"What is Spark's rsd?" — target relative standard error; it picks the register count.
"Can sketches from different systems merge?" — only if the format and precision match (BigQuery/Dataflow do; Redis's format is its own).

Worked example — Redis daily unique-visitor rollup

Detailed explanation. The most common production HLL job: track unique visitors per page per day in Redis, then roll up to weekly. Show the commands and the Python client calls.

Keys. uv:page:{id}:{date} — one HLL per page per day.
Add. PFADD on each pageview.
Read. PFCOUNT for the day; multi-key PFCOUNT or PFMERGE for the week.

Question. Track daily unique visitors for a page and produce the weekly unique count.

Input.

Field	Value
Daily key	uv:page:42:2026-09-05
Weekly key	uv:page:42:2026-W36
Add	PFADD per pageview
Weekly	PFMERGE 7 daily keys

Code.

# Redis CLI — one HLL key per page per day
PFADD uv:page:42:2026-09-05 visitor_a visitor_b visitor_c
PFADD uv:page:42:2026-09-05 visitor_a            # duplicate: no effect
PFCOUNT uv:page:42:2026-09-05                     # -> ~3 (approx daily uniques)

# Weekly rollup: merge 7 daily keys into a weekly key
PFMERGE uv:page:42:2026-W36 \
        uv:page:42:2026-09-01 uv:page:42:2026-09-02 uv:page:42:2026-09-03 \
        uv:page:42:2026-09-04 uv:page:42:2026-09-05 uv:page:42:2026-09-06 \
        uv:page:42:2026-09-07
PFCOUNT uv:page:42:2026-W36                        # -> approx weekly uniques (union)

# Union on the fly without materialising (multi-key PFCOUNT):
PFCOUNT uv:page:42:2026-09-01 uv:page:42:2026-09-02 uv:page:42:2026-09-03

import redis

r = redis.Redis()

def record_view(page_id: int, date: str, visitor_id: str) -> None:
    r.pfadd(f"uv:page:{page_id}:{date}", visitor_id)

def daily_uniques(page_id: int, date: str) -> int:
    return r.pfcount(f"uv:page:{page_id}:{date}")

def weekly_uniques(page_id: int, dates: list[str]) -> int:
    keys = [f"uv:page:{page_id}:{d}" for d in dates]
    return r.pfcount(*keys)                 # multi-key PFCOUNT = union estimate

record_view(42, "2026-09-05", "visitor_a")
record_view(42, "2026-09-05", "visitor_a")   # dup — free, no effect
record_view(42, "2026-09-05", "visitor_b")
print("daily :", daily_uniques(42, "2026-09-05"))
print("weekly:", weekly_uniques(42, [f"2026-09-0{d}" for d in range(1, 8)]))

Step-by-step explanation.

PFADD on each pageview routes the visitor id into the day's sketch. Re-adding the same visitor is idempotent, so refreshes and repeat visits do not inflate the count — the "distinct" semantics are built in.
PFCOUNT on the daily key returns the approximate distinct visitors for that day, at ~0.81% error, from a ≤ 12 KB structure regardless of how popular the page is.
The weekly rollup uses PFMERGE to union the seven daily sketches into a weekly key, then PFCOUNT reads it. Because merge dedups, a visitor who came every day is counted once in the week.
The multi-key PFCOUNT variant computes the union without materialising a merged key — handy for ad-hoc ranges. Crucially it returns the union, not the sum of the daily counts; expecting addition is the classic mistake.
Storage scales with the number of live (page, day) keys, not with visitor volume. Low-traffic pages stay in the sparse encoding (tens of bytes); only busy pages reach the 12 KB dense cap.

Output.

Command	Result
PFADD (new visitor)	1 (register changed)
PFADD (duplicate)	0 (no change)
PFCOUNT daily	~ distinct visitors that day
PFMERGE + PFCOUNT weekly	~ distinct visitors that week (union)
Memory per key	≤ 12 KB (sparse when small)

Rule of thumb. In Redis, one HLL key per (entity, time-bucket): PFADD on write, PFCOUNT to read, PFMERGE (or multi-key PFCOUNT) to roll up. Remember multi-key PFCOUNT is a union, never a sum.

Worked example — BigQuery APPROX_COUNT_DISTINCT and staged HLL_COUNT rollups

Detailed explanation. In a warehouse the everyday tool is APPROX_COUNT_DISTINCT, but the real power is the HLL_COUNT.* family that lets you store sketches and merge them later — the same "sketch per cell, merge on query" design from section 4, expressed in SQL.

Everyday. APPROX_COUNT_DISTINCT(user_id) in any SELECT.
Store. HLL_COUNT.INIT(user_id, 15) materialises a sketch per group.
Merge. HLL_COUNT.MERGE(sketch) for the count; HLL_COUNT.MERGE_PARTIAL(sketch) for staged rollups that emit a new sketch.

Question. Show the one-liner, then a two-stage rollup that stores daily sketches and merges them to weekly.

Input.

Field	Value
Fact table	events(user_id, event_date, country)
One-liner	APPROX_COUNT_DISTINCT(user_id)
Daily sketch	HLL_COUNT.INIT(user_id, 15)
Weekly merge	HLL_COUNT.MERGE(sketch)

Code.

-- 1. Everyday one-liner: approximate distinct users per day
SELECT event_date,
       APPROX_COUNT_DISTINCT(user_id) AS approx_dau
FROM   events
GROUP  BY event_date;

-- 2. Stage 1 — materialise one sketch per (country, day)
CREATE OR REPLACE TABLE uv_daily_sketch AS
SELECT country,
       event_date,
       HLL_COUNT.INIT(user_id, 15) AS sketch      -- precision 15
FROM   events
GROUP  BY country, event_date;

-- 3. Stage 2 — weekly uniques per country by MERGING daily sketches (no re-scan)
SELECT country,
       DATE_TRUNC(event_date, WEEK)      AS wk,
       HLL_COUNT.MERGE(sketch)           AS weekly_uniques
FROM   uv_daily_sketch
GROUP  BY country, wk;

-- 4. Staged rollup that emits a NEW sketch (weekly sketch kept for monthly merge)
CREATE OR REPLACE TABLE uv_weekly_sketch AS
SELECT country,
       DATE_TRUNC(event_date, WEEK)          AS wk,
       HLL_COUNT.MERGE_PARTIAL(sketch)       AS sketch   -- returns a sketch, not a count
FROM   uv_daily_sketch
GROUP  BY country, wk;

-- 5. Global distinct across any slice: merge the matching cells
SELECT HLL_COUNT.MERGE(sketch) AS distinct_users
FROM   uv_daily_sketch
WHERE  country IN ('US','CA')
  AND  event_date BETWEEN '2026-09-01' AND '2026-09-07';

Step-by-step explanation.

APPROX_COUNT_DISTINCT(user_id) is the drop-in replacement for COUNT(DISTINCT user_id) — same query shape, but backed by HLL so it stays fast and cheap on huge tables. Use it for any dashboard where ~1% error is fine.
HLL_COUNT.INIT(user_id, 15) builds and stores one sketch per group as a BYTES value. Precision 15 (BigQuery's default) gives ~0.65% error; you can pass 10–24 to trade accuracy for size.
HLL_COUNT.MERGE(sketch) unions the sketches in a group and returns the distinct count in one step — this is how the weekly-per-country query derives weekly uniques from stored daily sketches without re-reading events.
HLL_COUNT.MERGE_PARTIAL(sketch) unions sketches but returns a new sketch instead of a count, so you can build a weekly sketch table and later merge those into monthly, monthly into quarterly — staged rollups where each level is derived from the one below.
Any arbitrary slice (a set of countries over a date range) is answered by HLL_COUNT.MERGE over the matching daily cells — the SQL expression of "select cells, merge, count." The events table is scanned once to build the sketches; every downstream question hits the compact sketch table.

Output.

Query	Returns	Re-scans events?
APPROX_COUNT_DISTINCT	approx count	yes (one pass)
HLL_COUNT.INIT	stored sketch per cell	one-time build
HLL_COUNT.MERGE	count from merged sketches	no
HLL_COUNT.MERGE_PARTIAL	new merged sketch	no
slice query	count for any country/date set	no

Rule of thumb. Use APPROX_COUNT_DISTINCT for ad-hoc dashboards; use HLL_COUNT.INIT + HLL_COUNT.MERGE/MERGE_PARTIAL to store finest-grain sketches once and derive every rollup and slice without re-scanning the fact table.

Worked example — Spark approx_count_distinct and tuning rsd

Detailed explanation. In Spark the accuracy knob is rsd (relative standard deviation), and understanding that it is the 1.04/sqrt(m) error lets you set it correctly instead of guessing. Show the call and the memory/accuracy trade of tightening rsd.

The call. approx_count_distinct(col, rsd); default rsd = 0.05 (5% error).
The mapping. rsd = 1.04/sqrt(m) → m = (1.04/rsd)^2. Halving rsd quadruples m and the memory.
The advice. Set rsd to the error you actually need; over-tightening it (e.g. 0.001) blows up memory for accuracy nobody will notice.

Question. Count approximate distinct users per day in Spark at 2% error, and show what rsd implies for the register count.

Input.

Field	Value
Column	user_id
Target error	2% → rsd = 0.02
Implied m	(1.04/0.02)^2 ≈ 2,704
Default rsd	0.05 (5%)

Code.

from pyspark.sql import functions as F

# Approximate distinct users per day at ~2% error
daily_uniques = (
    events
    .groupBy("event_date")
    .agg(F.approx_count_distinct("user_id", rsd=0.02).alias("approx_dau"))
)
daily_uniques.show()

# What does rsd imply for register count / memory?
def m_from_rsd(rsd: float) -> int:
    return round((1.04 / rsd) ** 2)

for rsd in (0.05, 0.02, 0.01, 0.005):
    m = m_from_rsd(rsd)
    print(f"rsd={rsd:<6}  m≈{m:>8,}  memory≈{m*6/8/1024:6.1f} KB")

Step-by-step explanation.

approx_count_distinct("user_id", rsd=0.02) tells Spark to build HLL sketches sized so the relative standard error is ~2%. Spark computes per-partition sketches and merges them during the shuffle — the mergeability from section 4 is what makes this distributed.
The rsd parameter is the accuracy contract: rsd = 1.04/sqrt(m), so rsd = 0.02 implies m ≈ (1.04/0.02)^2 ≈ 2704 registers (roughly p = 12). You are choosing register count indirectly by choosing tolerable error.
The loop shows the quadratic cost: halving rsd from 0.02 to 0.01 quadruples m (~2,704 → ~10,816) and the per-sketch memory. Accuracy is expensive, and it is bought in squares.
The default rsd = 0.05 (5%) is coarse but cheap; for reporting-grade numbers 0.01–0.02 is typical. Going to 0.001 would demand ~1,000,000 registers per group — rarely justified.
Because Spark shuffles sketches rather than raw ids, a distinct-count aggregation over billions of rows stays cheap in network and memory — the same "move kilobytes not terabytes" win, wrapped in one function call.

Output.

rsd	Implied m	Approx memory	Use case
0.05	~433	~0.3 KB	rough, default
0.02	~2,704	~2.0 KB	reporting
0.01	~10,816	~7.9 KB	high accuracy
0.005	~43,264	~31.7 KB	rarely needed

Rule of thumb. In Spark, rsd is the relative standard error — set it to the accuracy you actually need. Because m = (1.04/rsd)^2, tightening rsd costs memory quadratically; 0.01–0.02 is the usual reporting range.

Data engineering interview question on choosing an HLL engine

A senior interviewer might ask: "You need real-time per-page unique visitors on a live dashboard and historical unique-user analytics with arbitrary date-range and segment slicing over a warehouse. Which HLL implementations do you use for each, how do the sketches flow between the real-time and batch worlds, and what precision do you standardise on?"

Solution Using Redis for real-time and BigQuery HLL sketches for historical, standardised at ~p=14/15

# Real-time tier (Redis) — sub-millisecond per-page uniques
PFADD  uv:page:42:2026-09-05  <visitor_id>      # on every pageview (idempotent)
PFCOUNT uv:page:42:2026-09-05                    # live dashboard read (~0.8% error)
# Nightly: export the day's raw (or de-duped) ids to the warehouse for the batch tier.

-- Batch/historical tier (BigQuery) — arbitrary slicing via stored sketches
-- Build finest-grain sketches once, precision 15 (~0.65% error).
CREATE OR REPLACE TABLE uv_daily_sketch AS
SELECT country, event_date,
       HLL_COUNT.INIT(user_id, 15) AS sketch
FROM   events
GROUP  BY country, event_date;

-- Any historical question = merge the matching cells (no fact re-scan)
SELECT HLL_COUNT.MERGE(sketch) AS distinct_users
FROM   uv_daily_sketch
WHERE  country IN ('US','CA','MX')
  AND  event_date BETWEEN '2026-08-01' AND '2026-08-31';

Step-by-step trace.

Concern	Real-time (Redis)	Historical (BigQuery)
Latency	sub-ms PFADD/PFCOUNT	seconds per query
Granularity	per (page, day)	per (country, day) sketch cells
Accuracy	p=14 → ~0.81%	precision 15 → ~0.65%
Rollups	PFMERGE	HLL_COUNT.MERGE / MERGE_PARTIAL
Slicing	key patterns	merge any matching cells
Cross-tier	export ids nightly	rebuild sketches from ids

Redis serves the live dashboard: PFADD on every event, PFCOUNT for instant per-page uniques, ≤ 12 KB per key. The warehouse serves history: build HLL_COUNT.INIT sketches per (country, day) once, then answer any date-range/segment question by merging cells. The two tiers do not share a binary sketch format (Redis's encoding is its own), so the hand-off is the raw/de-duped ids exported nightly and re-INIT-ed in BigQuery — each tier keeps its native, compatible sketches.

Output:

Need	Engine	API	Why
Live per-page uniques	Redis	PFADD/PFCOUNT	sub-ms, in-memory
Live rollup	Redis	PFMERGE	union daily keys
Historical slicing	BigQuery	HLL_COUNT.MERGE	any cell combination
Staged rollups	BigQuery	HLL_COUNT.MERGE_PARTIAL	daily→weekly→monthly sketches
Ad-hoc count	BigQuery	APPROX_COUNT_DISTINCT	one-off dashboards

Why this works — concept by concept:

Right tool per latency class — Redis's in-memory HLL answers in sub-millisecond time for a live dashboard, while BigQuery's stored sketches answer arbitrary historical slices in seconds; each plays to its strength instead of forcing one engine to do both.
Sketch-per-cell in the warehouse — HLL_COUNT.INIT per (country, day) plus HLL_COUNT.MERGE at query time is the section-4 design in SQL: compute finest grain once, derive every slice by merging, never re-scan facts.
Standardised precision — both tiers sit near p = 14/15 (~0.6–0.8% error), so accuracy is consistent across the product and nobody is surprised by a tier that reports a different number.
Cross-tier via ids, not binary sketches — Redis and BigQuery do not share an HLL wire format, so the nightly hand-off ships ids and re-builds sketches natively, rather than pretending incompatible sketches can be merged.
Cost — Redis: ≤ 12 KB per live key, O(1) ops. BigQuery: one fact scan to build sketches, then O(cells) merges per query and cells × sketch storage. Both are bounded and independent of raw event volume, which is the entire point of using HLL over exact distinct counts.

Cardinality
Topic — cardinality
Applied distinct-count problems (Redis / warehouse / Spark)

Practice →

Statistics
Topic — statistics
Error-budget and precision-tuning problems

Practice →

Cheat sheet — HyperLogLog recipes

When to reach for HLL. Use HyperLogLog when cardinality can exceed a few million, you can tolerate ~1% error, and you need fixed memory or mergeable partial counts. Use an exact set/COUNT(DISTINCT) when the count is small or must be perfect (billing, compliance). Use a Roaring bitmap when keys are small dense integers and you want exactness cheaply.
The estimator. E = alpha_m × m^2 / Σ_j 2^(-M[j]), a harmonic mean over m = 2^p registers. alpha_m = 0.7213/(1 + 1.079/m) for m ≥ 128 (0.673/0.697/0.709 for m=16/32/64). Register M[j] = max leading-zero-count (rho) routed to bucket j.
The add path. h = hash(item); idx = top p bits; rho = position of leftmost 1-bit in the remaining bits; M[idx] = max(M[idx], rho). O(1), idempotent for duplicates — which is why it counts distinct items.
Precision → memory → error. p=10→0.75 KB/~3.25%; p=12→3 KB/~1.6%; p=14→12 KB/~0.81% (Redis default); p=16→48 KB/~0.41%. Error = 1.04/sqrt(m); memory = m × 6 bits. Accuracy improves only with sqrt(memory).
Low-cardinality correction. If the raw estimate ≤ 2.5m and V registers are still zero, use linear counting E = m × ln(m/V) — far more accurate than harmonic mean when registers are mostly empty.
HLL++ upgrades. 64-bit hash (kills the 32-bit large-range correction and collision bias), an empirical bias-correction table for the intermediate range (removes branch-select discontinuities), and a sparse encoding for low cardinality (bytes instead of 12 KB, and more accurate).
Merge = element-wise max. merged[j] = max(A[j], B[j]). Lossless, commutative, associative, idempotent; "merge of parts" is byte-identical to "sketch of the whole." Requires identical precision p.
Never add HLL counts. To combine sketches, merge the register arrays then count once — adding two counts double-counts the overlap. Redis multi-key PFCOUNT and BigQuery HLL_COUNT.MERGE both compute the union, not the sum.
Rollups. Store finest-grain sketches (per day, per cell); derive weekly/monthly/segment counts by merging. Compute the expensive aggregation once; every coarser or sliced view is a cheap merge with no raw re-scan.
Intersections are noisy. |A ∩ B| = |A| + |B| - |A ∪ B| via inclusion-exclusion; subtracting error-bearing estimates amplifies error, so small overlaps of large sets are unreliable. Use Theta sketches or MinHash when set overlap must be accurate.
Redis recipe. One key per (entity, time-bucket): PFADD key elem… to write, PFCOUNT key to read (or PFCOUNT k1 k2… for a union), PFMERGE dst src… to materialise a rollup. Dense cap 12 KB; sparse for small keys.
Warehouse / Spark recipe. BigQuery: APPROX_COUNT_DISTINCT(col) for ad-hoc; HLL_COUNT.INIT(col, p) + HLL_COUNT.MERGE/MERGE_PARTIAL for stored, mergeable sketches. Spark: approx_count_distinct(col, rsd) where rsd = 1.04/sqrt(m) and m = (1.04/rsd)^2 — set rsd to the error you need; tightening it costs memory quadratically.

Frequently asked questions

What is HyperLogLog in one sentence?

HyperLogLog is a probabilistic data structure (a "sketch") that estimates the number of distinct elements in a stream — its cardinality — using a small fixed amount of memory and a bounded relative error, by hashing each element, measuring how many leading zeros appear in the hash, and keeping only the maximum such count per bucket in a tiny array of registers. Because it never stores the elements themselves, it can count billions of distinct values in around 12 kilobytes with roughly 0.8% error, which is why it powers approximate distinct count everywhere from Redis to BigQuery to Spark. It is the standard answer to "count uniques at scale" in both production systems and data-engineering interviews.

How accurate is HyperLogLog?

HyperLogLog's accuracy is set by its register count m = 2^p: the relative standard error is 1.04 / sqrt(m), which is about 0.81% at the common p = 14 (16,384 registers, 12 KB) configuration. That figure is a standard deviation, so roughly 68% of estimates fall within ±1 error-unit of the truth, ~95% within ±2, and ~99.7% within ±3 — a single reported number can be off by more than the headline percentage with a known, small probability. You improve accuracy only by adding registers, and because the error shrinks with the square root of m, halving the error costs four times the memory. Low-cardinality counts stay accurate thanks to a linear-counting correction, and modern HLL++ implementations use a 64-bit hash and an empirical bias table to stay unbiased across the whole range.

How much memory does HyperLogLog use?

A HyperLogLog uses m × 6 bits, where m = 2^p is the register count and 6 bits holds a leading-zero count up to 63 (enough for a 64-bit hash). At the default p = 14 that is 16,384 registers × 6 bits = 12,288 bytes, about 12 KB — and, crucially, this footprint is fixed regardless of cardinality, so the same 12 KB counts a hundred distinct values or a hundred billion. Coarser precisions are smaller (p = 10 ≈ 0.75 KB) and finer ones larger (p = 16 ≈ 48 KB). Implementations like Redis and Google's HLL++ also use a sparse representation for low-cardinality sketches, storing an explicit list of non-zero registers in as little as a few dozen bytes and only expanding to the dense array as the count grows.

Can you merge HyperLogLog sketches?

Yes — mergeability is HyperLogLog's defining superpower. The union of two sketches built at the same precision is the element-wise maximum of their register arrays, and this merge is lossless, commutative, associative, and idempotent, so "merge of the parts" equals "sketch of the whole" exactly. This is what makes HLL the default for distributed and streaming distinct counts: each partition, shard, or time-window builds its own sketch independently, and a reducer combines them by taking maxes — moving kilobytes of sketch instead of terabytes of raw keys. The one hard rule is that the sketches must share the same precision p; you cannot directly merge a p=12 and a p=14 sketch (you can only down-sample the finer one). Note that exact COUNT(DISTINCT) results cannot be merged this way, because two counts carry no information about their overlap.

HyperLogLog vs COUNT(DISTINCT) — when do I pick each?

Pick exact COUNT(DISTINCT) when the cardinality is small enough to fit an exact hash set in memory, or when the answer must be perfect — billing, licence counts, regulatory reporting, anything where an off-by-1% is unacceptable. Pick HyperLogLog when cardinality can reach millions or billions, when you need a fixed memory footprint you can capacity-plan, or when you need mergeable partial counts for distributed aggregation and rollups. The core trade is exactness versus scalability: exact counting is Θ(cardinality) memory and cannot be merged; HLL is Θ(1) memory with ~0.8% error and merges losslessly. Most warehouses make this explicit — COUNT(DISTINCT col) for exactness, APPROX_COUNT_DISTINCT(col) for the HLL-backed fast path — and senior engineers reach for the approximate version by default on dashboards and the exact version only when correctness is contractual.

Can HyperLogLog compute intersections?

Not directly — HyperLogLog only supports union (element-wise max) natively. You can estimate an intersection via inclusion-exclusion: |A ∩ B| = |A| + |B| - |A ∪ B|, where the union is obtained by merging. The problem is error amplification: each of the three terms carries its own ~0.8% error, and subtracting large, error-bearing estimates to produce a comparatively small result can make the intersection wildly inaccurate, especially when the true overlap is small relative to the set sizes. So HLL intersections are fine as a rough signal but should not be trusted for precise set-overlap, Jaccard similarity, or set-difference metrics. When you need accurate intersections, use Theta sketches (which support set operations directly with error bounds) or MinHash (built for Jaccard similarity).

Practice on PipeCode

Drill the cardinality practice library → for the distinct-count, sketch, and rollup problems senior interviewers love.
Sharpen the fundamentals on the data-structures practice library → for probabilistic structures — HyperLogLog, Bloom filters, count-min sketch, and MinHash.
Build estimation intuition on the statistics practice library → for standard error, bias correction, and confidence-band reasoning.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the memory-vs-accuracy trade against real graded inputs.

Lock in cardinality-estimation muscle memory

Docs explain the algorithm. PipeCode drills explain the decision — when an exact set OOMs and a sketch survives, why sampling fails for cardinality, when the low-cardinality linear-counting correction fires, why you merge sketches instead of adding counts, and when HLL intersections are too noisy to trust. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice cardinality problems →
Practice data-structure problems →

Bloom Filters for Data Engineers: Cheap Membership Tests

Gowtham Potureddi — Mon, 07 Sep 2026 15:52:16 +0000

bloom filters are the data structure you reach for when the real question is not "what is the value?" but "have I seen this key before?" — and you need the answer in constant time, in a handful of bits per element, against a set far too large to keep in memory as a hash table. A bloom filter is a probabilistic data structure that answers a membership test with one of exactly two verdicts: definitely not in the set, or possibly in the set. That asymmetry is the whole trick. It never says "yes" with certainty, but it never says "no" incorrectly — so a downstream system can treat a "definitely not" as gospel and skip an expensive lookup, a disk read, or a network round-trip entirely, while treating a "maybe" as a cheap pre-filter that occasionally lets a non-member through.

The price of that speed and space is a tunable false positive rate — the probability that the filter says "maybe" for a key it has never seen. You buy the filter's compactness with that error, and the entire engineering discipline of bloom filters is about sizing the bit array and choosing the number of hash functions so the false positive rate lands exactly where your pipeline can tolerate it. Get the math right and a filter for a hundred million keys fits in a couple of hundred megabytes at a tenth of a percent error; get it wrong and either the filter overflows into uselessness or you waste memory you did not need to spend. This guide walks the contract, the sizing math, the deletion-capable and growth-capable variants, the way LSM-tree databases wire a bloom filter in front of every SSTable, and a tuned Python implementation you can lift into a streaming dedup job.

When you want hands-on reps immediately after reading, drill the data-structures practice library →, rehearse hashing on the hash-table practice library →, and sharpen the sizing math on the optimization practice library →.

On this page

What a bloom filter actually is
The math — bits, hashes, and false-positive-rate sizing
Variants — counting and scalable bloom filters
Bloom filters in databases — LSM and SSTable
Building and tuning one in Python
Cheat sheet — bloom filter recipes
Frequently asked questions
Practice on PipeCode

1. What a bloom filter actually is

The one-sided-error membership test — "definitely not" is certain, "maybe" is probabilistic

The one-sentence invariant: a bloom filter is a fixed-size bit array plus k independent hash functions that records set membership by turning bits on, answers a membership test with either "definitely not present" (certain, zero false negatives) or "possibly present" (a small tunable false positive rate), and buys that compactness by discarding the ability to enumerate or delete the set it stores. Everything else about bloom filters — the sizing formulas, the counting variant, the LSM integration — is a consequence of this single asymmetric contract, and the fastest way to lose a senior interview is to describe a bloom filter as "a fast hash set" instead of naming the one-sided error up front.

The two moving parts.

The bit array. A vector of m bits, all initialised to 0. This is the entire stored state — there are no keys, no pointers, no values. A bloom filter for a billion URLs holds no URLs; it holds a few billion bits, most of them related to no single URL in particular.
The k hash functions. k independent hash functions h_1 … h_k, each mapping an element to a bit position in [0, m). In practice these are not k separate functions but two good hashes combined by double hashing (section 2). The choice of k is not free — there is a mathematically optimal value for any given m and expected element count n.
No stored elements. Because the filter never stores the element itself, its memory footprint is independent of how large each element is. A filter over 1 KB log lines costs exactly as many bits per element as a filter over 8-byte integers. This size-independence is the property that makes bloom filters irreplaceable at scale.

Insert and query — set bits, then test bits.

Insert(x). Compute the k hash positions for x and set each of those bits to 1 (a bitwise OR). If a bit was already 1 from a previous element, it stays 1 — collisions are expected and harmless on insert.
Query(x). Compute the same k positions for x. If any of those k bits is 0, x was definitely never inserted — return "definitely not". If all k bits are 1, return "maybe": either x was inserted, or the k bits it maps to were all coincidentally set by other elements (a false positive).
Why there are no false negatives. Insert only ever turns bits on, and a plain bloom filter never turns them off. So every bit that a genuine member set on insert is still 1 at query time. A genuine member can therefore never produce a 0 bit, which means it can never be reported "definitely not". Absence of deletion is exactly what guarantees absence of false negatives.

Why "probabilistic" is the honest label.

The false positive. As more elements are inserted, more bits flip to 1. Eventually a never-inserted key can hash to k positions that other elements already lit. The filter reports "maybe" for a key it has never seen — a false positive. The rate is not random noise; it is a precise function of m, n, and k.
The tunable error. You choose the false positive rate before you build the filter by sizing m for your expected n. A 1% rate costs about 9.6 bits per element; a 0.1% rate costs about 14.4 bits; a 0.01% rate about 19.2. The rate is a design parameter, not an accident.
The space win. An exact hash set of 100 million 40-byte URLs needs several gigabytes. A bloom filter at 1% false positive rate needs roughly 120 MB — one to two orders of magnitude smaller — because it stores bits about elements, not elements.

What interviewers listen for.

Do you say "no false negatives, only false positives" in the first two sentences? — required answer.
Do you name the verdict as "definitely not" vs "maybe" rather than "yes/no"? — senior signal.
Do you explain that memory is independent of element size? — senior signal.
Do you note that a plain bloom filter cannot delete or enumerate, and reach for a counting or cuckoo filter when deletion is required? — senior signal.
Do you frame the false positive rate as a design knob rather than "some small error"? — required answer.

Worked example — hand-simulating insert and query

Detailed explanation. The fastest way to internalise the contract is to run a tiny filter by hand: a 10-bit array, 3 hash functions, and three inserted elements, then probe both a member and a non-member. Because everything is small, you can watch each bit flip and see exactly how a false positive is born.

Filter shape. m = 10 bits, indices 0…9, k = 3 hash functions.
Hashes (illustrative). For each element the three functions produce three positions in [0, 10).
Members inserted. "alice", "bob", "carol".

Question. After inserting the three members, what does the filter answer for "bob" (a member) and for "dave" and "erin" (non-members)?

Input.

Element	h1	h2	h3	Role
alice	1	4	9	insert
bob	4	6	7	insert
carol	0	4	6	insert
dave	2	5	8	query
erin	1	6	7	query

Code.

Start:   index  0 1 2 3 4 5 6 7 8 9
         bits   0 0 0 0 0 0 0 0 0 0

insert alice {1,4,9}:   0 1 0 0 1 0 0 0 0 1
insert bob   {4,6,7}:   0 1 0 0 1 0 1 1 0 1
insert carol {0,4,6}:   1 1 0 0 1 0 1 1 0 1   <- final state

query bob   {4,6,7}: bits[4]=1, bits[6]=1, bits[7]=1  -> ALL set  -> "maybe" (true member)
query dave  {2,5,8}: bits[2]=0                          -> a 0     -> "definitely not"
query erin  {1,6,7}: bits[1]=1, bits[6]=1, bits[7]=1  -> ALL set  -> "maybe" (FALSE POSITIVE)

Step-by-step explanation.

The array starts all zero. Inserting "alice" sets bits 1, 4, and 9. Inserting "bob" sets 4 (already on — no change), 6, and 7. Inserting "carol" sets 0, 4 (already on), and 6 (already on). The final state has six bits lit.
Querying "bob" checks bits 4, 6, 7 — all 1, so the filter says "maybe". "bob" really is a member, so this is a true positive.
Querying "dave" checks bits 2, 5, 8. Bit 2 is 0, so the filter short-circuits to "definitely not" — and indeed "dave" was never inserted. A single 0 is a certificate of absence.
Querying "erin" checks bits 1, 6, 7. Bit 1 was set by "alice", bit 6 by "bob"/"carol", bit 7 by "bob". All three are 1, so the filter says "maybe" — but "erin" was never inserted. This is a false positive, produced entirely by other elements' bits coinciding.
Notice the query short-circuits: the moment it sees a 0, it can stop and answer "definitely not". Only "maybe" requires checking all k bits.

Output.

Query	Bits checked	Result	Truth
bob	4,6,7 → 1,1,1	maybe	true member
dave	2,5,8 → 0,…	definitely not	correct
erin	1,6,7 → 1,1,1	maybe	false positive

Rule of thumb. A bloom filter query stops at the first 0 and returns "definitely not"; it returns "maybe" only when every one of the k bits is set. False positives are not bugs — they are the designed-in cost of storing bits instead of elements, and their rate is what the sizing math controls.

Worked example — when a bloom filter beats an exact set

Detailed explanation. The decision to use a bloom filter is an economic one: it is worth it when the exact structure does not fit in memory, when a "definitely not" saves a genuinely expensive operation, and when the workload can tolerate a small rate of wasted work on false positives. Walk through the classic "should I hit the backend?" cache scenario.

The workload. A read-through cache in front of a slow object store. Most requested keys do not exist; each miss costs a 20 ms round-trip to confirm absence.
The alternative. An exact in-memory hash set of all existing keys — correct, but too large to hold on every edge node.
The bloom option. A small per-node bloom filter of all existing keys. "Definitely not" answers a miss locally in microseconds; "maybe" falls through to the real lookup.

Question. For 50 million existing keys and a request stream that is 80% non-existent keys, how much backend traffic does a 1% bloom filter remove?

Input.

Parameter	Value
Existing keys (n)	50,000,000
Requests	100,000,000
Fraction non-existent	80% (80,000,000)
Bloom false positive rate	1%
Exact set size (40-byte keys)	~3–4 GB
Bloom size @ 1%	~60 MB

Code.

# Traffic saved by a bloom pre-filter on the miss path
requests            = 100_000_000
nonexistent         = 80_000_000        # truly absent keys
fpr                 = 0.01

# Absent keys the bloom correctly rejects locally (no backend call)
rejected_locally    = nonexistent * (1 - fpr)   # 79,200,000
# Absent keys that slip through as false positives (wasted backend call)
false_positives     = nonexistent * fpr         #    800,000

print(f"absent requests             : {nonexistent:,}")
print(f"rejected locally by bloom   : {int(rejected_locally):,}")
print(f"false positives to backend  : {int(false_positives):,}")
print(f"backend miss-calls removed  : {rejected_locally / nonexistent:.1%}")

Step-by-step explanation.

Of the 80 million requests for absent keys, the bloom filter rejects (1 − 0.01) = 99% of them locally — 79.2 million round-trips never happen.
The remaining 1% — 800,000 requests — are false positives that fall through to the backend and confirm absence there. That is the wasted work you paid for with the false positive rate.
The existing-key requests (20 million) all hit "maybe" and proceed normally; the bloom never blocks a real member because it has no false negatives.
The memory trade is stark: the exact set costs 3–4 GB per node, the bloom about 60 MB — roughly 60× smaller — small enough to replicate to every edge node.
The net effect is that 79.2 million of 80 million miss round-trips vanish. The filter converted an expensive network confirmation of absence into a local bit test, at the cost of 0.8 million wasted confirmations.

Output.

Metric	Without bloom	With 1% bloom
Backend calls for absent keys	80,000,000	800,000
Per-node memory	3–4 GB (exact set)	~60 MB
Miss round-trips removed	0	79,200,000 (99%)
Wasted calls (false positives)	0	800,000

Rule of thumb. Reach for a bloom filter when a "definitely not" avoids expensive work (disk read, network call, decompression), when the exact set will not fit where you need it, and when the workload can absorb a small false-positive rate of wasted work. If any of those three is false, use an exact hash set instead.

Data structures interview question on the membership contract

A senior interviewer often opens with: "You are designing a service that ingests billions of events and must drop events it has already seen. Someone proposes a bloom filter for dedup. Explain precisely what guarantee a bloom filter gives you, what failure mode you inherit, and whether that failure mode is safe for a dedup use case — then state what you would change if it is not."

Solution Using the one-sided-error contract and its dedup implications

The guarantee
-------------
  query(x) == "definitely not"  =>  x was NEVER inserted   (certain)
  query(x) == "maybe"           =>  x MIGHT have been inserted
                                    (true member OR false positive)

  No false negatives. Possible false positives at rate p.

Applying it to dedup (keep first occurrence, drop repeats)
----------------------------------------------------------
  on event e:
      if e in bloom:            # "maybe"  -> treat as duplicate -> DROP
          drop(e)
      else:                     # "definitely not" -> first sight -> KEEP
          keep(e)
          bloom.add(e)

Failure mode inherited
----------------------
  A false positive => "maybe" for an event we have NOT seen
                   => we DROP a genuinely new event  (DATA LOSS)

  There are no false negatives, so we never KEEP a true duplicate.

Step-by-step trace.

Event	In bloom?	Verdict	Action	Correct?
e1 (new)	no	definitely not	keep + add	yes
e1 (repeat)	yes	maybe	drop	yes
e2 (new)	no	definitely not	keep + add	yes
e3 (new, false positive)	yes	maybe	drop	NO — data loss

e1 arrives new: the filter says "definitely not", so we keep it and insert it. Correct.
e1 arrives again: the filter says "maybe", we treat it as a duplicate and drop it. Correct — this is exactly the dedup we wanted.
e2 arrives new: "definitely not", keep and insert. Correct.
e3 arrives new but happens to hash to already-set bits: "maybe". We drop a genuinely new event. This is the inherited failure mode — a false positive in a dedup filter is silent data loss, not a duplicate that slips through.

Output:

Property	Bloom dedup result
Duplicates that slip through	0 (no false negatives)
New events wrongly dropped	~p × (new events) — data loss
Direction of error	drops real data, never keeps duplicates
Safe when	occasional lost event is acceptable (metrics, sampling)
Unsafe when	every event must survive (billing, audit)

Why this works — concept by concept:

One-sided error — the filter's only mistake is "maybe" for a non-member; it never says "definitely not" for a member. In dedup this means the error direction is dropping new data, never keeping duplicates. You must decide whether that direction is tolerable.
False positive as data loss — because a "maybe" triggers a drop, every false positive discards a real event. For lossy analytics (approximate counts, sampled telemetry) this is fine; for exactly-once billing it is a correctness bug.
The safe redesign — if loss is unacceptable, use the bloom only as a cheap pre-filter: on "maybe", confirm against an exact store before dropping. The bloom removes the expensive check for the 99% "definitely not" cases and only the rare "maybe" pays for an exact lookup.
Cost — O(k) hashing per event and ~9.6·n bits of memory at 1%. The exact-confirm fallback adds one authoritative lookup per false positive, i.e. O(p·n) extra lookups. That is the price of turning lossy dedup into lossless dedup.

DATA STRUCTURES
Topic — data-structures
Probabilistic and set-membership data-structure problems

Practice →

HASHING Topic — hash-table Hash-table and hashing fundamentals

Practice →

2. The math — bits, hashes, and false-positive-rate sizing

Bit-fill probability, the false-positive-rate formula, and how to size m and k

The mental model in one line: the false positive rate of a bloom filter is a closed-form function of three numbers — the bit-array size m, the number of inserted elements n, and the number of hash functions k — so you never guess a bloom filter's error, you compute it, and you invert the formula to size m for whatever error your pipeline can tolerate. Every senior bloom-filter conversation lives or dies on whether you can state optimal k = (m/n) ln 2 and "about 9.6 bits per element per 1% error" without reaching for a calculator.

The bit-fill probability — the foundation of every formula.

One bit staying zero. Each of the k·n bit-sets (n elements × k hashes) picks a position uniformly at random. The probability a specific bit is not chosen by one set is (1 − 1/m); after k·n sets it is (1 − 1/m)^{kn}, which for large m approximates e^{−kn/m}.
One bit being one. The complement: P(bit = 1) ≈ 1 − e^{−kn/m}. As you insert more elements (n grows) or use more hashes (k grows), more bits flip to 1 and the array saturates.
The load ratio. The exponent kn/m is the real driver. It is the expected number of bit-sets per bit. Keep it moderate and the array stays sparse; push it high and nearly every bit is 1 and the filter becomes useless.

The false-positive-rate formula.

The core equation. A false positive happens when all k queried bits are 1 for a non-member. Treating the bits as independent, p ≈ (1 − e^{−kn/m})^k. This is the number you quote and the number you minimise.
Independence caveat. The bits are not perfectly independent, so the exact rate is very slightly different, but the approximation is excellent for the m sizes used in practice and is the standard interview answer.
Monotonic in n. For fixed m and k, p only rises as n rises. A bloom filter has a design capacity: exceed the n you sized for and the error climbs past your target.

Optimal k — the number of hashes that minimises error.

The result. For fixed m and n, p is minimised at k* = (m/n) · ln 2 ≈ 0.693 · (m/n). Fewer hashes leave too many 0 bits testable (higher p for large keys); more hashes saturate the array too fast.
The error at optimal k. Substituting k* gives p ≈ (1/2)^{k*} = 2^{−k*}, equivalently p ≈ 0.6185^{m/n}. Each additional bit per element multiplies the error by about 0.6185.
The intuition. At the optimum, exactly half the bits are 1 and half are 0 — the array carries the maximum information per bit, and each hash independently has a 50% chance of hitting a 0 for a non-member.

Sizing — inverting the formula for a target p.

Bits from a target rate. Solve for m: m = −(n · ln p) / (ln 2)^2. Since (ln 2)^2 ≈ 0.4805, this is m/n ≈ −1.4427 · log2(p) bits per element.
The numbers to memorise. 1% → ~9.6 bits/elem, k=7; 0.1% → ~14.4 bits/elem, k=10; 0.01% → ~19.2 bits/elem, k=13. Each 10× reduction in error costs about 4.8 more bits per element.
Then k. With m chosen, set k = round((m/n) · ln 2). Round to the nearest integer; a fractional hash is not a thing.

Double hashing — faking k hashes with two.

The problem. Computing k independent, strong hashes per element is expensive when k is 7–13.
Kirsch–Mitzenmacher. Compute two hashes h1(x) and h2(x), then derive the i-th index as g_i(x) = (h1(x) + i · h2(x)) mod m for i = 0 … k−1. This yields asymptotically the same false-positive rate as k independent hashes.
The practical recipe. Use a single 128-bit hash (e.g. MurmurHash3) and split it into two 64-bit halves for h1 and h2. One hash call, k cheap combinations.

Worked example — computing the false positive rate for a given filter

Detailed explanation. Before sizing, make sure you can evaluate the forward formula: given m, n, k, what error do you actually have? This is the check you run when you inherit someone else's filter and want to know if it is over- or under-provisioned.

The filter. m = 8,000,000 bits, n = 1,000,000 elements, k = 5 hashes.
Load ratio. kn/m = 5·1,000,000 / 8,000,000 = 0.625.
Goal. Compute p and check whether k = 5 is optimal for this m/n.

Question. What is the false positive rate of this filter, and is its k chosen well?

Input.

Parameter	Value
m (bits)	8,000,000
n (elements)	1,000,000
k (hashes)	5
m/n (bits per element)	8

Code.

import math

m, n, k = 8_000_000, 1_000_000, 5

# Probability a given bit is still 0 after n*k sets
p_zero = math.exp(-k * n / m)          # e^{-kn/m}
p_one  = 1 - p_zero

# False positive rate: all k queried bits are 1
fpr = p_one ** k
print(f"P(bit=1)          = {p_one:.4f}")
print(f"false positive p  = {fpr:.4%}")

# Optimal k for this m/n, and the error it would give
k_opt = (m / n) * math.log(2)
fpr_opt = 0.5 ** k_opt
print(f"optimal k         = {k_opt:.2f} (use {round(k_opt)})")
print(f"fpr at optimal k  = {fpr_opt:.4%}")

Step-by-step explanation.

The load ratio kn/m = 0.625, so P(bit = 0) = e^{−0.625} ≈ 0.535 and P(bit = 1) ≈ 0.465. Fewer than half the bits are set — the filter is under-hashed for its size.
The false positive rate is 0.465^5 ≈ 0.0217, i.e. about 2.17%. That is the real error this filter delivers.
The optimal k for m/n = 8 is 8 · ln 2 ≈ 5.55, so k = 6 would be slightly better than k = 5.
At the optimal k, the error would be 0.5^{5.55} ≈ 0.0214 — about 2.14%. In this case k = 5 versus k = 6 barely matters because m/n = 8 fundamentally caps the error near 2%.
The lesson: with only 8 bits per element you are stuck near a 2% floor no matter how you pick k. To reach 1% you need more bits, not a cleverer k.

Output.

Quantity	Value
P(bit = 1)	0.465
False positive rate (k=5)	2.17%
Optimal k for m/n=8	~6
FPR at optimal k	2.14%
Verdict	error is bits-limited, not k-limited

Rule of thumb. When the false positive rate is too high, first check whether you are bits-limited: if m/n is small, no choice of k will save you. Add bits per element (raise m) to move the error floor, then set k = round((m/n)·ln 2).

Worked example — sizing a filter for a target false positive rate

Detailed explanation. The common task is the inverse: you know how many elements you expect and the error you can tolerate, and you must produce m and k. Do this for a URL-dedup filter at three target rates so you can feel the bits-per-element cost curve.

Expected elements. n = 10,000,000 distinct URLs.
Targets. 1%, 0.1%, 0.01%.
Deliverable. m in bits and megabytes, plus k, for each target.

Question. Size the bit array and choose k for n = 10M at each of the three target rates.

Input.

Target p	−ln p	m/n = −ln p / (ln 2)^2
0.01	4.605	9.585
0.001	6.908	14.378
0.0001	9.210	19.170

Code.

import math

def size_bloom(n: int, p: float) -> tuple[int, int]:
    """Return (m_bits, k) for n elements at target false positive rate p."""
    m = math.ceil(-(n * math.log(p)) / (math.log(2) ** 2))
    k = max(1, round((m / n) * math.log(2)))
    return m, k

for p in (0.01, 0.001, 0.0001):
    m, k = size_bloom(10_000_000, p)
    print(f"p={p:<7} m={m:>12,} bits  ({m/8/1e6:6.1f} MB)  k={k}")

# p=0.01    m=  95,850,584 bits  (  12.0 MB)  k=7
# p=0.001   m= 143,775,876 bits  (  18.0 MB)  k=10
# p=0.0001  m= 191,701,168 bits  (  24.0 MB)  k=13

Step-by-step explanation.

For 1%, m/n ≈ 9.585, so m ≈ 95.85M bits ≈ 12.0 MB, and k = round(9.585 · ln 2) = round(6.64) = 7.
For 0.1%, m/n ≈ 14.378, so m ≈ 143.8M bits ≈ 18.0 MB, and k = round(9.96) = 10.
For 0.01%, m/n ≈ 19.17, so m ≈ 191.7M bits ≈ 24.0 MB, and k = round(13.29) = 13.
Every 10× tightening of the error adds a flat ~4.8 bits per element (~6 MB here) and ~3 more hashes. The cost is linear in the number of nines you want, not exponential — going from 1% to 0.0001% only doubles the memory.
Crucially, none of these depend on how long the URLs are. Whether the URLs average 40 or 400 bytes, the filter is 12/18/24 MB. That is the size-independence property paying off.

Output.

Target p	m (bits)	Size	k
1%	95,850,584	12.0 MB	7
0.1%	143,775,876	18.0 MB	10
0.01%	191,701,168	24.0 MB	13

Rule of thumb. Memorise "~9.6 bits per element per 1%, plus ~4.8 bits for each extra 10×", and you can size any bloom filter in your head. Always size m for the maximum n you expect — a bloom filter sized for too few elements silently exceeds its target error as it fills.

Worked example — double hashing to avoid k separate hash calls

Detailed explanation. Computing 10 independent hashes per element would dominate the CPU cost of a hot dedup loop. The Kirsch–Mitzenmacher double-hashing trick computes one strong hash and derives all k indices from it, with no measurable loss in false-positive rate.

One 128-bit hash. Split into two 64-bit lanes h1, h2.
Derive k indices. g_i = (h1 + i·h2) mod m.
Guard. Ensure h2 is odd/non-zero relative to m so the sequence does not collapse to a single index.

Question. Implement the k index generator using double hashing from a single MurmurHash3 128-bit digest.

Input.

Component	Value
Base hash	mmh3.hash128 (128-bit)
Lanes	h1 = high 64 bits, h2 = low 64 bits
Derivation	g_i = (h1 + i·h2) mod m
k	7

Code.

import mmh3   # pip install mmh3 (MurmurHash3)

def bloom_indexes(item: str, m: int, k: int):
    """Yield k bit positions in [0, m) using Kirsch-Mitzenmacher double hashing."""
    digest = mmh3.hash128(item, signed=False)   # 128-bit
    h1 = digest & 0xFFFFFFFFFFFFFFFF            # low 64 bits
    h2 = digest >> 64                           # high 64 bits
    if h2 == 0:
        h2 = 1                                  # avoid a degenerate constant sequence
    for i in range(k):
        yield (h1 + i * h2) % m

print(list(bloom_indexes("alice", m=97, k=7)))
# e.g. [12, 45, 78, 14, 47, 80, 16]  (positions in [0, 97))

Step-by-step explanation.

A single mmh3.hash128 call produces 128 bits of well-distributed hash. Splitting it gives two 64-bit values that behave like two independent hash functions.
The generator produces g_0 = h1, g_1 = h1 + h2, g_2 = h1 + 2·h2, and so on, each reduced modulo m. Seven cheap additions replace seven full hash computations.
The if h2 == 0 guard prevents the pathological case where every g_i collapses to h1 — which would turn the k-hash filter into a 1-hash filter and wreck the error rate.
Because h1 and h2 are effectively independent, the derived indices are close enough to independent that the false-positive rate matches the k-independent-hash formula to within noise.
The same digest is reused for insert and query, so both paths cost exactly one hash call plus k modular additions — the double-hashing recipe is what makes bloom filters cheap enough for per-event use.

Output.

Metric	k independent hashes	Double hashing
Hash computations per op	k (e.g. 7)	1
Extra work per op	—	k additions + mods
False-positive rate	baseline	matches baseline
Typical speedup	1×	~5–10× on the hash step

Rule of thumb. Never compute k independent hashes in a production bloom filter. Take one 128-bit MurmurHash, split it into two 64-bit lanes, and derive all k indices with h1 + i·h2 mod m. It is the standard implementation in every serious library.

Optimization interview question on false-positive-rate sizing

A senior interviewer might ask: "You need a bloom filter for 100 million keys with at most a 0.1% false positive rate, and it must fit in under 256 MB of RAM. Show me the sizing math — bit-array size, bits per element, the number of hash functions — confirm it fits the memory budget, and explain what happens to the error if the key count grows to 150 million without a rebuild."

Solution Using the closed-form sizing formulas and a capacity-overrun analysis

import math

n_design = 100_000_000     # keys we size for
p_target = 0.001           # 0.1% target FPR
budget_bytes = 256 * 1024 * 1024

# 1. Size m and k for the design point
m = math.ceil(-(n_design * math.log(p_target)) / (math.log(2) ** 2))
k = max(1, round((m / n_design) * math.log(2)))
size_mb = m / 8 / 1e6
print(f"m = {m:,} bits = {size_mb:.1f} MB,  k = {k},  fits budget: {m/8 < budget_bytes}")

# 2. What the SAME filter's FPR becomes if n grows to 150M without a rebuild
def fpr(m, n, k):
    return (1 - math.exp(-k * n / m)) ** k

for n_actual in (100_000_000, 125_000_000, 150_000_000):
    print(f"n={n_actual:,}  ->  fpr={fpr(m, n_actual, k):.3%}")

Step-by-step trace.

Step	Computation	Value
bits per element	−ln(0.001)/(ln 2)^2	14.378
m	100M × 14.378	~1.4378 × 10^9 bits
m in MB	m / 8 / 1e6	~179.7 MB
budget check	179.7 MB < 256 MB	fits
k	round(14.378 × ln 2)	10

Bits per element for 0.1% is −ln(0.001)/(ln 2)^2 ≈ 14.378.
Multiplying by the design count gives m ≈ 1.4378 × 10^9 bits, which is ≈ 179.7 MB — comfortably under the 256 MB budget.
The optimal k is round(14.378 · ln 2) = round(9.966) = 10 hash functions.
At the design point (n = 100M), the achieved error is the target 0.1%.
Overrun analysis: keeping m and k fixed but pushing n to 125M raises the error to about 0.28%, and at 150M to about 0.64% — the load ratio kn/m climbed, so more bits are set and the error grew super-linearly past the design capacity.

Output:

Actual n	Load kn/m	False positive rate
100,000,000 (design)	~0.693	~0.10%
125,000,000	~0.867	~0.28%
150,000,000	~1.040	~0.64%

Why this works — concept by concept:

Bits-per-element formula — m/n = −ln p / (ln 2)^2 turns a target error directly into a bit budget. For 0.1% that is ~14.4 bits, so 100M keys need ~180 MB regardless of key length.
Optimal k — k = (m/n)·ln 2 ≈ 10 puts the array at the 50%-full sweet spot where each hash has an independent coin-flip chance of hitting a 0 for a non-member, minimising the error for that m.
Capacity is real — the sizing holds only at the design n. Because p = (1 − e^{−kn/m})^k rises with n, overrunning the count degrades the error fast: 1.5× the keys turned 0.1% into 0.64%, over 6× worse.
The fix for growth — if n may grow, either size m for the maximum expected count up front, or use a scalable bloom filter (section 3) that adds capacity while bounding the compounded error.
Cost — O(k) hashing per operation (10 modular additions via double hashing) and O(m) = ~180 MB of memory. The exact set of 100M 40-byte keys would cost several gigabytes, so the bloom is ~20× smaller at 0.1% error.

HASHING
Topic — hash-table
Hashing and hash-function distribution problems

Practice →

OPTIMIZATION Topic — optimization Space–time trade-off and sizing problems

Practice →

3. Variants — counting and scalable bloom filters

Deletion via counters, unbounded growth via a stack of filters, and the cuckoo alternative

The mental model in one line: the plain bloom filter's two hard limitations — it cannot delete and it has a fixed capacity — are each solved by a named variant: the counting bloom filter replaces every bit with a small counter so elements can be removed by decrementing, and the scalable bloom filter chains a geometric series of ordinary filters with tightening error targets so the set can grow without ever breaching a bounded compounded false positive rate. Knowing which variant answers which limitation, and what each costs, is the difference between "I'd use a bloom filter" and a defensible design.

Why a plain bloom cannot delete.

Shared bits. A single 1 bit may have been set by many elements. Clearing the k bits of one element to delete it would also clear bits that other elements rely on — instantly creating false negatives, which violates the core contract.
The consequence. A plain bloom filter is append-only. Its error rises monotonically with inserts and it can never shrink. For any workload with churn (a sliding window, an expiring cache, a "seen in the last hour" set) you need deletion, which means a different variant.

Counting bloom filters — bits become small counters.

The structure. Replace each bit with a c-bit counter (commonly 4 bits). Insert increments the k counters; delete decrements them; query tests whether all k counters are non-zero.
The space cost. A 4-bit counter is 4× the memory of a single bit, so a counting bloom filter costs roughly 4× a plain one for the same m. That is the price of deletion.
Counter overflow. A 4-bit counter saturates at 15. If a position is set more than 15 times, it must stick at 15 (saturating arithmetic) — decrementing a saturated counter would risk a false negative, so saturated counters are conventionally frozen. Choose c so overflow is astronomically unlikely: at optimal k, counters are Poisson-distributed with mean ln 2 ≈ 0.69, and 4 bits gives overflow odds around 10^{−15} per counter.
Delete safety. Only delete elements you know were inserted. Decrementing counters for a never-inserted element corrupts the filter and can create false negatives for real members.

Scalable bloom filters — a geometric stack.

The idea. Maintain a list of ordinary bloom filters. Insert always goes into the newest (active) filter. When the active filter reaches its capacity (its slice fills to the target error), freeze it and append a new, larger filter with a tighter target error. Query checks every filter and returns "maybe" if any says "maybe".
Bounding the compounded error. If filter i targets p_0 · r^i for a ratio r ∈ (0,1) (commonly 0.8–0.9), the overall error across the stack is bounded by p_0 / (1 − r) — a finite number no matter how many filters accumulate. Tightening each successive filter is what keeps the sum convergent.
The growth factor. Each new filter is sized larger than the last (a growth factor s, often 2), so the number of filters grows only logarithmically with the total element count. Query cost is O(number of filters) hash-and-test passes — small because the count is logarithmic.

The cuckoo filter — the modern alternative.

What it is. A cuckoo filter stores a short fingerprint of each element in a cuckoo hash table with two candidate buckets per element. It supports deletion natively (remove the fingerprint), and for target error rates below about 3% it is more space-efficient than a bloom filter.
Why it is often preferred. Lookups touch only two buckets (two cache lines), giving better locality than a bloom filter's k scattered bit probes; and deletion is exact rather than requiring 4× counters.
The trade-off. Inserts can fail when buckets are full (requiring a resize), and the space advantage reverses at very high error rates. For churny, latency-sensitive membership sets, cuckoo filters are the current default; bloom filters remain simplest for append-only sets.

Worked example — deleting from a counting bloom filter

Detailed explanation. Build a tiny counting bloom filter, insert two elements that share a bit position, delete one, and confirm the other still tests present — the exact scenario a plain bloom filter cannot handle without breaking.

Structure. m = 8 counters, k = 2 hashes.
Elements. "x" → positions {1, 4}; "y" → positions {4, 6}. They collide at position 4.
Operation. Insert both, then delete "x", then query "y".

Question. After deleting "x", does "y" still test present, and what does the shared counter at position 4 look like?

Input.

Op	Element	Positions	Counter effect
add	x	1, 4	+1 each
add	y	4, 6	+1 each
del	x	1, 4	−1 each
query	y	4, 6	test > 0

Code.

class CountingBloom:
    def __init__(self, m, k, hashfn):
        self.counts = [0] * m
        self.m, self.k, self.hashfn = m, k, hashfn

    def _pos(self, item):
        return self.hashfn(item, self.m, self.k)

    def add(self, item):
        for i in self._pos(item):
            if self.counts[i] < 15:          # saturating 4-bit counter
                self.counts[i] += 1

    def delete(self, item):
        for i in self._pos(item):
            if 0 < self.counts[i] < 15:       # never touch a saturated counter
                self.counts[i] -= 1

    def __contains__(self, item):
        return all(self.counts[i] > 0 for i in self._pos(item))

# positions: x -> {1,4}, y -> {4,6}
cb = CountingBloom(8, 2, lambda it, m, k: {"x": [1, 4], "y": [4, 6]}[it])
cb.add("x"); cb.add("y")
print(cb.counts)          # [0,1,0,0,2,0,1,0]  position 4 shared -> 2
cb.delete("x")
print(cb.counts)          # [0,0,0,0,1,0,1,0]  position 4 still 1
print("y" in cb)          # True  -> y survives x's deletion

Step-by-step explanation.

Inserting "x" increments counters 1 and 4; inserting "y" increments 4 and 6. Position 4 is shared, so its counter reaches 2 — it records that two elements depend on it.
Deleting "x" decrements counters 1 and 4. Counter 1 drops to 0 (no one else needs it), counter 4 drops to 1 (still needed by "y").
Querying "y" tests counters 4 and 6: both are > 0, so "y" still tests present. The deletion of "x" did not create a false negative for "y".
This is exactly what a plain bloom filter cannot do: clearing bit 4 to delete "x" would have wiped a bit "y" needs. The counter records how many elements share the position, so decrement-to-zero happens only when the last dependent is removed.
The saturating guards (< 15 on add, 0 < c < 15 on delete) protect against overflow corruption: once a counter hits its max it is frozen, because a decremented-from-saturation counter could under-count and eventually false-negative.

Output.

Stage	counts array	"y" present?
after add x, y	[0,1,0,0,2,0,1,0]	yes
after delete x	[0,0,0,0,1,0,1,0]	yes
shared counter (pos 4)	2 → 1	still protects y

Rule of thumb. Use a counting bloom filter only when you genuinely need deletes; it costs ~4× the memory of a plain filter. Always saturate counters at their max and never delete an element you did not provably insert — both mistakes silently reintroduce false negatives.

Worked example — bounding error with a scalable bloom filter

Detailed explanation. Show how a scalable bloom filter keeps a bounded compounded error as the element count blows past any single filter's capacity, by chaining filters with a tightening ratio.

Ratio. r = 0.9 (each new filter targets 0.9× the previous filter's error).
Base error. p_0 = 0.01 for the first filter.
Bound. Compounded error ≤ p_0 / (1 − r).

Question. With p_0 = 0.01 and r = 0.9, what is the guaranteed upper bound on the overall false positive rate, and what error does each of the first four filters target?

Input.

Filter i	Target error p_0·r^i
0	0.01
1	0.009
2	0.0081
3	0.00729

Code.

p0, r = 0.01, 0.9

# Per-filter target errors (each new filter is tighter)
targets = [p0 * r**i for i in range(4)]
print([f"{t:.5f}" for t in targets])   # ['0.01000','0.00900','0.00810','0.00729']

# Overall error is bounded by the sum of a geometric series: p0 / (1 - r)
bound = p0 / (1 - r)
print(f"compounded FPR bound = {bound:.3%}")   # 10.000%

# Tighter ratios give tighter overall bounds
for r_try in (0.5, 0.8, 0.9):
    print(f"r={r_try}  ->  bound = {p0/(1-r_try):.2%}")

Step-by-step explanation.

The active filter starts at p_0 = 0.01. When it fills, the next filter targets 0.01 · 0.9 = 0.009, the next 0.0081, and so on — each successive filter is stricter, so it contributes less error.
A query returns "maybe" if any filter says "maybe", so the overall error is at most the sum of the per-filter errors: p_0 · (1 + r + r^2 + …) = p_0 / (1 − r).
With r = 0.9, that bound is 0.01 / 0.1 = 0.10, i.e. the overall error never exceeds 10% no matter how many filters accumulate. The tightening series is what makes the infinite sum converge.
A smaller r gives a tighter bound (r = 0.5 → 2% bound) but forces each new filter to be much stricter and therefore larger; r closer to 1 saves memory but loosens the bound. r = 0.8–0.9 is the common compromise.
The number of filters grows logarithmically with total elements (because each is larger than the last by a growth factor), so query cost stays small — a handful of hash-and-test passes even for billions of elements.

Output.

Ratio r	Compounded FPR bound
0.5	2.00%
0.8	5.00%
0.9	10.00%

Rule of thumb. Use a scalable bloom filter when you cannot predict the final element count. Pick p_0 and r so that p_0/(1−r) is your acceptable ceiling; a tighter r costs more memory per growth step but guarantees a lower overall error.

Data engineering interview question on adding deletes to a dedup filter

A senior interviewer might ask: "Your streaming pipeline uses a plain bloom filter to dedup events over a rolling 24-hour window, but events older than 24 hours must be forgotten so their keys can recur. The plain filter never forgets and its error keeps climbing. Redesign the filter to support expiry, and justify the memory and correctness trade-offs of your choice."

Solution Using time-partitioned counting or rotating bloom filters

# Approach: a ring of per-hour bloom filters (rotating window).
# Each hour gets its own plain bloom; membership = OR across the last 24;
# expiry = drop the oldest filter. No counters needed, no delete corruption.

from collections import deque

class RotatingBloomWindow:
    def __init__(self, hours, make_bloom):
        self.hours = hours
        self.ring = deque(maxlen=hours)      # newest at right
        self.make_bloom = make_bloom
        self.ring.append(make_bloom())

    def rotate(self):                        # call once per hour
        self.ring.append(self.make_bloom())  # maxlen drops the oldest automatically

    def add(self, key):
        self.ring[-1].add(key)               # always write to the current hour

    def __contains__(self, key):
        return any(key in b for b in self.ring)   # seen in ANY of the last N hours

Step-by-step trace.

Hour	Action	Ring contents (oldest→newest)	"k1" seen?
00:00	add k1	[B0{k1}]	yes
01:00	rotate; add k2	[B0{k1}, B1{k2}]	yes
…	… 22 more rotations …	24 filters	yes
24:00	rotate (drops B0)	[B1{k2} … B24{}]	no (expired)

Each hour owns a plain bloom filter; add always targets the newest one, so writes are cheap and never corrupt older filters.
Membership is an OR across all filters in the ring: a key seen in any of the last 24 hours tests "maybe". This is the dedup window.
rotate() runs hourly, appending a fresh empty filter. Because the ring is a deque(maxlen=24), appending the 25th filter automatically evicts the oldest — that is the expiry.
After 24 rotations, the hour-0 filter (holding k1) falls out of the ring, so k1 now tests "definitely not" and its key can recur. Expiry happened by dropping a whole filter, not by deleting individual bits.
No counting counters are needed: because expiry drops an entire independent filter, there is never a shared-bit deletion, so no false negatives are introduced within any single filter's lifetime.

Output:

Design	Deletes?	Memory	False-negative risk
Plain bloom (no expiry)	no	1×	none, but error grows forever
Counting bloom + per-key TTL	yes	~4×	if you delete un-inserted keys
Rotating ring of 24 plain blooms	window expiry	~24× small filters	none

Why this works — concept by concept:

Rotating window — partitioning by time turns "delete old keys" into "drop the oldest filter", which is O(1) and cannot corrupt shared bits. Each filter is independent, so eviction is safe.
OR across the ring — a key is a duplicate if seen in any live hour; the union of the filters is itself a valid bloom-style membership test with a bounded combined error (sum of per-filter errors).
Sized per hour — each hourly filter is sized for one hour of distinct keys, so total memory is 24 × (hourly size), and you never over-provision one giant filter for a full day.
Counting bloom alternative — a single counting bloom with per-key TTL also works but costs ~4× and risks corruption if a key is decremented that was never added; the rotating ring avoids both by never deleting individual keys.
Cost — O(k) per add, O(24·k) per query (24 hash-and-test passes), and O(24 × hourly m) memory. The query fan-out is the price of safe, exact windowed expiry.

DATA STRUCTURES
Topic — data-structures
Counting, scalable, and windowed structure problems

Practice →

HASHING Topic — hash-table Fingerprint and cuckoo-hashing problems

Practice →

4. Bloom filters in databases — LSM and SSTable

The bloom filter that skips a disk read — how LSM engines avoid touching SSTables that cannot hold the key

The mental model in one line: an LSM bloom filter is a per-SSTable membership test that a log-structured-merge storage engine consults before reading a file from disk — if the filter says "definitely not", the engine skips that SSTable's random I/O entirely, and since a point lookup may have to check many SSTables across many levels, this single optimisation is what makes LSM key-value stores like RocksDB, Cassandra, and HBase read-efficient for keys that are absent or rare. This is the highest-leverage real-world use of bloom filters, and interviewers love it because it ties the abstract "definitely not" verdict to a concrete, expensive disk seek.

Why LSM point lookups are expensive.

The LSM shape. An LSM engine buffers writes in an in-memory memtable and periodically flushes them to immutable, sorted files called SSTables. Over time SSTables accumulate across levels (L0, L1, …), each level larger than the last, and compaction merges them.
The read fan-out. A GET key cannot know which SSTable holds the key (or whether any does). Without help, it must binary-search every candidate SSTable from newest to oldest until it finds the key or exhausts them all. Each SSTable check that misses is a wasted disk read — the essence of read amplification.
The worst case — absent keys. A lookup for a key that does not exist must, in the naive design, read every level's candidate SSTable to prove absence. Absent-key lookups are common (existence checks, upserts, cache-miss confirmations), so this is not a rare path.

How the bloom filter fixes it.

One filter per SSTable. When an SSTable is written, the engine builds a bloom filter over all the keys in that file and stores it (usually in a metadata block, cached in memory).
Check before you read. On GET key, for each candidate SSTable the engine first tests the key against that SSTable's bloom filter. "Definitely not" → skip the file, no disk read. "Maybe" → read the file (and possibly find the key, or discover the false positive).
The payoff. For an absent key, most SSTables answer "definitely not" and are skipped; only the ~p fraction that false-positive incur a disk read. Read amplification for absent keys drops from "read every level" to "read almost nothing".

RocksDB knobs every data engineer should know.

bits_per_key. The bloom filter's bits per element, set via the block-based table's filter policy (e.g. NewBloomFilterPolicy(10)). 10 bits ≈ 1% false positive rate is the common default; raise it to cut read amplification further at the cost of memory.
Whole-key vs prefix. whole_key_filtering builds the filter over full keys for point lookups. A prefix bloom (via a prefix_extractor) builds it over key prefixes so range scans with a common prefix can also be pruned.
Full vs partitioned filters. A full filter is one bloom per SSTable, loaded whole. A partitioned filter splits the bloom into blocks with a top-level index, so only the relevant block is paged into memory — important when the total filter set is too large to pin in RAM. Newer engines also offer a Ribbon filter, a more space-efficient successor to the classic bloom at the same error rate.

Beyond LSM — bloom filters across the data stack.

Join pruning / runtime filters. In Spark, Impala, and cloud warehouses, the engine can build a bloom filter over the join keys of the smaller (build) side and broadcast it to pre-filter the larger (probe) side, discarding rows that cannot match before the shuffle. This is the "bloom filter join" or "runtime filter" optimisation.
Partition and file pruning. Query engines keep bloom filters (or the related min/max and dictionary stats) per data file (Parquet/ORC) so a predicate can skip whole files that cannot contain a value.
Deduplication and one-hit-wonder caches. CDNs use a bloom filter to avoid caching objects requested only once ("admit to cache only on the second request"), and ingestion pipelines use them to drop already-seen records cheaply.

Worked example — read amplification with and without a bloom filter

Detailed explanation. Quantify the disk reads an absent-key lookup costs in an LSM tree with and without per-SSTable bloom filters, to see why the optimisation is not optional at scale.

Tree shape. 7 candidate SSTables must be checked for a point lookup.
Key status. The key is absent (the common existence-check path).
Filter error. 1% per SSTable bloom.

Question. For 1,000,000 lookups of absent keys against 7 SSTables, how many disk reads happen with and without bloom filters?

Input.

Parameter	Value
SSTables checked per lookup	7
Lookups (all absent keys)	1,000,000
Bloom FPR per SSTable	1%
Disk read cost	~1 random seek each

Code.

sstables_per_lookup = 7
lookups             = 1_000_000
fpr                 = 0.01

# Without bloom: every candidate SSTable is read to prove absence
reads_without = lookups * sstables_per_lookup

# With bloom: only false-positive SSTables are read (expected fpr each)
reads_with = lookups * sstables_per_lookup * fpr

print(f"disk reads without bloom : {reads_without:,}")   # 7,000,000
print(f"disk reads with bloom    : {int(reads_with):,}") #    70,000
print(f"read amplification cut    : {1 - reads_with/reads_without:.1%}")  # 99.0%

Step-by-step explanation.

Without bloom filters, proving an absent key is truly absent requires reading all 7 candidate SSTables — 7 million disk seeks for a million lookups.
With a 1% bloom per SSTable, each SSTable answers "definitely not" 99% of the time and is skipped. Only the 1% false-positive cases trigger a real read.
The expected reads are 1,000,000 × 7 × 0.01 = 70,000 — a 100× reduction, from 7 million seeks to 70 thousand.
The false-positive reads are not errors in the result (the SSTable is read, the key is confirmed absent, the lookup returns "not found" correctly) — they are just wasted I/O, and their volume is exactly the false-positive rate.
This is why every production LSM engine ships bloom filters on by default: absent-key and rare-key lookups dominate many workloads, and the disk-seek savings are enormous.

Output.

Metric	Without bloom	With 1% bloom
Disk reads (1M absent lookups)	7,000,000	70,000
Reduction	—	99.0%
Extra CPU	none	k hashes per SSTable check
Memory for filters	none	~10 bits/key per SSTable

Rule of thumb. For LSM point-lookup workloads with many absent or rare keys, per-SSTable bloom filters cut disk reads by roughly (1 − fpr) per skipped file. The memory (~10 bits/key at 1%) is almost always worth it; only disable blooms for scan-only workloads where point lookups never happen.

Worked example — configuring and tightening a RocksDB bloom filter

Detailed explanation. Show the RocksDB configuration surface for bloom filters and how raising bits_per_key trades memory for fewer false-positive disk reads.

Filter policy. NewBloomFilterPolicy(bits_per_key).
Placement. In the block-based table options.
Tightening. Move from 10 to 16 bits/key to cut the false-positive read rate.

Question. How does raising RocksDB's bits_per_key from 10 to 16 change the false-positive rate and the filter memory for a 100-million-key column family?

Input.

Parameter	10 bits/key	16 bits/key
Keys	100,000,000	100,000,000
Approx FPR	~1%	~0.05%
Filter memory	~125 MB	~200 MB

Code.

// RocksDB (C++) — attach a bloom filter policy to the table format
#include "rocksdb/table.h"
#include "rocksdb/filter_policy.h"

rocksdb::BlockBasedTableOptions table_opts;
table_opts.filter_policy.reset(
    rocksdb::NewBloomFilterPolicy(16 /* bits_per_key */));
table_opts.whole_key_filtering   = true;   // point-lookup filter over full keys
table_opts.partition_filters     = true;   // page filter blocks on demand
table_opts.cache_index_and_filter_blocks = true;

rocksdb::Options options;
options.table_factory.reset(
    rocksdb::NewBlockBasedTableFactory(table_opts));

import math
# The FPR that bits_per_key implies, at the engine's optimal k
def fpr_for_bits(bits_per_key: float) -> float:
    k = bits_per_key * math.log(2)          # optimal k
    return 0.5 ** k                         # ~ 0.5^k at optimal k

for b in (10, 16):
    print(f"bits_per_key={b}  ->  FPR ≈ {fpr_for_bits(b):.3%}")
# bits_per_key=10  ->  FPR ≈ 0.819%
# bits_per_key=16  ->  FPR ≈ 0.216%

Step-by-step explanation.

NewBloomFilterPolicy(16) tells RocksDB to build each SSTable's bloom with 16 bits per key instead of the default 10. RocksDB internally chooses the optimal number of hash probes.
At 10 bits/key the implied false-positive rate is roughly 0.8–1%; at 16 bits/key it drops to about 0.2% — a ~4× reduction in wasted disk reads on absent keys.
The cost is memory: filter size scales linearly with bits_per_key, so 100M keys go from ~125 MB to ~200 MB of filter data (which you want cached in RAM to avoid reading the filter from disk).
partition_filters = true splits the filter so only the needed block is loaded, keeping RAM pressure manageable when the total filter set is large; cache_index_and_filter_blocks pins them in the block cache.
whole_key_filtering = true builds the filter for full-key point lookups; if your workload is prefix-scan heavy you would instead configure a prefix_extractor and a prefix bloom.

Output.

bits_per_key	Approx FPR	Filter memory (100M keys)	Extra absent-read reduction
10 (default)	~0.8%	~125 MB	baseline
16	~0.2%	~200 MB	~4× fewer FP reads

Rule of thumb. Tune RocksDB bits_per_key upward only if bloom false positives (visible as bloom_useful / bloom_full_positive stats) are causing measurable read I/O; otherwise the default 10 (~1%) is the right memory/read balance. Always cache index and filter blocks so the bloom check itself never hits disk.

Systems interview question on cutting read amplification

A senior interviewer might ask: "Your RocksDB-backed key-value service has high read latency dominated by disk seeks, and profiling shows most GETs are for keys that do not exist (an existence-check workload). Walk me through why this is slow, how bloom filters fix it, how you would configure them, and how you would verify the fix from the engine's statistics."

Solution Using per-SSTable bloom filters with verified statistics

Diagnosis
---------
  existence-check GET for an ABSENT key must, without blooms, read one
  candidate SSTable per level to prove absence  ->  N random disk seeks
  per lookup, where N = number of levels/SSTables checked.

Fix
---
  1. Enable a bloom filter policy per SSTable (bits_per_key = 10, ~1% FPR).
  2. Cache index + filter blocks so the bloom check never reads disk.
  3. On GET: test the key against each SSTable's bloom FIRST.
        "definitely not" -> skip the file (0 disk seeks)
        "maybe"          -> read the file (found, or a false positive)

Verify from RocksDB statistics
------------------------------
  bloom_filter_useful          -> count of SSTable reads AVOIDED by a bloom
  bloom_filter_full_positive   -> "maybe" verdicts that led to a read
  false-positive ratio ≈ full_true_positive vs full_positive
  read latency p99             -> should drop sharply for absent keys

Step-by-step trace.

GET (absent key)	SSTables	Blooms say	Disk reads
before fix	6 checked	n/a	6
after fix (typical)	6 checked	6× "definitely not"	0
after fix (false positive)	6 checked	5× not, 1× "maybe"	1

Before the fix, an absent-key GET reads all 6 candidate SSTables to prove the key is nowhere — 6 disk seeks per lookup, dominating latency.
After enabling bloom filters, each SSTable is bloom-tested first. For a truly absent key, all 6 blooms usually say "definitely not", so zero disk reads happen.
The occasional false positive (≈1% per SSTable) causes a single wasted read, confirmed absent in that file; the lookup result is still correct.
bloom_filter_useful counts every avoided SSTable read — it should climb to roughly 6 × lookups × (1 − fpr), quantifying the benefit directly from the engine.
p99 read latency for the existence-check workload collapses because the common path went from 6 seeks to 0; only the rare false-positive path pays for I/O.

Output:

Metric	Before blooms	After blooms
Disk seeks per absent GET	~6	~0 (≈0.06 with 1% FP over 6 files)
Dominant cost	random disk I/O	in-memory bloom checks
p99 read latency	high	sharply lower
Verification signal	—	bloom_filter_useful rising

Why this works — concept by concept:

Per-SSTable bloom — each file carries a membership test over its own keys, so a "definitely not" is a certificate that the key is not in that file — no need to read it. Absence proofs become memory operations, not disk seeks.
Cached filter blocks — pinning index and filter blocks in the block cache ensures the bloom check itself never causes I/O; otherwise you would trade a data seek for a filter seek.
False positives cost I/O, not correctness — a "maybe" that turns out wrong causes one wasted read but never a wrong answer, because the SSTable read authoritatively confirms presence or absence.
Statistics close the loop — bloom_filter_useful and the full-positive counters let you prove the optimisation is working and tune bits_per_key against real false-positive read volume rather than guessing.
Cost — O(levels × k) in-memory hash checks per GET plus ~10 bits/key of cached filter memory, in exchange for eliminating O(levels) random disk seeks per absent lookup. On disk-bound workloads that is a 10–100× latency win.

DATA STRUCTURES
Topic — data-structures
Storage-engine and index structure problems

Practice →

OPTIMIZATION Topic — optimization Read-amplification and I/O reduction problems

Practice →

5. Building and tuning one in Python

A production-shaped BloomFilter class, sizing helpers, and measuring observed versus predicted error

The mental model in one line: a production-quality bloom filter in Python is about sixty lines — a bit array backed by a bytearray, hash functions reduced to one MurmurHash split by double hashing, and two sizing helpers that turn (n, target p) into (m, k) — and the discipline that separates a toy from a tool is measuring the observed false positive rate against the predicted one after you build it, because a mis-sized filter or a bad hash silently drifts off target. Everything in the earlier sections comes together here as runnable code you can drop into a dedup or pre-filter stage.

The building blocks.

The bit array. A bytearray(ceil(m/8)) gives m addressable bits; set bit i with arr[i >> 3] |= (1 << (i & 7)) and test it with arr[i >> 3] & (1 << (i & 7)). For serious workloads the bitarray library is faster, but bytearray has zero dependencies.
The hashing. One mmh3.hash128 call split into two 64-bit lanes, combined by double hashing to yield k positions. Reuse the same digest for insert and query so both cost one hash call.
The parameters. Store m, k, and the design n on the object so you can compute expected error, detect saturation, and serialise the filter with its shape.

The sizing helpers.

optimal_m(n, p). ceil(−n·ln p / (ln 2)^2) — bits for n elements at target error p.
optimal_k(m, n). max(1, round((m/n)·ln 2)) — the error-minimising hash count.
Design for the max. Always pass the maximum expected n; a filter sized for too few elements exceeds its target error as it fills.

Tuning and verification.

Predict then measure. Compute the expected error from (m, n, k), then insert n known elements and query a large batch of known non-members, counting how many come back "maybe". Observed and predicted should agree within noise.
Watch saturation. Track the fraction of set bits; at the design point it should be ~50%. Much higher means you have overrun n and the error is above target.
Resize by rebuild. A plain bloom cannot grow in place. When load approaches n, allocate a larger filter and re-insert (from source data, or from a scalable-filter chain). Union/intersection are only valid between filters with identical m and k.

Operational notes.

Serialisation. Persist m, k, n, and the raw bytearray bytes. On load, reconstruct with the same parameters — a filter is meaningless without its shape.
Union of filters. Two filters with the same m and k can be OR-ed bit-for-bit to get a filter of the union of their sets — useful for merging per-partition filters in a distributed job.
Thread safety. Inserts are bit-OR, which is idempotent and order-independent, but concurrent writers still need synchronisation on the underlying bytes in CPython for correctness of the read-modify-write on each byte.

Worked example — a minimal but correct BloomFilter class

Detailed explanation. Assemble the full class: constructor sizes m and k from (n, p), add sets bits via double hashing, the in membership test checks them, and a helper reports the current set-bit fraction for saturation monitoring.

Inputs. capacity (design n) and error_rate (target p).
Backing store. bytearray.
Hashing. mmh3.hash128 split into two lanes.

Question. Implement a dependency-light BloomFilter with add, membership test, and a fill_ratio diagnostic.

Input.

Method	Behaviour
constructor `(capacity, error_rate)`	size m, k; allocate bytearray
`add(item)`	set k bits
`item in bf` membership test	test k bits (all set → maybe)
`fill_ratio()`	fraction of bits set (saturation check)

Code.

import math
import mmh3

class BloomFilter:
    def __init__(self, capacity: int, error_rate: float = 0.01):
        self.n = capacity
        self.p = error_rate
        self.m = max(8, math.ceil(-(capacity * math.log(error_rate)) / (math.log(2) ** 2)))
        self.k = max(1, round((self.m / capacity) * math.log(2)))
        self.bits = bytearray((self.m + 7) // 8)     # ceil(m/8) bytes
        self.count = 0                               # elements added

    def _indexes(self, item: str):
        item = item if isinstance(item, (bytes, str)) else str(item)
        d = mmh3.hash128(item, signed=False)
        h1, h2 = d & 0xFFFFFFFFFFFFFFFF, d >> 64
        if h2 == 0:
            h2 = 1
        for i in range(self.k):
            yield (h1 + i * h2) % self.m

    def add(self, item) -> None:
        for idx in self._indexes(item):
            self.bits[idx >> 3] |= (1 << (idx & 7))
        self.count += 1

    def __contains__(self, item) -> bool:
        return all(self.bits[idx >> 3] & (1 << (idx & 7)) for idx in self._indexes(item))

    def fill_ratio(self) -> float:
        set_bits = sum(bin(b).count("1") for b in self.bits)
        return set_bits / self.m

bf = BloomFilter(capacity=1_000_000, error_rate=0.01)
print(f"m={bf.m:,} bits ({bf.m//8//1000} KB), k={bf.k}")
bf.add("user:42")
print("user:42" in bf, "user:99" in bf)      # True  False (almost surely)

Step-by-step explanation.

The constructor sizes m from (capacity, error_rate) and k from (m, capacity), then allocates ceil(m/8) bytes — for 1M keys at 1% that is ~1.2 MB and k = 7.
_indexes computes one 128-bit MurmurHash, splits it into two 64-bit lanes, and yields k positions via double hashing — one hash call per operation.
add ORs each of the k bits on and bumps count, which lets fill_ratio and saturation checks reason about how full the filter is.
The in membership test returns True only if all k bits are set; the all(...) generator short-circuits at the first 0, so a "definitely not" is cheap.
fill_ratio popcounts the whole array; at the design load it should read ~0.5 (the optimal-k sweet spot). A reading well above 0.5 is a saturation alarm meaning count has exceeded the design n.

Output.

Call	Result
`m`, `k` for (1M, 1%)	9,585,059 bits (~1.2 MB), k=7
`"user:42" in bf` (added)	True
`"user:99" in bf` (absent)	False (≈99% of the time)
`fill_ratio()` after 1 add	~7 / m ≈ 0

Rule of thumb. Keep m, k, and count on the object so the filter can self-report saturation. Back it with bytearray for zero dependencies or bitarray for speed, and always hash once per operation via double hashing — never call the hash function k times.

Worked example — measuring observed versus predicted false positive rate

Detailed explanation. A sized filter is a claim; the empirical test is the proof. Insert n known members, then probe a large disjoint set of known non-members and count "maybe" verdicts — the observed rate should match the formula.

Design. n = 100,000 at p = 1%.
Probe. 200,000 known non-members.
Compare. Observed FP fraction versus (1 − e^{−kn/m})^k.

Question. Build the filter, insert 100k members, probe 200k non-members, and compare the observed false-positive rate to the predicted one.

Input.

Parameter	Value
Members inserted	100,000
Non-members probed	200,000
Target p	1%
Prediction	(1 − e^{−kn/m})^k

Code.

import math

bf = BloomFilter(capacity=100_000, error_rate=0.01)

# Insert 100k known members
for i in range(100_000):
    bf.add(f"member:{i}")

# Probe 200k known NON-members (disjoint namespace)
false_positives = sum(1 for i in range(200_000) if f"absent:{i}" in bf)
observed = false_positives / 200_000

# Predicted FPR from the actual m, k, n
predicted = (1 - math.exp(-bf.k * bf.n / bf.m)) ** bf.k

print(f"m={bf.m:,}  k={bf.k}  fill={bf.fill_ratio():.3f}")
print(f"observed FPR  = {observed:.3%}")
print(f"predicted FPR = {predicted:.3%}")
# typical:
# m=958,506  k=7  fill=0.500
# observed FPR  = 1.02%
# predicted FPR = 1.00%

Step-by-step explanation.

The filter is sized for 100k at 1%, giving m ≈ 958,506 bits and k = 7.
Inserting exactly the design count of members brings the fill ratio to ~0.5 — the optimal-k half-full state the formula assumes.
Probing 200k non-members in a disjoint namespace (absent:* vs member:*) counts genuine false positives, since none of those keys were inserted.
The observed rate (~1.02%) matches the predicted rate (1.00%) within sampling noise — confirmation that both the sizing and the double hashing are correct.
If observed sharply exceeded predicted, the usual culprits are a weak or mis-split hash, an off-by-one in the bit indexing, or having inserted more than the design n (check fill_ratio — well above 0.5 means saturation).

Output.

Quantity	Value
Fill ratio after 100k inserts	~0.500
Observed FPR (200k probes)	~1.02%
Predicted FPR	~1.00%
Agreement	within noise → filter is healthy

Rule of thumb. Always measure the observed false-positive rate against the predicted one on a disjoint non-member probe set before trusting a bloom filter in production. A fill ratio near 0.5 at the design count and observed ≈ predicted are your two green lights; a fill ratio far above 0.5 means you have overrun capacity.

Python interview question on building and tuning a dedup filter

A senior interviewer might ask: "Build a bloom filter for a streaming pipeline that must dedup roughly 5 million distinct records per hour at a 0.5% false positive rate, using constant memory. Show the sizing, the class, how you would detect when the filter has been overrun, and what you would do when it approaches capacity."

Solution Using a sized BloomFilter with saturation detection and rebuild

import math

class DedupFilter:
    def __init__(self, capacity: int, error_rate: float):
        self.bf = BloomFilter(capacity, error_rate)
        self.capacity = capacity

    def seen(self, key: str) -> bool:
        """Return True if key is a probable duplicate; else record it and return False."""
        if key in self.bf:
            return True                       # "maybe" -> treat as duplicate
        self.bf.add(key)
        return False

    def is_overrun(self) -> bool:
        # Two signals: element count past design n, or fill ratio past ~0.5
        return self.bf.count > self.capacity or self.bf.fill_ratio() > 0.55

# Size for 5M/hour at 0.5%
dedup = DedupFilter(capacity=5_000_000, error_rate=0.005)
print(f"m={dedup.bf.m:,} bits ({dedup.bf.m//8//1_000_000} MB), k={dedup.bf.k}")

for rec_id in stream_of_records():            # pseudo-source
    if not dedup.seen(rec_id):
        emit(rec_id)
    if dedup.is_overrun():
        dedup = DedupFilter(5_000_000, 0.005)  # rotate to a fresh hourly filter

Step-by-step trace.

Step	Value	Reasoning
bits per element	−ln(0.005)/(ln 2)^2 ≈ 11.03	0.5% target
m	5,000,000 × 11.03	~55.2M bits ≈ 6.9 MB
k	round(11.03 × ln 2)	8
overrun signal 1	count > 5,000,000	past design capacity
overrun signal 2	fill_ratio > 0.55	array too saturated
action on overrun	allocate fresh filter	constant memory, reset error

Bits per element for 0.5% is −ln(0.005)/(ln 2)^2 ≈ 11.03, so a 5M-key filter needs ~55.2M bits ≈ 6.9 MB — constant, small, predictable.
k = round(11.03 · ln 2) = 8 hash probes minimise the error at that size.
seen(key) implements dedup with the one-sided contract: a "maybe" is treated as a duplicate (dropped), a "definitely not" is a first sight (emitted and recorded).
is_overrun() watches two independent signals — the element count exceeding the design n, and the fill ratio climbing past ~0.55 — either of which means the error has drifted above target.
On overrun the pipeline rotates to a fresh filter (the hourly boundary), keeping memory constant and resetting the error to target; a scalable bloom filter is the alternative if the window cannot be rotated.

Output:

Metric	Value
Filter memory	~6.9 MB (constant)
k (hash probes)	8
Duplicate handling	"maybe" → drop; "definitely not" → emit + record
Overrun detection	count > n OR fill_ratio > 0.55
Recovery	rotate to a fresh sized filter

Why this works — concept by concept:

Sized for the hourly count — sizing m for the maximum 5M distinct keys holds the error at 0.5% for the whole hour; sizing for fewer would let the error climb as the filter fills.
One-sided dedup — seen maps "maybe" → duplicate and "definitely not" → new, so the only error is occasionally dropping a genuinely new record, never emitting a duplicate. Choose this only where a small drop rate is acceptable, or add an exact-confirm on "maybe".
Dual overrun signals — element count catches known overrun; fill ratio catches it empirically even if the count estimate is wrong. Together they detect capacity breach before the error blows past target.
Rotate for constant memory — allocating a fresh filter at the window boundary keeps memory flat and resets the error, which is exactly the rotating-window pattern from section 3 applied to a single stream.
Cost — O(k) = 8 hash-derived bit ops per record and a fixed ~6.9 MB of memory regardless of key size or stream length. The exact alternative (a growing hash set of 5M keys) would cost far more memory and grow unbounded across hours.

PYTHON
Topic — optimization
Space-efficient sizing and tuning problems

Practice →

PYTHON
Topic — data-structures
Implement-a-data-structure problems in Python

Practice →

Cheat sheet — bloom filter recipes

The membership contract. A bloom filter answers only "definitely not" (certain) or "maybe" (probabilistic). No false negatives, possible false positives. Never describe it as a fast hash set — name the one-sided error first. A "definitely not" is a certificate of absence; a "maybe" is a member or a false positive.
The two parts. A bit array of m bits plus k hash functions. Insert sets k bits (bitwise OR); query tests k bits (all-1s → "maybe", any-0 → "definitely not"). Memory is independent of element size — a filter over 1 KB log lines costs the same bits per element as one over 8-byte ints.
The false-positive-rate formula. p ≈ (1 − e^{−kn/m})^k. It rises monotonically with n, so every bloom filter has a design capacity; overrun it and the error climbs past target.
Optimal k. k = (m/n)·ln 2 ≈ 0.693·(m/n), rounded to an integer. At the optimum the array is ~50% full and p ≈ 0.5^k. Fewer hashes under-fill, more hashes over-saturate.
Sizing for a target p. m = −(n·ln p)/(ln 2)^2, i.e. m/n ≈ −1.4427·log2(p) bits per element. Memorise: 1% → ~9.6 bits, k=7; 0.1% → ~14.4 bits, k=10; 0.01% → ~19.2 bits, k=13. Each extra 10× of accuracy costs a flat ~4.8 bits per element.
Double hashing. Never compute k independent hashes. Take one 128-bit MurmurHash, split into h1 (low 64) and h2 (high 64), derive index i as (h1 + i·h2) mod m for i = 0…k−1 (Kirsch–Mitzenmacher). Guard h2 ≠ 0.
Deletion → counting bloom. Replace bits with small (usually 4-bit) counters; add increments, delete decrements, query tests all-nonzero. Costs ~4× memory. Saturate counters at max and freeze them; never delete a key you did not provably insert (risks false negatives).
Unknown growth → scalable bloom. Chain filters with tightening error p_0·r^i; overall error is bounded by p_0/(1−r). Pick r (0.8–0.9 common) so the bound is your acceptable ceiling; number of filters grows logarithmically with element count.
Churn / windowing. For a rolling window, rotate a ring of per-slice plain filters and drop the oldest to expire keys — O(1) expiry, no shared-bit deletion, no false negatives. Query is an OR across the ring.
Modern alternative → cuckoo filter. Stores fingerprints in a cuckoo hash table; supports deletes natively, better lookup locality (two buckets), and is more space-efficient than bloom below ~3% error. Prefer it for churny, latency-sensitive sets; bloom stays simplest for append-only.
LSM / SSTable usage. One bloom per SSTable; a "definitely not" skips the file's random disk read, cutting read amplification for absent/rare keys by ~(1 − fpr) per skipped file. RocksDB: NewBloomFilterPolicy(bits_per_key) (default ~10 → ~1%), whole_key_filtering for point lookups, prefix_extractor for prefix scans, partition_filters + cached filter blocks so the check never hits disk.
Join & file pruning. Warehouses and Spark build a bloom over the small side's join keys and broadcast it to prune the large side before shuffle (runtime/bloom-filter join); per-file blooms in Parquet/ORC skip files that cannot contain a predicate value.
Verify before trusting. After building, probe a large disjoint set of known non-members and compare observed FP fraction to (1 − e^{−kn/m})^k. A fill ratio ~0.5 at the design count and observed ≈ predicted are your two green lights; fill far above 0.5 means capacity overrun.

Frequently asked questions

What is a bloom filter in one sentence?

A bloom filter is a space-efficient probabilistic data structure that stores set membership in a bit array using k hash functions, answering a membership test with either "definitely not in the set" (always correct) or "possibly in the set" (correct except for a small, tunable false positive rate). It never produces false negatives, so a "definitely not" can be trusted to skip an expensive lookup, disk read, or network call, and it stores bits about elements rather than the elements themselves — making its memory independent of how large each element is. That combination of constant-time queries and roughly 9.6 bits per element at 1% error is why bloom filters appear everywhere from LSM databases to CDN caches to distributed joins.

Why can't a bloom filter have false negatives?

Because inserting an element only ever turns bits on, and a plain bloom filter never turns bits off. Every bit that a genuine member set during insertion is therefore still 1 at query time, so a real member can never produce a 0 bit, and a query only reports "definitely not" when it finds a 0. The one-sided error is a direct consequence of the append-only bit array: the moment you introduce deletion (clearing bits), you risk clearing a bit another element depends on and creating a false negative — which is exactly why deletion requires the counting bloom variant with per-position counters instead of single bits.

How many bits per element do I need?

The bits-per-element cost is −ln(p)/(ln 2)^2 ≈ −1.4427·log2(p), which depends only on the target false positive rate p, not on the element size. The numbers worth memorising are 1% → ~9.6 bits, 0.1% → ~14.4 bits, 0.01% → ~19.2 bits, with each additional 10× of accuracy adding a flat ~4.8 bits per element. So a filter for 100 million keys at 0.1% needs about 180 MB regardless of whether the keys are 8-byte integers or 400-byte URLs. Once you fix m, set the hash count to the optimum k = round((m/n)·ln 2).

Can you delete from a bloom filter?

Not from a plain bloom filter — its bits are shared across elements, so clearing one element's bits could break another element's membership and create a false negative. To support deletion you use a counting bloom filter, which replaces each bit with a small counter (commonly 4 bits): insert increments the k counters, delete decrements them, and query tests that all k are non-zero. This costs roughly 4× the memory of a plain filter and requires that you only ever delete elements you actually inserted. For churny or latency-sensitive sets, a cuckoo filter (which stores fingerprints and supports exact deletion with better locality) or a rotating ring of plain filters for window expiry are often better choices than a counting bloom.

Where are bloom filters used in databases?

The flagship use is in log-structured-merge (LSM) storage engines like RocksDB, LevelDB, Cassandra, HBase, and ScyllaDB, where each immutable SSTable carries a LSM bloom filter over its keys. On a point lookup, the engine tests the key against each SSTable's bloom before reading the file from disk; a "definitely not" lets it skip that file's random I/O entirely, which slashes read amplification for absent and rare keys. Bloom filters also power join pruning (broadcasting a bloom of the small side's join keys to pre-filter the large side in Spark and cloud warehouses), per-file pruning in Parquet/ORC, and one-hit-wonder cache admission in CDNs. In every case the "definitely not" verdict is what converts an expensive operation into a skipped one.

Bloom filter vs hash set vs cuckoo filter — when do I use each?

Use an exact hash set when correctness is absolute and the set fits in memory where you need it — it has zero false positives but stores every element, so it is the largest of the three. Use a bloom filter when the set is too large for an exact structure, a "definitely not" saves real work, and a small false-positive rate of wasted work is acceptable; it is the simplest and most compact option for append-only sets. Use a cuckoo filter when you need the space savings of a probabilistic structure and deletion or better lookup locality — it stores fingerprints, supports exact deletes, touches only two cache lines per query, and beats bloom on space below about 3% error, at the cost of possible insert failures near full load. In short: exactness → hash set; append-only compactness → bloom; deletes and locality → cuckoo.

Practice on PipeCode

Drill the data-structures practice library → for the probabilistic-structure, set-membership, and storage-engine index problems senior interviewers reach for.
Rehearse on the hash-table practice library → for the hashing, collision, fingerprinting, and double-hashing fundamentals that bloom filters are built on.
Sharpen the sizing math on the optimization practice library → for the space–time trade-off, read-amplification, and false-positive-rate tuning scenarios.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the bloom-filter sizing formula against real graded inputs.

Lock in bloom filter muscle memory

Docs explain the bit array. PipeCode drills explain the decision — when a "definitely not" is worth a whole disk read, when the false-positive rate blows past target because you overran capacity, when a counting bloom's 4× memory is justified, and when a cuckoo filter is the better call. Pipecode.ai is Leetcode for Data Engineering — structure-first practice tuned for the production trade-offs senior data engineers actually face.

Practice data-structure problems →
Practice hashing problems →

Retries, Timeouts & Circuit Breakers for Data Pipelines

Gowtham Potureddi — Sun, 06 Sep 2026 11:37:12 +0000

retries timeouts and circuit breakers are the three controls that decide whether a data pipeline degrades gracefully or falls over the first time a dependency hiccups — and they are the single set of patterns senior data engineers are expected to reach for by name the moment an interviewer says "the upstream API started returning 503s at 2 a.m." Every non-trivial pipeline is a chain of remote calls: an HTTP extractor pulling from a partner API, a warehouse COPY staging a batch to Snowflake, a streaming sink flushing to Kafka, an enrichment step calling a geocoding service. Each hop can fail transiently, hang forever, or brown out under load, and the difference between a pipeline that self-heals and one that pages you is entirely in how you handle those failures — whether you retry blindly and cause a retry storm, whether you call without a timeout and hang a worker, whether you keep hammering a dead dependency instead of failing fast.

This guide is the walkthrough you wished existed the first time you had to defend a resilience design on a whiteboard. It builds the failure model in layers: the failure taxonomy that decides whether a call is even safe to retry, exponential backoff with jitter to spread the retries out and stop the thundering herd, the timeout budget that carries one deadline down a multi-hop call chain, the circuit breaker state machine and the bulkhead that isolates pools so one slow dependency can't sink the whole worker, and finally the correct order to nest all four primitives around a single idempotency-safe call. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works. All examples are Python against typical data-pipeline dependencies, but the mental model carries over to any language and any orchestration engine.

When you want hands-on reps immediately after reading, drill the defensive-coding practice library →, rehearse failure paths on the exception-handling practice library →, and wire the patterns into real jobs on the ETL practice library →.

On this page

The pipeline failure taxonomy
Retries with backoff and jitter
Timeouts and deadline budgets
Circuit breakers and bulkheads
Composing resilience in a pipeline
Cheat sheet — resilience recipes
Frequently asked questions
Practice on PipeCode

1. The pipeline failure taxonomy

Classify the failure before you react — retryable, fatal, or poison decides everything downstream

The one-sentence invariant: resilience engineering starts by classifying every failure into transient (retry with backoff), persistent (fail fast and alert), or poison (quarantine the input), because the wrong reaction to a failure class — retrying a fatal error, aborting on a transient blip, or replaying a poison record forever — is more damaging than the original failure. Data engineers get this wrong most often by treating "the call failed" as a single event when it is really a family of events with wildly different correct responses. A 503 from an overloaded API wants a backed-off retry; a 401 from an expired credential wants an immediate abort and a page; a malformed record that crashes your parser wants to be shunted to a dead-letter queue so the other 999,999 records in the batch still land.

The three failure classes that decide your reaction.

Transient. The call would likely succeed if you tried again in a moment: HTTP 429/503, connection reset, DNS blip, a brief network partition, a warehouse "too many concurrent queries" throttle. These are the only failures you should retry. The correct reaction is backoff + jitter, bounded by a retry budget and a deadline.
Persistent. The call will fail identically no matter how many times you retry: HTTP 401/403 (bad credentials), 400 (malformed request), 404 (wrong URL), a schema mismatch, a NOT NULL violation on a required column. Retrying these wastes time and hides the real problem. The correct reaction is fail fast, surface the error, and alert.
Poison. The input is the problem, not the dependency: a single record whose payload crashes the transform, an un-decodable byte sequence, a value that violates a downstream constraint. Retrying reprocesses the same bad record forever (a "poison pill" that blocks the queue). The correct reaction is to quarantine the record to a dead-letter store and continue with the rest of the batch.

The four resilience primitives — one job each.

Retries convert a transient failure into a success by trying again, spaced out by backoff and jitter. They do nothing for persistent or poison failures.
Timeouts bound how long any single call may take, converting a hang (the worst failure mode — it consumes a worker indefinitely) into a fast, retryable failure.
Circuit breakers stop calling a dependency that is persistently failing, converting a slow cascade of timeouts into an instant fail-fast, and giving the dependency room to recover.
Bulkheads isolate resources (connection pools, thread pools, worker slots) per dependency so that one saturated dependency cannot consume all the capacity and starve the healthy ones.

The 2026 reality — partial failure is the default, not the exception.

Distributed pipelines fail partially. A pipeline touching five services has, at any instant, a non-trivial probability that at least one is degraded. Designing for "everything is up" is designing for a state that rarely holds.
Blind retries amplify outages. The single most common self-inflicted incident is the retry storm: a dependency slows down, every client retries immediately, the added load pushes the dependency fully over, and the retries now guarantee it can never recover. Retries without backoff, jitter, and a budget are a loaded gun pointed at your own infrastructure.
Idempotency is the precondition. You may only safely retry a call whose repetition has no extra effect — a GET, an upsert keyed by a natural key, a COPY into an idempotent staging table. Retrying a non-idempotent POST that charges a card or appends a duplicate row is a correctness bug wearing a resilience costume.

What interviewers listen for.

Do you classify the failure before choosing a reaction? — senior signal. Weak candidates say "add retries"; strong candidates say "is it transient, persistent, or poison?"
Do you name idempotency as the precondition for retries without being prompted? — required answer.
Do you describe a retry storm and how backoff + jitter + a budget prevent it? — senior signal.
Do you distinguish a timeout (bounds one call) from a circuit breaker (bounds calling a dead dependency) — not conflate them? — required answer.
Do you send poison records to a dead-letter queue rather than failing the whole batch or retrying forever? — senior signal.

Worked example — classify a batch of pipeline failures

Detailed explanation. The most useful artifact for a resilience interview is a failure-classification table you can build on the spot. Every senior discussion converges on "which of these do we retry?" within the first few minutes; having a crisp mapping in your head is what separates a fluent answer from a hand-wave. Walk through classifying the failures a typical ingestion task actually sees in a week.

The task. An hourly extractor pulls JSON from a partner REST API, transforms each record, and upserts into a Postgres staging table.
The failures observed. A mixed bag: throttling, expired tokens, one un-parseable record, a network reset, a NOT NULL violation.
The goal. Assign each failure a class and a reaction, so the retry logic never fires on the wrong thing.

Question. Classify each observed failure and state the correct reaction for the extractor.

Input.

Observed failure	HTTP / error	Class	Correct reaction
API returned "rate limit exceeded"	429	transient	retry with backoff
Auth token expired overnight	401	persistent	refresh token or fail + alert
One record has a non-UTF-8 byte	ValueError in parse	poison	dead-letter the record
Connection reset mid-response	ConnectionResetError	transient	retry with backoff
Upsert hit a NOT NULL on `email`	IntegrityError 23502	persistent (bad data mapping)	fail + alert

Code.

# Classify a raised exception into a resilience action.
import httpx
import psycopg2

RETRYABLE_STATUS = {408, 425, 429, 500, 502, 503, 504}

class FailureClass:
    TRANSIENT  = "transient"    # retry with backoff
    PERSISTENT = "persistent"   # fail fast + alert
    POISON     = "poison"       # dead-letter the record

def classify(exc: Exception) -> str:
    # Network-level blips are transient by nature.
    if isinstance(exc, (httpx.ConnectError, httpx.ReadError, ConnectionResetError, TimeoutError)):
        return FailureClass.TRANSIENT

    # HTTP status: split retryable server/throttle errors from client errors.
    if isinstance(exc, httpx.HTTPStatusError):
        code = exc.response.status_code
        if code in RETRYABLE_STATUS:
            return FailureClass.TRANSIENT
        if 400 <= code < 500:
            return FailureClass.PERSISTENT      # bad request/creds; retry won't help
        return FailureClass.TRANSIENT

    # A record we cannot even parse is a poison pill, not a dependency fault.
    if isinstance(exc, (UnicodeDecodeError, ValueError, KeyError)):
        return FailureClass.POISON

    # A constraint violation is bad data mapping; retrying replays the same failure.
    if isinstance(exc, psycopg2.IntegrityError):
        return FailureClass.PERSISTENT

    # Unknown: be conservative — do not retry something you don't understand.
    return FailureClass.PERSISTENT

Step-by-step explanation.

Network-level exceptions (ConnectError, ReadError, ConnectionResetError, TimeoutError) are transient by definition — the request never got a definitive answer, so trying again is both safe (for an idempotent call) and likely to succeed.
HTTP status errors split on the code. The RETRYABLE_STATUS set holds throttles (429), request timeouts (408), and server errors (500/502/503/504) — all "the server is struggling, back off and retry." Any other 4xx is a client error: retrying an expired token or a malformed body reproduces the failure exactly.
A record you cannot parse — a bad byte, a missing key, an un-coercible value — is a poison failure. The dependency is fine; the input is broken. Retrying reprocesses the same bad record forever, so it must be dead-lettered.
A Postgres IntegrityError (a NOT NULL or FK or unique violation) is classified persistent here because it signals a mapping bug: the same row will violate the same constraint on every retry. Some teams route it to poison instead, dead-lettering the offending row; the key point is that it is not transient.
The default branch is deliberately conservative: an unknown exception is treated as persistent (fail fast), never as transient. Retrying something you do not understand is how a small bug becomes a retry storm.

Output.

Failure	classify() returns	Downstream action
429 Too Many Requests	transient	enqueue for backed-off retry
401 Unauthorized	persistent	stop, refresh creds, alert
non-UTF-8 record	poison	write to dead-letter table
connection reset	transient	enqueue for backed-off retry
NOT NULL violation	persistent	stop, fix mapping, alert

Rule of thumb. Never wire a retry decorator to "any exception." Build a classify() function first; retry only the transient class, fail fast on persistent, and dead-letter poison. The classifier is the load-bearing decision — everything else is mechanism.

Worked example — the resilience-primitive decision tree

Detailed explanation. Given a failing call, the senior engineer runs a short decision tree to pick which primitive applies. Codifying the tree makes the interview answer reproducible: an interviewer can hand you any failure scenario and you can name the right primitive in seconds. Walk the tree over three canonical scenarios.

Q1. Is the call idempotent (safe to repeat)? → no = do not retry; make it idempotent first (idempotency key / upsert). yes = go to Q2.
Q2. Is the failure transient? → no = fail fast (persistent) or dead-letter (poison). yes = go to Q3.
Q3. Could the call hang? → yes = wrap in a timeout first. Always, in practice.
Q4. Is the dependency failing repeatedly across many calls? → yes = add a circuit breaker so you stop hammering it. no = plain retry + backoff is enough.
Q5 (parallel). Does this dependency share a pool with others that must stay healthy? → yes = put it behind a bulkhead.

Question. Walk the tree for three scenarios and record the primitives each needs.

Input.

Scenario	Idempotent?	Transient?	Can hang?	Repeated failure?
GET partner API (occasional 503)	yes (GET)	yes	yes	sometimes
POST charge to payment API	no	yes	yes	rare
Warehouse load sharing one pool with 3 sinks	yes (COPY to stage)	yes	yes	during brownouts

Code.

# Decision-tree helper (illustrative) — returns the primitives to apply.
def pick_primitives(idempotent: bool,
                    transient: bool,
                    can_hang: bool,
                    repeated_failure: bool,
                    shares_pool: bool) -> list[str]:
    primitives: list[str] = []

    if not idempotent:
        primitives.append("make-idempotent-first")   # prerequisite, not optional

    if can_hang:
        primitives.append("timeout")                 # always, in practice

    if transient and idempotent:
        primitives.append("retry+backoff+jitter")

    if repeated_failure:
        primitives.append("circuit-breaker")

    if shares_pool:
        primitives.append("bulkhead")

    return primitives


print(pick_primitives(True,  True, True, False, False))
# → ['timeout', 'retry+backoff+jitter']

print(pick_primitives(False, True, True, False, False))
# → ['make-idempotent-first', 'timeout', 'retry+backoff+jitter']

print(pick_primitives(True,  True, True, True,  True))
# → ['timeout', 'retry+backoff+jitter', 'circuit-breaker', 'bulkhead']

Step-by-step explanation.

Scenario 1 — an idempotent GET that occasionally 503s and can hang. The tree yields timeout + retry+backoff+jitter. No breaker needed because the failures are occasional, not sustained; no bulkhead because it has its own client.
Scenario 2 — a non-idempotent POST that charges a card. The tree first demands make-idempotent-first: attach an idempotency key so a retried charge is deduped server-side. Only then are timeout and retry safe.
Scenario 3 — a warehouse load that shares one connection pool with three other sinks and browns out under load. The tree yields the full stack: timeout, retry, circuit breaker (sustained brownouts), and bulkhead (shared pool must not be starved).
The tree makes explicit that primitives compose: most real calls need at least a timeout and a retry; the breaker and bulkhead are added when the failure is sustained or the resources are shared.
The most important branch is the first one. If a call is not idempotent, no amount of retry machinery is safe — the answer is to make it idempotent, not to skip the retry.

Output.

Scenario	Primitives to apply
GET partner API	timeout, retry+backoff+jitter
POST charge	idempotency key, timeout, retry+backoff+jitter
Shared-pool warehouse load	timeout, retry, circuit breaker, bulkhead

Rule of thumb. Walk the tree in order: idempotency first, then timeout, then retry, then breaker, then bulkhead. The order is not arbitrary — it is the same order you will nest the primitives in code (section 5).

Worked example — the cost of a naive retry loop

Detailed explanation. The classic anti-pattern is for _ in range(5): try call except: continue with no delay, no jitter, and no idempotency check. It looks like resilience and is actually an outage amplifier. Quantify what it does to a struggling dependency so you can argue against it with numbers, not opinions.

The setup. 200 pipeline workers each call the same API. The API slows down (a GC pause, a deploy). Every worker's call fails.
The naive loop. Each worker immediately retries up to 5 times with zero delay.
The effect. Instead of 200 requests, the API now receives up to 1,000 requests in a tight burst — precisely when it is least able to serve them.

Question. Quantify the request amplification of a naive tight retry loop versus a single attempt.

Input.

Parameter	Value
Concurrent workers	200
Naive retries per worker	5
Delay between retries	0 (tight loop)
Dependency state	overloaded, ~0% success

Code.

# Naive tight retry loop — DO NOT ship this.
def fetch_naive(client, url):
    for _ in range(5):
        try:
            return client.get(url)          # no timeout, no backoff, no jitter
        except Exception:
            continue                        # immediately hammer again
    raise RuntimeError("all retries failed")

# Amplification model
WORKERS      = 200
RETRIES      = 5
success_rate = 0.0     # dependency is down

# With ~0% success, every worker exhausts all attempts.
total_requests_naive  = WORKERS * (1 + RETRIES)   # first try + 5 retries
total_requests_single = WORKERS * 1

print(f"naive  : {total_requests_naive} requests in a tight burst")
print(f"single : {total_requests_single} requests")
print(f"amplification: {total_requests_naive / total_requests_single:.0f}x")
# naive  : 1200 requests in a tight burst
# single : 200 requests
# amplification: 6x

Step-by-step explanation.

When the dependency is healthy, the naive loop is invisible — the first attempt succeeds and the retries never fire. This is exactly why the anti-pattern survives code review: it looks harmless in the happy path.
When the dependency is down, success rate drops to ~0%, so every worker burns all six attempts (one initial + five retries). Two hundred workers become 1,200 requests — a 6× amplification — delivered in a tight burst with no spacing.
The burst arrives precisely when the dependency is weakest. A dependency that might have recovered from 200 requests is now guaranteed to stay down under 1,200. The retries have converted a brownout into an outage.
There is no jitter, so all 200 workers retry in lockstep — a synchronized "thundering herd" that hits the dependency in coordinated waves rather than a smooth trickle.
There is no timeout, so if the dependency hangs (rather than erroring), each worker blocks indefinitely — the retry count never even gets a chance to matter, and 200 workers are consumed forever.

Output.

Retry strategy	Requests to a down dependency	Recovery odds
Single attempt	200	dependency may recover
Naive tight loop (×6)	1,200	dependency pinned down
Backoff + jitter + budget	~200–400, spread over time	dependency recovers

Rule of thumb. Every retry loop must add three things the naive version lacks: a delay that grows (backoff), randomness so clients de-synchronize (jitter), and a ceiling on total retries across the fleet (a budget). Without all three, "adding retries" makes outages worse, not better — which is exactly what section 2 fixes.

Python
Topic — defensive-coding
Defensive-coding problems on failure classification

Practice →

Python Topic — exception-handling Exception-handling problems on retryable vs fatal errors

Practice →

2. Retries with backoff and jitter

`exponential backoff` grows the gap between attempts and `jitter` scatters them — together they turn retries from an outage amplifier into a recovery mechanism

The mental model in one line: a correct retry policy multiplies the wait between attempts (exponential backoff), adds randomness so independent clients stop retrying in lockstep (jitter), caps the total retries across the fleet (a retry budget), and fires only on idempotent, transient failures — the four elements together let a struggling dependency recover instead of being pinned down by a synchronized herd of immediate retries. Every senior data engineer has shipped a retry loop; every senior data engineer has, at least once, watched a naive one turn a five-minute blip into a two-hour incident.

Why fixed-delay retries synchronize into a thundering herd.

Fixed delay is deterministic. If every client retries exactly 1 second after a failure, and all clients failed at the same instant (because the dependency went down at once), they all retry at the same instant — a coordinated wave that hits the dependency as hard as the original burst.
Exponential backoff spreads waves apart in time. Waiting 1s, then 2s, then 4s, then 8s means the fleet's retries thin out: fewer clients are still retrying by attempt 4, and they are spaced further apart. The dependency gets breathing room that grows with each round.
But backoff alone still synchronizes. Even with exponential backoff, if all clients start their backoff at the same moment, attempt 2 for everyone lands ~2s later, attempt 3 ~6s later, and so on — the waves are further apart but still waves. Jitter is what breaks the synchronization within each wave.

The three jitter recipes — from the AWS backoff-and-jitter playbook.

Full jitter. sleep = random_between(0, min(cap, base * 2**attempt)). Each client waits a uniformly random amount between zero and the current exponential ceiling. This maximally de-correlates clients and is the recommended default for most workloads.
Equal jitter. sleep = half + random_between(0, half) where half = min(cap, base * 2**attempt) / 2. Keeps a guaranteed minimum wait (half the ceiling) plus a random half — useful when you want backoff to still grow visibly but with spread.
Decorrelated jitter. sleep = min(cap, random_between(base, prev_sleep * 3)). The next wait is a random value up to 3× the previous wait, which climbs quickly but stays random and self-limits at the cap. AWS's testing found decorrelated jitter competitive with full jitter and often better at draining a backlog fast.

The retry budget — capping the blast radius.

The problem. Per-call retries (e.g. "retry up to 5 times") bound one call but not the fleet. Under a broad outage, thousands of calls each retrying 5× is still a storm.
The token bucket. Maintain a shared budget (a token bucket) that refills slowly and is debited on every retry. When the bucket is empty, retries are skipped — the call fails fast instead of adding to the storm. A common rule of thumb is a budget of ~10–20% of the success traffic: retries may add at most 20% load on top of normal requests.
The effect. During a broad outage the budget drains quickly and most retries are suppressed, so the fleet's total load on the dependency stays bounded — the single most effective retry-storm defense.

Idempotency — the precondition for any retry.

The rule. You may only retry a call whose repetition is harmless. GET, HEAD, an upsert keyed by a natural key, a COPY into a staging table that dedupes on load — all safe. A raw INSERT that appends, a POST that charges, an "increment counter" — not safe.
The idempotency key. For unsafe operations, attach a client-generated idempotency key (a UUID per logical operation). The server stores processed keys and returns the original result on a duplicate — turning a non-idempotent call into a safely-retryable one.
The interview line. "I only retry idempotent operations; for non-idempotent ones I add an idempotency key so the retry is deduped server-side." Say this unprompted.

Common interview probes on retries.

"Why add jitter?" — required answer: to de-synchronize independent clients and prevent a thundering herd.
"What is a retry storm and how do you prevent it?" — backoff + jitter + a fleet-wide retry budget.
"Which failures do you retry?" — only idempotent, transient ones.
"How many retries?" — bounded by both a max-attempts count and the remaining deadline budget (section 3).

Worked example — exponential backoff with full jitter

Detailed explanation. The canonical retry helper: try a call up to N times, sleeping a full-jittered exponential delay between attempts, retrying only on classified-transient failures, and giving up when attempts or a max elapsed time run out. Build it from scratch so every knob is explicit.

Base. 0.5s initial backoff.
Cap. 30s ceiling on any single sleep.
Attempts. 6 maximum.
Jitter. Full jitter — uniform between 0 and the current ceiling.

Question. Implement a retry wrapper with exponential backoff and full jitter that retries only transient failures.

Input.

Parameter	Value
base	0.5 s
cap	30 s
max_attempts	6
jitter	full (uniform 0..ceiling)
retry predicate	classify(exc) == transient

Code.

import random
import time

def retry_full_jitter(fn, *, base=0.5, cap=30.0, max_attempts=6):
    """Call fn(); retry transient failures with full-jitter exponential backoff."""
    attempt = 0
    while True:
        try:
            return fn()
        except Exception as exc:
            attempt += 1
            if classify(exc) != FailureClass.TRANSIENT or attempt >= max_attempts:
                raise                      # non-transient, or out of attempts
            # Full jitter: uniform between 0 and the exponential ceiling.
            ceiling = min(cap, base * (2 ** (attempt - 1)))
            sleep_s = random.uniform(0, ceiling)
            print(f"attempt {attempt} failed ({exc!r}); sleeping {sleep_s:.2f}s "
                  f"(ceiling {ceiling:.1f}s)")
            time.sleep(sleep_s)

Step-by-step explanation.

The loop calls fn() and returns immediately on success — the happy path pays nothing. All retry machinery lives on the exception path.
On failure it increments the attempt counter, then consults classify(exc). If the failure is not transient, or attempts are exhausted, it re-raises — the wrapper never swallows a fatal error or loops forever.
ceiling = min(cap, base * 2**(attempt-1)) computes the exponential ceiling: 0.5s, 1s, 2s, 4s, 8s… clamped at the 30s cap. This is the upper bound on the wait, not the wait itself.
random.uniform(0, ceiling) applies full jitter — the actual sleep is a uniform random draw between 0 and the ceiling. Two workers that failed at the same instant now sleep different amounts, so their next attempts land at different times.
time.sleep(sleep_s) waits, then the loop tries again. The combination of a growing ceiling and per-attempt randomness is what turns a synchronized herd into a smooth, thinning trickle of retries.

Output.

Attempt	Ceiling	Example jittered sleep
1	0.5 s	0.31 s
2	1.0 s	0.74 s
3	2.0 s	0.12 s
4	4.0 s	3.55 s
5	8.0 s	2.90 s
6	(raise — attempts exhausted)	—

Rule of thumb. Default to full jitter with an explicit base, cap, and max_attempts, and gate every retry on a transient classification. The jitter is not optional polish — it is the difference between retries that help and retries that synchronize into a storm.

Worked example — a fleet-wide retry budget (token bucket)

Detailed explanation. Per-call max_attempts bounds one call; it does nothing to stop ten thousand calls each retrying during a broad outage. A shared token-bucket budget caps the fleet's retry load: retries spend tokens, the bucket refills slowly, and when it is empty retries are suppressed. Build a thread-safe budget and wire it into the retry decision.

Refill. Tokens accrue at a rate tied to success traffic (e.g. 0.2 tokens per successful call → retries may add at most 20% load).
Cost. Each retry costs 1 token.
Suppression. No token → skip the retry, fail fast.

Question. Implement a retry budget and modify the retry loop to consult it before each retry.

Input.

Parameter	Value
Budget ratio	0.2 (retries ≤ 20% of success traffic)
Max bucket	100 tokens
Cost per retry	1 token
On empty	skip retry, raise

Code.

import threading

class RetryBudget:
    """Token bucket capping fleet-wide retries to a fraction of success traffic."""
    def __init__(self, ratio=0.2, max_tokens=100.0):
        self.ratio      = ratio
        self.max_tokens = max_tokens
        self.tokens     = max_tokens
        self._lock      = threading.Lock()

    def on_success(self):
        # Every success earns partial credit toward future retries.
        with self._lock:
            self.tokens = min(self.max_tokens, self.tokens + self.ratio)

    def try_spend(self) -> bool:
        # Spend one token for a retry; return False if the budget is exhausted.
        with self._lock:
            if self.tokens >= 1.0:
                self.tokens -= 1.0
                return True
            return False


BUDGET = RetryBudget(ratio=0.2, max_tokens=100.0)

def retry_with_budget(fn, *, base=0.5, cap=30.0, max_attempts=6):
    attempt = 0
    while True:
        try:
            result = fn()
            BUDGET.on_success()            # replenish on every success
            return result
        except Exception as exc:
            attempt += 1
            if classify(exc) != FailureClass.TRANSIENT or attempt >= max_attempts:
                raise
            if not BUDGET.try_spend():     # fleet-wide brake
                raise RuntimeError("retry budget exhausted — failing fast") from exc
            ceiling = min(cap, base * (2 ** (attempt - 1)))
            time.sleep(random.uniform(0, ceiling))

Step-by-step explanation.

The bucket starts full (100 tokens) and every successful call adds ratio (0.2) tokens back, capped at max_tokens. In steady state, plenty of successes keep the bucket topped up, so occasional retries always find a token.
Each retry calls try_spend(), which atomically debits one token or returns False. The lock makes it safe across the many threads a worker pool runs.
During a broad outage, successes stop, so the bucket stops refilling. The first ~100 retries drain it; after that, try_spend() returns False and every further retry is suppressed — the loop raises immediately instead of adding to the storm.
This is the crucial fleet-level property: no matter how many calls are failing, the total retry load is bounded by the bucket size plus the (now-zero) refill rate. The dependency sees normal traffic plus a small, bounded retry overhead, never a 6× amplification.
When the dependency recovers, successes resume, the bucket refills, and retries are re-enabled automatically — no manual intervention, no config flip.

Output.

Fleet state	Successes	Bucket behavior	Retries allowed?
Healthy	many	stays near full	yes
Brief blip	mostly succeeding	drains slowly, refills	yes
Broad outage	~0	drains to 0, no refill	suppressed after ~100
Recovering	resuming	refills at 0.2/success	re-enabled gradually

Rule of thumb. Bound retries in two dimensions: per-call (max_attempts) and fleet-wide (a token-bucket budget of ~10–20% of success traffic). The per-call cap stops one call from looping; the budget stops the fleet from storming.

Worked example — idempotency keys make an unsafe call retryable

Detailed explanation. Retries are only safe on idempotent operations. When you must retry something that mutates state exactly-once (a payment, a "create shipment", an append), attach a client-generated idempotency key so the server dedupes duplicates. Walk through the pattern on a non-idempotent "register a payout" call.

The key. A UUID generated once per logical operation and reused across all retries of that operation.
The server contract. On first sight of a key, process and store the result under it; on a duplicate key, return the stored result without re-processing.
The result. The client can retry freely; the effect happens at most once.

Question. Make a non-idempotent payout call safely retryable with an idempotency key.

Input.

Component	Value
Operation	POST /payouts (non-idempotent)
Idempotency key	UUID per logical payout
Server store	processed_keys table
Retry policy	full jitter + budget

Code.

import uuid

def create_payout(client, account_id, amount_cents, idem_key):
    """Non-idempotent server call made safe by an idempotency key header."""
    resp = client.post(
        "/payouts",
        json={"account_id": account_id, "amount_cents": amount_cents},
        headers={"Idempotency-Key": idem_key},   # the safety mechanism
    )
    resp.raise_for_status()
    return resp.json()

def payout_once(client, account_id, amount_cents):
    # Generate the key ONCE, outside the retry loop, so every retry reuses it.
    idem_key = str(uuid.uuid4())
    return retry_with_budget(
        lambda: create_payout(client, account_id, amount_cents, idem_key)
    )

-- Server-side dedupe table backing the Idempotency-Key contract
CREATE TABLE processed_payouts (
    idem_key     UUID        PRIMARY KEY,
    account_id   BIGINT      NOT NULL,
    amount_cents BIGINT      NOT NULL,
    result       JSONB       NOT NULL,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
);

-- On each request: try to claim the key; if it already exists, return the stored result.
INSERT INTO processed_payouts (idem_key, account_id, amount_cents, result)
VALUES (%s, %s, %s, %s)
ON CONFLICT (idem_key) DO NOTHING
RETURNING result;

Step-by-step explanation.

payout_once generates the idempotency key once, outside the retry loop. This is the single most common mistake: generating the key inside the retried lambda would produce a new key per attempt, defeating the whole mechanism.
Every retry of create_payout sends the same Idempotency-Key header. From the server's perspective, all attempts of one logical payout are tagged identically.
Server-side, the INSERT ... ON CONFLICT (idem_key) DO NOTHING atomically claims the key. On the first attempt the row is inserted and the payout is processed; on any retry the conflict fires, no duplicate payout is created, and the stored result is returned.
This converts a non-idempotent operation into one that is safe to retry: the money moves at most once, no matter how many times the network eats the response and the client retries.
With idempotency in place, the payout call can now flow through the same retry_with_budget wrapper as any read — the earlier failure classes and budget apply unchanged. Idempotency is the key that unlocks retries for mutating operations.

Output.

Attempt	Idempotency-Key	Server action	Payouts created
1 (response lost)	k-abc	insert key, process	1
2 (retry)	k-abc	conflict → return stored	still 1
3 (retry)	k-abc	conflict → return stored	still 1

Rule of thumb. Generate the idempotency key once per logical operation, reuse it across every retry, and back it with an ON CONFLICT DO NOTHING claim server-side. This is the bridge that lets you retry mutating calls without double-effects.

Data-engineering interview question on resilient retries

A senior interviewer might ask: "You own an hourly extractor that pulls from a partner REST API. During their nightly maintenance the API returns bursts of 503s and occasionally hangs. Your current code retries immediately five times and has twice caused their on-call to page you back for 'hammering us during maintenance.' Redesign the retry logic: which failures you retry, the backoff strategy, how you stop the fleet from storming, and how you keep the extractor's own SLA."

Solution Using decorrelated jitter with a retry budget and per-attempt deadline check

import random
import time

def retry_decorrelated(fn, *, base=0.5, cap=20.0, max_attempts=6,
                       deadline_s=None, budget: "RetryBudget" = BUDGET):
    """Retry transient failures with decorrelated jitter, a fleet budget,
    and an overall deadline so retries never overrun the task SLA."""
    start   = time.monotonic()
    sleep_s = base
    attempt = 0
    while True:
        try:
            result = fn()
            budget.on_success()
            return result
        except Exception as exc:
            attempt += 1
            if classify(exc) != FailureClass.TRANSIENT or attempt >= max_attempts:
                raise
            if not budget.try_spend():
                raise RuntimeError("retry budget exhausted") from exc
            # Decorrelated jitter: next wait is random up to 3x the previous.
            sleep_s = min(cap, random.uniform(base, sleep_s * 3))
            # Never sleep past the overall deadline.
            if deadline_s is not None:
                remaining = deadline_s - (time.monotonic() - start)
                if remaining <= 0:
                    raise TimeoutError("deadline exceeded before retry") from exc
                sleep_s = min(sleep_s, remaining)
            time.sleep(sleep_s)

# Wire it to the extractor with a 90s task deadline.
def extract_hourly(client, url):
    return retry_decorrelated(
        lambda: client.get(url, timeout=httpx.Timeout(connect=1.0, read=5.0, timeout=8.0)),
        base=0.5, cap=20.0, max_attempts=6, deadline_s=90.0,
    )

Step-by-step trace.

Step	Value	Reasoning
Retry predicate	classify(exc) == transient	503/reset retried; 401/400 fail fast
Backoff	decorrelated jitter (random up to 3× prev)	climbs fast, stays de-synchronized
Cap	20 s per sleep	one slow retry can't eat the whole deadline
Fleet brake	token budget (20% of success)	suppresses retries during broad outage
Overall deadline	90 s	retries never overrun the task SLA
Per-call timeout	connect 1s / read 5s / total 8s	a hang becomes a fast retryable failure

After the redesign, a 503 burst triggers backed-off, jittered retries that spread across the fleet instead of arriving in lockstep; the shared budget suppresses retries once the partner is broadly down, so the extractor stops hammering during maintenance; and the 90-second deadline guarantees the hourly task still finishes (or fails cleanly) within its slot instead of retrying into the next hour.

Output:

Metric	Before (naive)	After (decorrelated + budget + deadline)
Requests to a down partner	6× amplification, bursty	~1× + bounded budget overhead
Retry timing	synchronized waves	de-synchronized spread
Behavior during maintenance	pages partner on-call	suppressed by budget
Task overrun risk	retries into next hour	capped at 90 s deadline
Hang handling	worker blocked forever	per-call timeout → fast retry

Why this works — concept by concept:

Transient-only retry — the classify() gate ensures only 503s, throttles, and network resets are retried; a 401 or a malformed request fails fast and pages a human instead of looping uselessly.
Decorrelated jitter — random.uniform(base, prev * 3) climbs quickly (draining a backlog fast) while keeping every client's timing independent, so the fleet never re-synchronizes into a herd.
Retry budget — the shared token bucket caps fleet-wide retry load to ~20% of success traffic; during a broad outage it drains and suppresses retries, which is the actual retry-storm defense (per-call caps alone are not).
Overall deadline — clamping each sleep to the remaining budget and aborting when it hits zero guarantees the hourly extractor finishes within its slot; resilience must never come at the cost of the task's own SLA.
Cost — O(max_attempts) calls in the worst case per task, bounded further by the deadline; the budget adds one atomic token op per call. Compared to the naive loop's 6× burst, this is bounded load that lets the dependency recover — the difference between a self-healing pipeline and a self-inflicted outage.

Python
Topic — exception-handling
Exception-handling problems on retry and backoff

Practice →

Python Topic — defensive-coding Defensive-coding problems on idempotent retries

Practice →

3. Timeouts and deadline budgets

A `timeout` bounds one call; a `timeout budget` carries one deadline down the whole chain so a slow hop can't blow the end-to-end SLA

The mental model in one line: every remote call must carry a timeout — separate connect, read, and total tiers — and a multi-hop pipeline must carry a single end-to-end deadline that each hop decrements and propagates downstream, so that no call hangs forever, no downstream timeout exceeds the time actually left, and the whole chain fails fast the instant the budget is spent. The missing timeout is the most dangerous bug in resilience engineering because it is invisible until a dependency hangs — and then it consumes a worker indefinitely, which no retry, breaker, or bulkhead can fix after the fact.

The three timeout tiers — set all of them, always.

Connect timeout. How long to wait to establish the connection (TCP + TLS handshake). Should be short — 1–2 seconds — because a healthy endpoint connects fast; a slow connect means the endpoint or network is unhealthy and you want to fail quickly to a retry.
Read timeout. How long to wait for the response body after the request is sent (the server's time to produce and stream data). Tuned to the operation: a few seconds for a lookup, longer for a report query. This is the tier that catches a server that accepted the request but is grinding.
Total (overall) timeout. A hard ceiling on the entire call including connect, all reads, and any redirects/retries within the client. This is the backstop that guarantees the call cannot exceed a known wall-clock bound even if the finer tiers are misconfigured.

The missing-timeout trap.

The default is often "wait forever." Many clients and drivers default to no timeout. A single call to a hung dependency then blocks its worker thread indefinitely.
Hangs are worse than errors. An error is fast and retryable; a hang silently consumes capacity. A pool of 50 workers all calling one hung dependency is 50 workers gone — the pipeline stalls with no error to alert on.
The rule. Never make a network call — HTTP, database, queue, RPC, DNS — without an explicit timeout. "No timeout" is a latent outage waiting for the dependency to hang.

The deadline budget — one clock for the whole chain.

The idea. Instead of each hop having an independent timeout, the entry point sets one end-to-end deadline (e.g. 10 seconds). Each hop computes how much of that budget remains, uses it (minus a margin) as its timeout, and passes the reduced budget to the next hop.
The invariant. A downstream call's timeout must be less than the remaining budget. Calling a downstream service with a 30-second timeout when only 2 seconds of budget remain is pointless work — the caller will have already given up.
Propagation. The remaining deadline travels with the request (an absolute timestamp in a header, a gRPC deadline, a context object). Downstream services honor it and refuse work they can't finish in time.

Timeout × retry interaction — retries live inside the deadline.

The trap. A per-call timeout of 8s plus 5 retries can consume 40+ seconds — far past a 10-second task deadline. The retry count and the timeout must be reconciled against the budget.
The fix. The retry loop checks the remaining budget before each attempt (as in section 2's retry_decorrelated), and never sleeps or retries past it. Retries are opportunistic within the deadline, not additive to it.
The math. With a 10s budget and an 8s per-call timeout, there is room for at most one attempt plus a short retry — not five. The budget, not the retry count, is the real limit.

Common interview probes on timeouts.

"What timeouts do you set on an HTTP call?" — required answer: connect, read, and total — not one blanket value.
"What happens if you don't set a timeout?" — the call can hang forever and consume the worker.
"What is a deadline budget?" — one end-to-end deadline decremented and propagated per hop.
"How do retries fit inside a deadline?" — retries only run while budget remains; the deadline caps total time, not the retry count.

Worked example — connect / read / total timeout tiers

Detailed explanation. The canonical HTTP client setup for a data-pipeline extractor: distinct connect, read, and total timeouts tuned to the operation, so a slow connect fails in a second while a legitimately long report read gets the time it needs — all under a hard total ceiling. Configure it explicitly.

Connect. 1s — a healthy endpoint connects fast.
Read. 5s for a normal API; 30s for a heavy report endpoint.
Total. 8s (normal) — hard ceiling regardless of the finer tiers.

Question. Configure an httpx client with tiered timeouts and show how each tier fires.

Input.

Tier	Normal endpoint	Report endpoint
connect	1 s	1 s
read	5 s	30 s
total	8 s	35 s
pool wait	2 s	2 s

Code.

import httpx

# Tiered timeouts — never a single blanket value.
NORMAL = httpx.Timeout(connect=1.0, read=5.0, write=5.0, pool=2.0, timeout=8.0)
REPORT = httpx.Timeout(connect=1.0, read=30.0, write=5.0, pool=2.0, timeout=35.0)

def fetch(client: httpx.Client, url: str, *, heavy: bool = False):
    tier = REPORT if heavy else NORMAL
    try:
        resp = client.get(url, timeout=tier)
        resp.raise_for_status()
        return resp.json()
    except httpx.ConnectTimeout:
        raise RuntimeError(f"connect > {tier.connect}s — endpoint/network unhealthy")
    except httpx.ReadTimeout:
        raise RuntimeError(f"read > {tier.read}s — server accepted but is grinding")
    except httpx.PoolTimeout:
        raise RuntimeError(f"pool wait > {tier.pool}s — client out of connections")

Step-by-step explanation.

httpx.Timeout takes independent connect, read, write, pool, and overall timeout values. Setting them separately lets each catch a distinct failure mode instead of one blanket number that is either too tight for reads or too loose for connects.
connect=1.0 fails fast when the endpoint can't be reached — a slow connect almost always means the network or endpoint is unhealthy, and you want that to become a quick, retryable error rather than a long stall.
read=5.0 (or 30 for reports) bounds the server's time to produce the body. A ReadTimeout means the server accepted the request but is grinding — distinct from a connect failure and often worth a different reaction.
pool=2.0 bounds how long the call waits for a free connection from the client's pool. A PoolTimeout means your side is out of connections (a local bottleneck), which is exactly the signal a bulkhead (section 4) acts on.
The overall timeout=8.0 is the hard ceiling — even if the finer tiers are misconfigured, no call exceeds it. Distinguishing the exception types lets the caller log precisely which tier fired, which is invaluable when debugging a flaky dependency.

Output.

Failure mode	Exception raised	Meaning
Endpoint unreachable	ConnectTimeout	network/endpoint down → retry
Server accepted, grinding	ReadTimeout	slow response → retry / breaker
No free client connection	PoolTimeout	local pool exhausted → bulkhead
Everything slow	overall timeout	hard ceiling hit

Rule of thumb. Always set connect, read, and total timeouts as separate values tuned to the operation, and distinguish the exception types so logs tell you which tier fired. A single blanket timeout is either too tight for slow reads or too loose to catch a hung connect.

Worked example — a deadline budget propagated down a call chain

Detailed explanation. A three-hop enrichment pipeline (extract → enrich → load) must finish within one end-to-end deadline. Instead of three independent timeouts that can sum past the SLA, carry one deadline: each hop computes the remaining budget and uses it (minus a margin) as its own timeout. Build the propagation with an absolute-deadline object.

Entry deadline. 10 seconds total.
Per-hop margin. 200 ms reserved for local processing between hops.
Propagation. Each hop passes the remaining budget to the next.

Question. Implement a Deadline that each hop consults to derive its own timeout, and abort the chain when the budget is spent.

Input.

Hop	Nominal cost	Timeout derived from
extract	~4 s	remaining budget (10 s)
enrich	~4 s	remaining budget (~6 s)
load	~2 s	remaining budget (~2 s)

Code.

import time
import httpx

class Deadline:
    """One end-to-end deadline, propagated and decremented per hop."""
    def __init__(self, budget_s: float, margin_s: float = 0.2):
        self.expiry_at = time.monotonic() + budget_s
        self.margin    = margin_s

    def remaining(self) -> float:
        return self.expiry_at - time.monotonic()

    def timeout_for_hop(self) -> float:
        # A hop may use the remaining budget minus a margin — never more.
        left = self.remaining() - self.margin
        if left <= 0:
            raise TimeoutError("deadline exceeded before hop could start")
        return left


def run_chain(client, record, budget_s=10.0):
    dl = Deadline(budget_s)

    # Hop 1 — extract: timeout bounded by the whole remaining budget.
    raw = client.get(f"/source/{record}",
                     timeout=httpx.Timeout(dl.timeout_for_hop())).json()

    # Hop 2 — enrich: less budget remains; downstream timeout shrinks accordingly.
    enriched = client.post("/enrich", json=raw,
                           timeout=httpx.Timeout(dl.timeout_for_hop())).json()

    # Hop 3 — load: whatever is left; if <= margin, we abort instead of calling.
    client.post("/load", json=enriched,
                timeout=httpx.Timeout(dl.timeout_for_hop()))
    return {"loaded": record, "budget_left_s": round(dl.remaining(), 2)}

Step-by-step explanation.

Deadline records an absolute expiry (monotonic() + budget) rather than a duration. Absolute deadlines propagate correctly across hops because "how much time is left" is always expiry - now, regardless of how long earlier hops took.
timeout_for_hop() returns the remaining budget minus a small margin. This is the invariant in code: a hop's timeout is derived from what's left, so a downstream call can never be given more time than the whole chain has.
If a hop has already overrun (remaining ≤ margin), timeout_for_hop() raises TimeoutError before making the call — refusing to start work that cannot finish in time. This is deadline-aware load shedding: you don't pay for a call you'll abandon.
As the chain progresses, each hop sees a smaller budget: extract gets ~10s, enrich gets ~6s, load gets ~2s. The downstream timeout automatically shrinks — no per-hop constant to keep in sync, no risk of the sum exceeding the SLA.
monotonic() (not time.time()) is used throughout so the deadline is immune to wall-clock adjustments (NTP steps, leap seconds) that could otherwise make "remaining" jump.

Output.

Hop	Remaining on entry	Timeout used	Remaining on exit
extract	10.0 s	9.8 s	6.1 s
enrich	6.1 s	5.9 s	2.0 s
load	2.0 s	1.8 s	0.3 s
(if enrich overran to 0.1 s)	0.1 s	raise TimeoutError	chain aborts

Rule of thumb. Propagate one absolute deadline down the chain and derive each hop's timeout from the remaining budget minus a margin. Independent per-hop timeouts that can sum past the SLA are a latent breach; a shared, decrementing deadline makes the SLA a hard guarantee.

Worked example — reconciling retries with the remaining budget

Detailed explanation. Retries and deadlines interact: a generous per-call timeout plus several retries can silently overrun the task SLA. The fix is to let the retry loop and the timeout both read the same deadline, so retries only happen while budget remains and each attempt's timeout is clamped to what's left. Walk through the math and the code.

Budget. 10s task deadline.
Per-call timeout. min(8s, remaining budget).
Retries. Allowed only while remaining > 0.

Question. Combine the retry loop and the deadline so total time never exceeds the budget.

Input.

Parameter	Value
Task deadline	10 s
Nominal per-call timeout	8 s
Backoff	full jitter, base 0.5 s
Constraint	total wall-clock ≤ 10 s

Code.

def call_within_deadline(client, url, *, budget_s=10.0, nominal_timeout=8.0,
                         base=0.5, max_attempts=6):
    dl      = Deadline(budget_s, margin_s=0.1)
    attempt = 0
    while True:
        # Each attempt's timeout is the smaller of nominal and what's left.
        per_call = min(nominal_timeout, dl.timeout_for_hop())
        try:
            r = client.get(url, timeout=httpx.Timeout(per_call))
            r.raise_for_status()
            return r.json()
        except Exception as exc:
            attempt += 1
            if classify(exc) != FailureClass.TRANSIENT or attempt >= max_attempts:
                raise
            ceiling  = min(4.0, base * (2 ** (attempt - 1)))
            sleep_s  = random.uniform(0, ceiling)
            # Do not sleep past the deadline; if no budget remains, give up.
            if dl.remaining() - sleep_s <= dl.margin:
                raise TimeoutError("no budget left for another attempt") from exc
            time.sleep(sleep_s)

Step-by-step explanation.

A single Deadline governs both the per-call timeout and the retry decision, so there is one source of truth for "how much time is left" — the loop can never disagree with the timeout.
per_call = min(nominal_timeout, dl.timeout_for_hop()) clamps each attempt's timeout to the remaining budget. Early attempts get up to the nominal 8s; later attempts get less as the budget shrinks. No attempt can be granted more time than the task has.
Before sleeping for backoff, the loop checks dl.remaining() - sleep_s: if sleeping would leave no time for another attempt, it raises immediately rather than burning the budget on a wait that leads nowhere.
The effect is that the retry count becomes secondary — the deadline is the real limit. With a 10s budget and an 8s first attempt, there is only room for one retry with a short backoff, and the code enforces exactly that.
This is the correct reconciliation the interview probes for: retries are opportunistic within the deadline, never additive to it. The task finishes within its SLA whether it succeeds on attempt one or exhausts its budget.

Output.

Attempt	Budget left	per_call timeout	Outcome
1	10.0 s	8.0 s	fails at 8 s
backoff	2.0 s	sleep 0.4 s	ok, budget remains
2	1.6 s	1.5 s	fails at 1.5 s
backoff	0.1 s	—	raise (no budget)

Rule of thumb. Feed the retry loop and the per-call timeout from one shared deadline. When they read the same clock, retries automatically fit inside the SLA and the task never overruns — the number of retries you actually get is dictated by the budget, not a constant.

Data-engineering interview question on deadline budgets

A senior interviewer might ask: "You have a multi-hop enrichment pipeline — extract from a source API, call a geocoding service, then load to the warehouse — with a strict 12-second per-record SLA because it runs inline in a streaming consumer. Occasionally the geocoder gets slow and the whole record processing blows past 12 seconds, backing up the consumer lag. Design the timeout and deadline strategy so no record ever exceeds 12 seconds end-to-end, including retries."

Solution Using a propagated deadline budget with tiered timeouts and deadline-aware retries

import time
import httpx

class Deadline:
    def __init__(self, budget_s, margin_s=0.15):
        self.expiry_at = time.monotonic() + budget_s
        self.margin    = margin_s
    def remaining(self):
        return self.expiry_at - time.monotonic()
    def timeout_for_hop(self, cap=None):
        left = self.remaining() - self.margin
        if left <= 0:
            raise TimeoutError("deadline exceeded")
        return min(left, cap) if cap else left

def process_record(client, record, budget_s=12.0):
    dl = Deadline(budget_s)

    # Hop 1 — extract (retryable within the shared deadline).
    raw = _hop(client, "GET", f"/source/{record}", None, dl, cap=5.0)

    # Hop 2 — geocode (the slow one): capped and deadline-bounded.
    geo = _hop(client, "POST", "/geocode", {"addr": raw["addr"]}, dl, cap=4.0)

    # Hop 3 — load (idempotent COPY into staging).
    _hop(client, "POST", "/load", {**raw, **geo}, dl, cap=3.0)
    return {"record": record, "ms_left": round(dl.remaining() * 1000)}

def _hop(client, method, url, body, dl: Deadline, cap):
    attempt = 0
    while True:
        per_call = dl.timeout_for_hop(cap=cap)     # min(remaining, cap)
        try:
            r = client.request(method, url, json=body,
                               timeout=httpx.Timeout(connect=1.0, read=per_call,
                                                     write=per_call, pool=1.0,
                                                     timeout=per_call))
            r.raise_for_status()
            return r.json() if r.content else {}
        except Exception as exc:
            attempt += 1
            if classify(exc) != FailureClass.TRANSIENT or attempt >= 3:
                raise
            sleep_s = random.uniform(0, min(0.5, dl.remaining() / 4))
            if dl.remaining() - sleep_s <= dl.margin:
                raise TimeoutError(f"{url}: no budget for retry") from exc
            time.sleep(sleep_s)

Step-by-step trace.

Step	Value	Reasoning
Entry deadline	12 s (absolute, monotonic)	matches the streaming SLA
Per-hop cap	extract 5 s / geocode 4 s / load 3 s	keeps any one hop from eating all budget
Per-call timeout	min(remaining − margin, cap)	never exceeds what the chain has left
Retry within hop	≤ 3, transient-only, budget-checked	retries fit inside the deadline
Geocoder slow path	capped at 4 s, then fail/degrade	slow hop can't back up the consumer
Abort condition	remaining ≤ margin	shed load rather than overrun

After the redesign, each record carries one 12-second monotonic deadline. The slow geocoder is capped at whatever is left up to 4 seconds; if it can't answer in time the hop aborts and the record is handled by the degraded path (e.g. loaded without geocoding, flagged for backfill) — the consumer never blocks past 12 seconds, so lag stays flat even during a geocoder brownout.

Output:

Metric	Before	After
Worst-case per-record time	unbounded (geocoder-driven)	≤ 12 s hard
Consumer lag during brownout	grows	flat
Slow-hop handling	blocks the chain	capped + degraded path
Retry overrun	possible	impossible (budget-checked)
Timeout coverage	partial	connect/read/total on every hop

Why this works — concept by concept:

Absolute monotonic deadline — one Deadline object per record expresses the 12-second SLA as an absolute instant; every hop and every retry reads remaining() from the same clock, so nothing can independently overrun.
Per-hop cap plus remaining budget — timeout_for_hop(cap) uses min(remaining − margin, cap), so no hop hogs the budget and none is ever granted more time than the chain has left.
Deadline-aware retries — retries are transient-only, capped at three, and each checks that a backoff sleep still leaves budget; retries live inside the deadline rather than adding to it.
Degraded path on abort — when the budget is spent the hop raises instead of blocking, letting the consumer fall back (load without geocode, flag for backfill) so lag never grows — availability over completeness under time pressure.
Cost — O(hops × attempts) calls bounded hard by 12 s wall-clock; monotonic arithmetic is O(1) per check. Compared to independent per-hop timeouts that can sum to 30s+, the shared deadline converts a soft target into a hard guarantee — the property a streaming SLA actually requires.

Python
Topic — defensive-coding
Defensive-coding problems on timeouts and deadlines

Practice →

ETL Topic — etl ETL problems on SLA-bounded multi-hop pipelines

Practice →

4. Circuit breakers and bulkheads

A `circuit breaker` stops calling a dependency that keeps failing; a `bulkhead` isolates pools so one sick dependency can't drown the healthy ones

The mental model in one line: a circuit breaker is a three-state machine (closed → open → half-open) that watches the recent failure ratio for a dependency, trips open to fail fast when failures exceed a threshold, waits a reset timeout, then admits a single probe on half-open to test recovery — and a bulkhead is a bounded resource pool per dependency so that a saturated or slow dependency consumes only its own slice of capacity and cannot starve the rest of the pipeline. Retries and timeouts protect a single call; breakers and bulkheads protect the whole worker from a dependency that is broadly, persistently sick.

The three breaker states.

Closed. Normal operation — calls pass through to the dependency. The breaker records outcomes (success/failure) in a rolling window. As long as the failure ratio stays below the threshold, it stays closed.
Open. Tripped — calls fail immediately without touching the dependency (a fast CircuitOpenError, optionally a fallback). This is the key behavior: instead of every call timing out slowly against a dead dependency, they fail in microseconds, freeing workers and giving the dependency room to recover.
Half-open. After a reset timeout, the breaker admits a single trial call (a probe). If it succeeds, the breaker closes (recovery confirmed); if it fails, the breaker re-opens and the reset timer restarts. Half-open prevents a stampede of calls the instant the timer expires.

The trip condition — ratio over a rolling window, not a raw count.

Why ratio, not count. "Trip after 5 failures" trips on 5 failures out of 5 or 5 out of 5,000 — very different situations. A failure ratio over a rolling window (e.g. "> 50% of the last 20 calls failed") trips only when the dependency is genuinely, broadly unhealthy.
The minimum-volume guard. Require a minimum number of calls in the window before the ratio can trip (e.g. at least 10 calls). This stops a single early failure from tripping the breaker before there is enough signal.
The window. A rolling count-based window (last N calls) or time-based window (last T seconds). Count-based is simpler and common; time-based adapts better to bursty traffic.

Bulkheads — isolation so one leak doesn't sink the ship.

The metaphor. A ship's hull is divided into sealed compartments (bulkheads); a breach floods one compartment, not the whole vessel. In software, a bulkhead is a bounded pool (connections, threads, semaphore permits) per dependency.
The failure it prevents. Without bulkheads, one slow dependency's calls pile up and consume every worker/connection in a shared pool; healthy dependencies then can't get a connection and the whole pipeline stalls — a cascading failure from one sick component.
The mechanism. Give each dependency its own bounded pool (or a semaphore capping concurrent calls). When dependency B is slow and its pool fills, calls to B are rejected fast (or queued briefly), but dependencies A and C still have their own capacity and keep working.

Fallbacks and load shedding when the breaker is open.

Fallback. When the breaker is open, return a degraded-but-useful result instead of an error where possible: a cached value, a default, a "skip enrichment and flag for backfill" path. Availability over completeness under stress.
Load shedding. If there is no useful fallback, fail fast and cheaply — an open breaker is load shedding, shedding load off the struggling dependency so it can recover.
The interview line. "An open breaker fails fast; where a degraded result is acceptable I attach a fallback (cache, default, skip-and-backfill) so the pipeline stays available."

Common interview probes on breakers and bulkheads.

"How does a circuit breaker differ from a retry?" — retry re-attempts one call; a breaker stops attempting a broadly-failing dependency.
"What are the three states?" — closed, open, half-open.
"What trips it?" — a failure ratio over a rolling window, guarded by a minimum call volume.
"What is a bulkhead?" — per-dependency resource isolation so one sick dependency can't starve the others.
"What does half-open do?" — admits one probe to test recovery without a stampede.

Worked example — a rolling-window circuit breaker

Detailed explanation. Build a thread-safe circuit breaker with the three states, a rolling-window failure ratio, a minimum-volume guard, and a reset timeout with a half-open probe. This is the exact shape interviewers expect you to be able to sketch.

Threshold. Trip at > 50% failures.
Window. Last 20 outcomes.
Min volume. At least 10 calls before tripping.
Reset timeout. 30 seconds open before a half-open probe.

Question. Implement the breaker and its call() guard.

Input.

Parameter	Value
failure_threshold	0.5
window_size	20
min_volume	10
reset_timeout_s	30

Code.

import time
import threading
from collections import deque

class CircuitOpenError(Exception):
    pass

class CircuitBreaker:
    def __init__(self, failure_threshold=0.5, window_size=20,
                 min_volume=10, reset_timeout_s=30.0):
        self.threshold   = failure_threshold
        self.min_volume  = min_volume
        self.reset_after = reset_timeout_s
        self.window      = deque(maxlen=window_size)   # True=failure
        self.state       = "closed"
        self.opened_at   = 0.0
        self._lock       = threading.Lock()

    def _failure_ratio(self):
        if len(self.window) < self.min_volume:
            return 0.0                                 # not enough signal
        return sum(self.window) / len(self.window)

    def call(self, fn):
        with self._lock:
            if self.state == "open":
                if time.monotonic() - self.opened_at >= self.reset_after:
                    self.state = "half-open"           # admit one probe
                else:
                    raise CircuitOpenError("circuit open — failing fast")

        try:
            result = fn()
        except Exception:
            self._record(True)                         # failure
            raise
        else:
            self._record(False)                        # success
            return result

    def _record(self, failed: bool):
        with self._lock:
            if self.state == "half-open":
                # A probe result decides the outcome definitively.
                if failed:
                    self.state, self.opened_at = "open", time.monotonic()
                else:
                    self.state, self.window = "closed", deque(maxlen=self.window.maxlen)
                return
            self.window.append(failed)
            if self.state == "closed" and self._failure_ratio() > self.threshold:
                self.state, self.opened_at = "open", time.monotonic()

Step-by-step explanation.

The window is a bounded deque of booleans (True = failure). _failure_ratio() returns 0 until at least min_volume outcomes exist — the minimum-volume guard that stops one early failure from tripping the breaker prematurely.
call() first checks state under the lock. If open and the reset timeout has elapsed, it transitions to half-open to admit a single probe; if open and still cooling down, it raises CircuitOpenError immediately — the fast-fail that defines the open state.
On the actual call, success and failure are both recorded via _record(). In the closed state, each outcome appends to the window and the ratio is re-checked; crossing the threshold trips the breaker open and stamps opened_at.
In the half-open state, the probe's result is decisive: a success closes the breaker (and resets the window for a clean slate); a failure re-opens it and restarts the reset timer. Only one probe is admitted because the state flips off "half-open" as soon as the probe runs.
All state transitions happen under the lock, so a pool of concurrent workers sees a consistent breaker — critical because breakers are inherently shared across the callers of one dependency.

Output.

Event sequence	State after
12 calls, 7 fail (58% > 50%)	open
calls during 30 s window	CircuitOpenError (fast fail)
30 s elapse, next call	half-open (probe admitted)
probe succeeds	closed (window reset)
probe fails	open (timer restarts)

Rule of thumb. Trip on a failure ratio over a rolling window with a minimum-volume guard, fail fast while open, and admit exactly one half-open probe per reset interval. A count-based "trip after N failures" breaker misfires on both high- and low-traffic dependencies.

Worked example — a bounded bulkhead pool

Detailed explanation. A bulkhead caps how many calls to a given dependency can be in flight at once, so a slow dependency consumes only its own slice of workers. Implement it as a semaphore with a bounded acquire timeout: if no permit is available quickly, the call is rejected fast rather than queuing forever.

Max concurrent. 10 in-flight calls to this dependency.
Acquire timeout. 100 ms to get a permit, else reject.
Effect. Dependency B saturating its bulkhead cannot touch A's or C's permits.

Question. Implement a bulkhead and wrap a dependency call in it.

Input.

Parameter	Value
max_concurrent	10
acquire_timeout_s	0.1
on rejection	raise BulkheadFullError

Code.

import threading

class BulkheadFullError(Exception):
    pass

class Bulkhead:
    """Cap concurrent in-flight calls to one dependency."""
    def __init__(self, max_concurrent=10, acquire_timeout_s=0.1):
        self._sem     = threading.BoundedSemaphore(max_concurrent)
        self._timeout = acquire_timeout_s

    def call(self, fn):
        acquired = self._sem.acquire(timeout=self._timeout)
        if not acquired:
            raise BulkheadFullError("bulkhead full — shedding load fast")
        try:
            return fn()
        finally:
            self._sem.release()


# One bulkhead PER dependency — isolation is the whole point.
BULKHEADS = {
    "source_api": Bulkhead(max_concurrent=20, acquire_timeout_s=0.1),
    "geocoder":   Bulkhead(max_concurrent=5,  acquire_timeout_s=0.05),
    "warehouse":  Bulkhead(max_concurrent=10, acquire_timeout_s=0.2),
}

def call_dependency(name, fn):
    return BULKHEADS[name].call(fn)

Step-by-step explanation.

Each Bulkhead wraps a BoundedSemaphore sized to the maximum concurrent calls that dependency should ever have in flight. The semaphore is the hard cap on concurrency for that one dependency.
acquire(timeout=...) tries to take a permit but waits at most acquire_timeout_s. If the bulkhead is full (all permits held by slow in-flight calls), the acquire fails fast and raises BulkheadFullError instead of queuing indefinitely — fast rejection is the point.
The try/finally guarantees the permit is released even if fn() raises, so a failing dependency never leaks permits and permanently shrinks its own pool.
Crucially, there is one bulkhead per dependency. The geocoder gets only 5 permits; even if all five are stuck on a slow geocoder, the source API's 20 permits and the warehouse's 10 are untouched — those dependencies keep flowing. This is the isolation that prevents a cascade.
Sizing is a capacity decision: set each bulkhead to the concurrency the dependency can actually handle (or that you're willing to spend on it), not to the worker count. A too-large bulkhead defeats the isolation; a too-small one throttles healthy traffic.

Output.

Dependency	Permits	State when geocoder is slow
source_api	20	flowing normally
geocoder	5	full → BulkheadFullError (shed)
warehouse	10	flowing normally
worker pool overall	—	not starved; cascade prevented

Rule of thumb. Give every dependency its own bounded bulkhead sized to what that dependency can handle, and reject fast when it's full. Shared pools let one slow dependency consume all capacity; per-dependency bulkheads contain the damage to one compartment.

Worked example — breaker plus fallback for a degraded path

Detailed explanation. An open breaker fails fast — but "fail" doesn't have to mean "error to the caller." Where a degraded result is acceptable, attach a fallback so the pipeline stays available while the dependency recovers. Walk through a geocoder call that falls back to "no geocode, flag for backfill" when the breaker is open.

Primary. Call the geocoder through the breaker.
Fallback. On CircuitOpenError (or a call failure), return a record marked geocoded=False for later backfill.
Effect. Records keep flowing; geocoding is filled in later.

Question. Wrap the geocoder in a breaker with a backfill fallback.

Input.

Component	Value
Dependency	geocoder
Breaker	50% / 20-window / 30 s reset
Fallback	geocoded=False, needs_backfill=True

Code.

GEOCODER_BREAKER = CircuitBreaker(failure_threshold=0.5, window_size=20,
                                  min_volume=10, reset_timeout_s=30.0)

def geocode_with_fallback(client, record):
    def primary():
        r = client.post("/geocode", json={"addr": record["addr"]},
                        timeout=httpx.Timeout(connect=1.0, read=3.0, timeout=4.0))
        r.raise_for_status()
        return {**record, **r.json(), "geocoded": True, "needs_backfill": False}

    try:
        return GEOCODER_BREAKER.call(primary)
    except CircuitOpenError:
        # Breaker open — skip fast, degrade gracefully.
        return {**record, "geocoded": False, "needs_backfill": True}
    except Exception:
        # Call failed (already recorded by the breaker) — degrade too.
        return {**record, "geocoded": False, "needs_backfill": True}

Step-by-step explanation.

The primary() closure is the real geocoder call, wrapped with tiered timeouts so a slow geocoder becomes a fast failure the breaker can count.
GEOCODER_BREAKER.call(primary) runs the call through the breaker. While closed, it passes through and records the outcome; once the failure ratio trips it open, subsequent calls raise CircuitOpenError in microseconds without touching the geocoder.
On CircuitOpenError, the fallback returns the record marked geocoded=False, needs_backfill=True. The pipeline keeps moving — records are loaded without geocoding and picked up by a later backfill job — instead of stalling on a dead dependency.
A raw call failure (breaker still closed but this call errored) is also degraded to the fallback, so the caller has one consistent contract: it always gets a record back, geocoded or flagged.
The net effect is graceful degradation: during a geocoder outage the pipeline's throughput is unaffected, only the completeness of geocoding is temporarily reduced — exactly the trade a resilient pipeline should make under stress.

Output.

Geocoder state	Breaker	Result
Healthy	closed	geocoded=True
Failing (below trip)	closed	this record degraded; failure recorded
Broadly down	open	geocoded=False, fast (no call)
Recovering	half-open probe	one probe; closes on success

Rule of thumb. Pair a circuit breaker with a fallback wherever a degraded result is acceptable. The breaker sheds load off the sick dependency; the fallback keeps the pipeline available — throughput preserved, completeness backfilled later.

Systems interview question on circuit breakers and bulkheads

A senior interviewer might ask: "Your ingestion worker pool calls three dependencies — a source API, a geocoder, and the warehouse — all sharing one HTTP connection pool. Last week the geocoder had a brownout: its calls went from 50 ms to 8 seconds, the shared pool filled with stuck geocoder calls, and the entire worker fleet stalled — even source-API and warehouse calls couldn't get a connection. Design the fix so one sick dependency can never again take down the whole worker."

Solution Using per-dependency bulkheads plus a circuit breaker with fallback

# Per-dependency isolation: each dependency gets its own bulkhead AND breaker.
BULKHEADS = {
    "source_api": Bulkhead(max_concurrent=20, acquire_timeout_s=0.1),
    "geocoder":   Bulkhead(max_concurrent=5,  acquire_timeout_s=0.05),
    "warehouse":  Bulkhead(max_concurrent=10, acquire_timeout_s=0.2),
}
BREAKERS = {
    "source_api": CircuitBreaker(0.5, 20, 10, 30.0),
    "geocoder":   CircuitBreaker(0.5, 20, 10, 30.0),
    "warehouse":  CircuitBreaker(0.6, 20, 10, 20.0),
}

def guarded_call(name, fn, fallback=None):
    """Bulkhead (outer) → circuit breaker (inner) → the call."""
    def breakered():
        return BREAKERS[name].call(fn)
    try:
        return BULKHEADS[name].call(breakered)
    except (BulkheadFullError, CircuitOpenError):
        if fallback is not None:
            return fallback()
        raise

def process(client, record):
    raw = guarded_call("source_api",
                       lambda: _get(client, f"/source/{record}"))
    geo = guarded_call("geocoder",
                       lambda: _post(client, "/geocode", {"addr": raw["addr"]}),
                       fallback=lambda: {"geocoded": False, "needs_backfill": True})
    return guarded_call("warehouse",
                        lambda: _post(client, "/load", {**raw, **geo}))

Step-by-step trace.

Layer	Component	Effect during geocoder brownout
Bulkhead (per dep)	geocoder capped at 5 permits	stuck calls consume only 5 slots, not the fleet
Circuit breaker (per dep)	geocoder trips open at 50%	calls fail fast in µs after the trip
Fallback	geocoded=False + backfill flag	records keep flowing
source_api bulkhead	20 permits, untouched	source calls unaffected
warehouse bulkhead	10 permits, untouched	warehouse loads unaffected
worker fleet	not starved	no cascade; throughput preserved

After the fix, the geocoder brownout is contained: its bulkhead caps stuck calls at 5 in-flight, its breaker trips open within a rolling window so further calls fail in microseconds, and the fallback flags records for backfill. The source-API and warehouse dependencies keep their own permits and breakers and flow normally. The single-dependency brownout that previously stalled the whole fleet now degrades only the geocoding completeness for a few minutes.

Output:

Metric	Before (shared pool)	After (bulkhead + breaker)
Blast radius of one slow dep	entire worker fleet	one dependency's slice
Geocoder brownout effect	full stall	geocode degraded, rest flows
Time wasted on dead dep	8 s per call × many	µs fast-fail after trip
Recovery	manual restart	automatic via half-open probe
Throughput during incident	~0	near-normal (minus geocode)

Why this works — concept by concept:

Per-dependency bulkhead — each dependency's bounded semaphore means stuck geocoder calls can occupy at most 5 slots; the source API's 20 and the warehouse's 10 are physically separate, so one leak floods one compartment only.
Per-dependency circuit breaker — the geocoder's breaker trips on its own failure ratio and fails its calls fast, converting 8-second hangs into microsecond rejections that free workers immediately.
Bulkhead outer, breaker inner — the bulkhead admits the call into the dependency's capacity slice first; the breaker then decides whether to attempt or fast-fail. This ordering keeps rejected calls from ever consuming a permit's worth of time.
Fallback for graceful degradation — where a degraded result is acceptable (geocoding), the fallback keeps records flowing; where it isn't (warehouse load), the error propagates. Availability is chosen per dependency, deliberately.
Cost — one semaphore acquire and one breaker state-check per call, both O(1); a small amount of per-dependency config. Compared to a shared pool with no isolation, this converts an unbounded cascade into a bounded, self-recovering degradation — the exact property that keeps one sick dependency from taking down the fleet.

Python
Topic — defensive-coding
Defensive-coding problems on circuit breakers

Practice →

Python Topic — exception-handling Exception-handling problems on fallbacks and fail-fast

Practice →

5. Composing resilience in a pipeline

Nest the primitives in the right order — bulkhead → circuit breaker → timeout → retry → idempotent call — or they fight each other

The mental model in one line: the four resilience primitives compose in a strict nesting order — the bulkhead is outermost (admit into the dependency's capacity slice), then the circuit breaker (fail fast if the dependency is broadly sick), then the timeout (bound this attempt), then the retry loop (re-attempt transient failures within the deadline), wrapping the innermost idempotent call — and getting the order wrong makes the primitives undermine each other, e.g. retrying outside the breaker so retries keep a tripped breaker from ever resetting. Composition is where junior implementations fall apart: each primitive is correct in isolation but wired in the wrong order, so they cancel out.

The nesting order and why each layer sits where it does.

Bulkhead (outermost). Admission control first: if the dependency's capacity slice is full, reject before spending any breaker or retry work. A rejected call should cost almost nothing.
Circuit breaker (next). If the dependency is broadly failing, fail fast before the timeout and retry machinery runs. No point timing out and retrying a dependency the breaker already knows is dead.
Timeout (next). Bound this attempt. The timeout sits inside the breaker so that each timed-out attempt is recorded as a failure the breaker can count toward tripping.
Retry (inner). Re-attempt transient failures, each attempt getting its own timeout, all within the shared deadline. Retry sits inside the breaker so a tripped breaker stops retries — and inside the deadline so retries never overrun the SLA.
Idempotent call (innermost). The actual operation, which must be idempotent (or carry an idempotency key) for any of the above to be safe.

Why retry is inside the breaker, not outside.

Retry outside the breaker. If retries wrap the breaker, then when the breaker is open, the retry loop keeps re-invoking it — each invocation immediately raises CircuitOpenError, the retry catches it, waits, and tries again, hammering a breaker that is trying to stay open to let the dependency rest. The retries prevent the reset from ever helping.
Retry inside the breaker. With retries inside, a tripped breaker raises once and the caller sees the open circuit; retries only happen while the breaker is closed and calls are actually reaching the dependency. This is the correct coupling.

Why retry is inside the deadline, outside the timeout.

Inside the deadline. The retry loop must read the shared deadline (section 3) so its total time — attempts plus backoff sleeps — fits the SLA. Retries are opportunistic within the budget.
Outside the (per-call) timeout. Each individual attempt gets its own timeout; the retry loop wraps those attempts. So "timeout" bounds one attempt and "retry" governs how many attempts — the timeout is inside the retry loop, the deadline is outside it.

Observability — one metric per layer.

Retry metrics. Retry count, retry-budget-exhausted count, final outcome (success-after-retry vs exhausted).
Timeout metrics. Per-tier timeout counts (connect/read/total), deadline-exceeded count.
Breaker metrics. State (closed/open/half-open) as a gauge, trip count, half-open probe outcomes.
Bulkhead metrics. In-flight gauge, rejection count, queue/acquire wait.
The rule. Every layer must emit a metric, because when an incident happens you need to know which layer fired. A resilient stack with no observability is a black box during the exact moment you need to see inside it.

Failure injection — prove it works before production does.

Why. Resilience code runs on the unhappy path, which normal testing rarely exercises. Untested resilience is often subtly broken (a swallowed exception, a wrong nesting order) and you find out during the real outage.
How. Inject faults deliberately: a test double that returns 500s, adds latency, or hangs; a chaos step in staging that kills a dependency. Assert the breaker trips, the fallback fires, the deadline holds.
The interview line. "I test the resilience stack with failure injection — I assert the breaker trips at the threshold, retries respect the budget, and the deadline is never exceeded."

Common interview probes on composition.

"In what order do you nest the primitives?" — bulkhead → breaker → timeout → retry → idempotent call.
"Why is retry inside the breaker?" — so a tripped breaker stops retries instead of being hammered.
"How do retries and the deadline relate?" — retries live inside the deadline; the deadline caps total time.
"How do you know the resilience works?" — failure injection + a metric per layer.

Worked example — the full resilience decorator stack

Detailed explanation. Assemble all four primitives into one composable wrapper in the correct nesting order, around an idempotent call, reading a shared deadline. This is the artifact a senior candidate should be able to produce end-to-end.

Order. bulkhead(breaker(retry(timeout(idempotent call)))).
Deadline. Shared across the retry loop and per-attempt timeout.
Fallback. Optional, applied when the outer layers reject.

Question. Compose bulkhead, breaker, retry, and timeout around an idempotent call with a shared deadline and optional fallback.

Input.

Layer	Responsibility
bulkhead	admission / capacity isolation
breaker	fail fast on broad failure
retry	re-attempt transient, within deadline
timeout	bound each attempt
call	idempotent operation

Code.

def resilient_call(name, make_call, *, budget_s=10.0, max_attempts=4,
                   fallback=None):
    """Compose the four primitives in the correct nesting order."""
    dl = Deadline(budget_s, margin_s=0.1)

    def one_attempt():
        # timeout is applied by make_call using the remaining budget.
        per_call = dl.timeout_for_hop(cap=budget_s)
        return make_call(per_call)                 # innermost: idempotent call + timeout

    def with_retry():
        attempt = 0
        while True:
            try:
                return one_attempt()
            except Exception as exc:
                attempt += 1
                if classify(exc) != FailureClass.TRANSIENT or attempt >= max_attempts:
                    raise
                if not BUDGET.try_spend():
                    raise
                sleep_s = random.uniform(0, min(2.0, 0.5 * 2 ** (attempt - 1)))
                if dl.remaining() - sleep_s <= dl.margin:
                    raise TimeoutError("deadline") from exc
                time.sleep(sleep_s)

    def with_breaker():
        return BREAKERS[name].call(with_retry)     # retry INSIDE the breaker

    try:
        return BULKHEADS[name].call(with_breaker)  # bulkhead OUTERMOST
    except (BulkheadFullError, CircuitOpenError, TimeoutError) as exc:
        if fallback is not None:
            return fallback()
        raise

Step-by-step explanation.

A single Deadline is created once at the top and read by both one_attempt() (for the per-call timeout) and with_retry() (for the backoff budget check), so every layer agrees on the remaining time.
one_attempt() is the innermost layer: it derives the per-call timeout from the deadline and invokes the idempotent make_call. Timeout wraps the call directly.
with_retry() wraps one_attempt — retries transient failures, checks the retry budget, and never sleeps past the deadline. This is the retry layer, sitting outside the per-call timeout but inside the breaker.
with_breaker() wraps with_retry — so the breaker sees the net result of the retry loop, and, crucially, when the breaker is open it raises before any retry runs. Retries are inside the breaker, as required.
BULKHEADS[name].call(with_breaker) is the outermost layer: admission control first. If any outer layer rejects (bulkhead full, breaker open, deadline exceeded), the optional fallback fires. The nesting reads inside-out exactly as bulkhead → breaker → timeout → retry → call.

Output.

Condition	Which layer acts	Result
Dependency healthy	call	success
Transient blip	retry (inside breaker)	success after retry
Attempt hangs	timeout → retry	fast retry
Dependency broadly down	breaker	fast-fail → fallback
Capacity slice full	bulkhead	fast-reject → fallback
Budget spent	deadline	abort → fallback

Rule of thumb. Compose inside-out: idempotent call, wrapped by timeout, wrapped by retry, wrapped by breaker, wrapped by bulkhead. Reading the code from the innermost function outward should spell the nesting order — if it doesn't, the layers will fight.

Worked example — the nesting-order trace under a brownout

Detailed explanation. To cement why the order matters, trace one call through the stack during a dependency brownout, then trace the same call with retry mistakenly placed outside the breaker to show the failure. This contrast is the interview's favorite "gotcha."

Scenario. The dependency has been failing for 40 seconds; the breaker is open.
Correct order. Retry inside breaker.
Wrong order. Retry outside breaker.

Question. Trace a call under both orderings and show how the wrong order defeats the breaker.

Input.

Setup	Value
Breaker state	open (tripped 40 s ago, 30 s reset)
Retry policy	4 attempts, backoff
Dependency	still down

Code.

CORRECT: bulkhead( breaker( retry( timeout(call) ) ) )
  1. bulkhead: permit acquired (dependency idle, plenty free)
  2. breaker.call(): state == open, reset not elapsed
        -> raises CircuitOpenError immediately
  3. retry NEVER runs (it's inside the breaker)
  4. caller catches CircuitOpenError -> fallback
  => 1 fast rejection, dependency untouched, breaker keeps resting

WRONG: retry( bulkhead( breaker( timeout(call) ) ) )
  1. attempt 1: breaker open -> CircuitOpenError
  2. retry catches it, sleeps, attempt 2: breaker open -> CircuitOpenError
  3. retry catches it, sleeps, attempt 3: ... attempt 4: ...
  => 4 invocations hammering a breaker that is TRYING to stay open;
     retries burn the deadline re-poking a dependency meant to rest

Step-by-step explanation.

In the correct order, the breaker sits outside the retry loop. When the breaker is open, breaker.call() raises CircuitOpenError once, the retry loop never executes (it is the argument to breaker.call), and the caller degrades to a fallback. One fast rejection; the dependency is left alone to recover.
In the wrong order, retry sits outside the breaker. Each retry attempt calls into the breaker, which is open and raises CircuitOpenError; the retry loop treats that as a failure to retry, sleeps, and tries again — up to four times.
The wrong order defeats the breaker's entire purpose: the breaker trips open precisely to stop traffic and let the dependency rest, but the outer retry loop keeps re-invoking it, spending backoff sleeps and deadline budget on calls guaranteed to fast-fail.
There is a subtler harm: if CircuitOpenError were (incorrectly) classified as transient, the retries would also count as breaker-adjacent churn and delay the caller's fallback, worsening latency during the exact incident the breaker exists to handle.
The lesson generalizes: the breaker must be able to suppress retries, which is only possible when it wraps them. Any ordering where retry is outside the breaker reintroduces the storm the breaker was added to prevent.

Output.

Ordering	Invocations while open	Deadline burned	Breaker helped?
Retry inside breaker (correct)	1 fast-fail	~0	yes — dependency rests
Retry outside breaker (wrong)	4 + backoff sleeps	most of it	no — breaker hammered

Rule of thumb. Retry must live inside the circuit breaker. If a tripped breaker doesn't stop your retries, your nesting order is wrong — and you've rebuilt the retry storm the breaker was supposed to prevent.

Worked example — a failure-injection test for the stack

Detailed explanation. Resilience code lives on the unhappy path, so it must be tested with injected faults. Write a test that drives the composed stack against a fault-injecting dependency and asserts the breaker trips, the fallback fires, and the deadline holds. This is what "I test my resilience" concretely means.

Fault double. A fake dependency that returns 503s, then recovers.
Assertions. Breaker trips within the window; fallback returns; total time ≤ deadline.

Question. Write a failure-injection test that proves the stack degrades correctly.

Input.

Injected behavior	Assertion
15 consecutive 503s	breaker trips to open
calls while open	fallback returned, fast
recovery after reset	half-open probe closes breaker

Code.

import time

class FaultyDependency:
    """Injects failures, then recovers — for testing the resilience stack."""
    def __init__(self, fail_first=15):
        self.calls = 0
        self.fail_first = fail_first
    def __call__(self, timeout):
        self.calls += 1
        if self.calls <= self.fail_first:
            raise httpx.HTTPStatusError("503", request=None,
                                        response=httpx.Response(503))
        return {"ok": True, "call": self.calls}

def test_stack_degrades_and_recovers():
    dep  = FaultyDependency(fail_first=15)
    name = "geocoder"
    BREAKERS[name] = CircuitBreaker(0.5, 20, 10, reset_timeout_s=1.0)

    fallbacks = 0
    start = time.monotonic()
    for _ in range(20):
        try:
            resilient_call(name, dep, budget_s=2.0, max_attempts=2,
                           fallback=lambda: (_ for _ in ()).throw(CircuitOpenError()))
        except CircuitOpenError:
            fallbacks += 1

    # 1. The breaker must have tripped open under sustained 503s.
    assert BREAKERS[name].state in ("open", "half-open")
    # 2. Once open, calls fast-fail into the fallback.
    assert fallbacks > 0
    # 3. No run exceeded its 2s deadline.
    assert time.monotonic() - start < 20 * 2.0

    # 4. After the reset timeout, a probe should close the breaker.
    time.sleep(1.1)
    result = resilient_call(name, dep, budget_s=2.0, max_attempts=2)
    assert result["ok"] is True
    assert BREAKERS[name].state == "closed"

Step-by-step explanation.

FaultyDependency is a deterministic fault double: the first 15 calls raise a 503, then every subsequent call succeeds. This lets the test drive the breaker across all three states on demand.
The loop makes 20 calls through the full resilient_call stack. The early calls fail and feed the breaker's window; once the failure ratio crosses 50% with enough volume, the breaker trips open and later calls fast-fail into the fallback.
Assertion 1 checks the breaker actually tripped (state open or half-open) — proving the trip condition works under sustained failure, not just in theory.
Assertion 3 checks the deadline held: 20 calls each bounded by a 2-second budget must finish well under 40 seconds, proving retries never overran the SLA even while everything was failing.
After sleeping past the 1-second reset timeout, one more call exercises the half-open probe against the now-recovered dependency; assertion 4 confirms the probe succeeds and the breaker closes — proving automatic recovery. The test covers trip, fast-fail, deadline, and recovery: the whole state machine.

Output.

Assertion	Proves
breaker state open/half-open	trip condition fires under load
fallbacks > 0	open breaker fast-fails to fallback
total time < N × budget	deadline holds during failure
final state closed + ok	half-open probe recovers automatically

Rule of thumb. Ship a failure-injection test with every resilience stack. If you have not watched your breaker trip, your fallback fire, and your deadline hold in a test, you do not know they work — and the first real outage is a terrible place to find out.

Systems interview question on composing resilience

A senior interviewer might ask: "Design an end-to-end resilient ingestion task for an Airflow DAG that pulls from a partner API, enriches via a geocoder, and loads to Snowflake. It must survive transient blips, dependency brownouts, and hangs, finish within a 60-second task SLA, never cause a retry storm, and be observable enough that on-call can tell which failure mode fired. Walk me through the full composition and how you'd verify it."

Solution Using the full composed stack with shared deadline, per-dependency isolation, and per-layer metrics

def ingest_record(client, record, metrics, budget_s=60.0):
    """End-to-end resilient ingestion: bulkhead > breaker > timeout > retry > idempotent call."""
    dl = Deadline(budget_s, margin_s=0.5)

    def make(name, method, url, body, cap, fallback=None):
        def one(per_call):
            r = client.request(method, url, json=body,
                               timeout=httpx.Timeout(connect=1.0, read=per_call,
                                                     write=per_call, pool=1.0,
                                                     timeout=per_call))
            r.raise_for_status()
            return r.json() if r.content else {}

        def with_retry():
            attempt = 0
            while True:
                try:
                    return one(dl.timeout_for_hop(cap=cap))
                except Exception as exc:
                    attempt += 1
                    metrics.incr(f"{name}.attempt")
                    if classify(exc) != FailureClass.TRANSIENT or attempt >= 3:
                        raise
                    if not BUDGET.try_spend():
                        metrics.incr(f"{name}.budget_exhausted"); raise
                    sleep_s = random.uniform(0, min(2.0, 0.5 * 2 ** (attempt - 1)))
                    if dl.remaining() - sleep_s <= dl.margin:
                        metrics.incr(f"{name}.deadline_exceeded"); raise TimeoutError from exc
                    time.sleep(sleep_s)

        def with_breaker():
            br = BREAKERS[name]
            metrics.gauge(f"{name}.breaker_state", {"closed":0,"half-open":1,"open":2}[br.state])
            return br.call(with_retry)

        try:
            return BULKHEADS[name].call(with_breaker)
        except (BulkheadFullError, CircuitOpenError, TimeoutError) as exc:
            metrics.incr(f"{name}.shed.{type(exc).__name__}")
            if fallback is not None:
                return fallback()
            raise

    raw = make("source_api", "GET",  f"/source/{record}", None, cap=20.0)
    geo = make("geocoder",   "POST", "/geocode", {"addr": raw["addr"]}, cap=10.0,
               fallback=lambda: {"geocoded": False, "needs_backfill": True})
    # Idempotent load: MERGE/COPY keyed on record id — safe to retry.
    make("warehouse", "POST", "/load", {**raw, **geo, "idem_key": record}, cap=15.0)
    metrics.gauge("record.ms_left", dl.remaining() * 1000)
    return {"record": record, "geocoded": geo.get("geocoded", False)}

Step-by-step trace.

Layer	Component	Observability signal
Deadline	one 60 s monotonic budget	`record.ms_left` gauge
Bulkhead	per-dependency semaphore	`<dep>.shed.BulkheadFullError`
Circuit breaker	per-dependency state machine	`<dep>.breaker_state` gauge (0/1/2)
Timeout	tiered, derived from remaining budget	`<dep>.deadline_exceeded`
Retry	transient-only, budget + deadline bounded	`<dep>.attempt`, `<dep>.budget_exhausted`
Idempotent load	MERGE/COPY keyed on record id	safe to retry

After deployment, each record carries one 60-second deadline down all three hops. The source API and warehouse use their own bulkheads and breakers; the geocoder degrades to a backfill flag when its breaker opens. Every layer emits a metric, so on-call sees exactly which mode fired — a geocoder.breaker_state = 2 gauge plus rising geocoder.shed counts says "geocoder brownout, breaker open, degrading gracefully" at a glance. A failure-injection test in CI proves the trip, fallback, and deadline before the code ever ships.

Output:

Metric	Value / behavior
Worst-case task time	≤ 60 s (shared deadline)
Retry storm risk	none (budget + backoff + jitter)
One-dependency brownout	contained (bulkhead + breaker + fallback)
Hang handling	tiered timeouts → fast retry
Diagnosability	one metric per layer per dependency
Pre-production confidence	failure-injection test in CI

Why this works — concept by concept:

One shared deadline — a single 60-second monotonic budget threads through every hop and retry, so the composed stack respects the task SLA as a hard bound rather than a hopeful sum of per-call timeouts.
Correct nesting order — bulkhead → breaker → timeout → retry → idempotent call means admission control runs first, a sick dependency fast-fails before retry work, and retries sit inside the breaker so a trip actually suppresses them.
Per-dependency isolation — separate bulkheads and breakers keep a brownout in one dependency from starving the others; the fallback chooses availability over completeness where that trade is acceptable (geocoding) and not where it isn't (the load).
A metric per layer — retry counts, breaker-state gauges, shed counts, and deadline-exceeded counters make the stack a glass box during an incident, so on-call diagnoses the failure mode in seconds instead of guessing.
Cost — O(hops × attempts) calls, hard-bounded by 60 s, plus O(1) semaphore, breaker, and budget checks per call and a handful of metric emissions. Compared to an unguarded task that hangs, storms, and cascades, this is the difference between a pipeline that pages on-call at 2 a.m. and one that degrades gracefully and self-recovers — verified by failure injection before it ever runs in production.

ETL
Topic — etl
ETL problems on end-to-end resilient ingestion

Practice →

Python
Topic — defensive-coding
Defensive-coding problems on composing resilience

Practice →

Cheat sheet — resilience recipes

Failure classification first. Split every failure into transient (429/503/reset → retry with backoff), persistent (401/400/404/schema → fail fast + alert), or poison (un-parseable record → dead-letter). Build a classify(exc) function; wire retries to the transient class only. Never retry "any exception."
Exponential backoff with full jitter. ceiling = min(cap, base * 2**(attempt-1)); sleep = random.uniform(0, ceiling). Grows the gap and scatters the timing so independent clients de-synchronize. Defaults: base 0.5 s, cap 20–30 s, max 4–6 attempts. Jitter is mandatory, not optional.
Decorrelated jitter (backlog drain). sleep = min(cap, random.uniform(base, prev_sleep * 3)). Climbs faster than full jitter while staying de-correlated — AWS's recommended variant for draining a backlog quickly after an outage.
Retry budget token bucket. Shared bucket refilling at ~10–20% of success traffic, debited 1 token per retry; when empty, suppress retries and fail fast. This is the actual retry-storm defense — per-call max_attempts bounds one call, the budget bounds the fleet.
Idempotency is the retry precondition. Only retry idempotent calls (GET, upsert, keyed COPY). For non-idempotent ones, generate an idempotency key once per logical op, reuse it across retries, and dedupe server-side with INSERT ... ON CONFLICT (idem_key) DO NOTHING RETURNING result.
Timeout tiers — always all three. httpx.Timeout(connect=1.0, read=5.0, timeout=8.0) (+ pool). Connect short (unhealthy endpoint fails fast), read tuned to the operation, total as the hard ceiling. Never make any network call without an explicit timeout — the missing timeout is a latent hang.
Deadline budget propagation. Set one absolute monotonic() deadline at the entry point; each hop uses min(remaining − margin, cap) as its timeout and passes the reduced budget down. A downstream timeout must be < the remaining budget. Retries live inside the deadline, never additive to it.
Circuit breaker state machine. Closed → (failure ratio > threshold over a rolling window, min-volume guarded) → Open → (after reset timeout) → Half-open → (probe succeeds → Closed / probe fails → Open). Trip on a ratio, not a raw count; admit exactly one half-open probe.
Bulkhead pool sizing. One bounded semaphore per dependency, sized to what that dependency can handle (not the worker count), with a short acquire timeout that rejects fast when full. Isolation means one slow dependency floods one compartment, never the whole worker fleet.
Nesting order mnemonic. bulkhead → breaker → timeout → retry → idempotent call (outer to inner). Retry lives inside the breaker (so a trip suppresses retries) and inside the deadline (so retries fit the SLA); the per-call timeout is inside the retry loop.
Fallbacks and load shedding. When a breaker opens or a bulkhead rejects, return a degraded-but-useful result where acceptable (cache, default, skip-and-backfill); otherwise fail fast. An open breaker is load shedding — it sheds load off the sick dependency so it can recover.
Observability + failure injection. Emit one metric per layer per dependency — retry count, budget-exhausted, breaker-state gauge (0/1/2), bulkhead rejections, deadline-exceeded. Prove the stack with a fault-injecting test double: assert the breaker trips, the fallback fires, and the deadline holds before production does.

Frequently asked questions

What are retries, timeouts, and circuit breakers in one sentence?

They are the three core resilience controls for remote calls in a data pipeline: a retry re-attempts a transient failure (spaced out by exponential backoff and jitter, gated on idempotency), a timeout bounds how long any single call may take so a hung dependency can't consume a worker forever, and a circuit breaker stops calling a dependency that is persistently failing so you fail fast instead of piling up slow timeouts. They solve different problems — retries recover from blips, timeouts cap latency, breakers stop cascades — and a resilient pipeline composes all three (plus bulkheads) around every idempotent call. Every senior data-engineering interview probes them because they are the load-bearing patterns for any pipeline that touches a network.

Why add jitter to exponential backoff?

Because exponential backoff alone still lets independent clients retry in lockstep. If a dependency goes down and a hundred workers all fail at the same instant, plain exponential backoff has them all wait 1 s, then all wait 2 s, then all wait 4 s — the retries arrive in synchronized waves that hammer the dependency exactly when it's trying to recover (the "thundering herd"). Jitter adds randomness to each wait so the waves smear out into a smooth trickle. Full jitter (random(0, ceiling)) maximally de-correlates clients and is the common default; decorrelated jitter (random(base, prev*3)) drains a backlog faster. Without jitter, "adding backoff" only spaces the herds apart in time; it doesn't break the herd itself.

What is a timeout budget / deadline propagation?

A timeout budget (or deadline budget) is a single end-to-end time limit set at a pipeline's entry point and shared across every downstream hop, rather than each hop having an independent timeout. The entry point records an absolute deadline (e.g. monotonic() + 10 s); each hop computes the remaining budget, uses it (minus a small margin) as its own timeout, and passes the reduced budget to the next hop — this is deadline propagation. The invariant is that a downstream call's timeout must be less than the remaining budget, so the chain fails fast the instant the budget is spent instead of letting three independent 8-second timeouts sum to 24 seconds and blow a 10-second SLA. Retries also read the same deadline, so they fit inside the budget rather than adding to it.

How does a circuit breaker differ from a retry?

A retry re-attempts one call, assuming the failure is transient and the next attempt might succeed. A circuit breaker watches many calls to a dependency and, when the recent failure ratio crosses a threshold, stops attempting that dependency entirely for a cooldown period — it "trips open" and fails calls fast without touching the dependency. The distinction is scope and intent: retries are optimistic ("try again, it might work"); the breaker is pessimistic ("this dependency is broadly down, stop hammering it and let it recover"). They compose — retry inside the breaker — so occasional blips are retried while a sustained outage trips the breaker and short-circuits the retries. Retrying outside the breaker is a classic bug: the retries keep re-poking a breaker that is trying to stay open.

What is a bulkhead and when do I need one?

A bulkhead is per-dependency resource isolation — a bounded pool (connections, threads, or semaphore permits) dedicated to each dependency — named after a ship's sealed compartments that stop one breach from flooding the whole hull. You need one whenever multiple dependencies share a resource pool and one of them can get slow: without isolation, a slow dependency's calls pile up, consume every connection or worker in the shared pool, and starve the healthy dependencies too, turning one component's brownout into a full pipeline stall (a cascading failure). With a bulkhead, each dependency gets its own bounded slice; when one saturates, its calls are rejected fast while the others keep flowing. The classic trigger is exactly the interview scenario: "one slow dependency stalled the entire worker fleet" — the fix is per-dependency bulkheads.

In what order should I nest the resilience primitives?

From outermost to innermost: bulkhead → circuit breaker → timeout → retry → idempotent call. The bulkhead is outermost so admission/capacity control runs first and a rejected call costs almost nothing. The circuit breaker is next so a broadly-failing dependency fast-fails before any timeout or retry work happens. The timeout bounds each individual attempt. The retry loop wraps those attempts — and it must sit inside the breaker (so a tripped breaker suppresses retries instead of being hammered) and inside the shared deadline (so retries fit the SLA rather than adding to it). The innermost call must be idempotent, or carry an idempotency key, or none of the retrying above is safe. Getting the order wrong — most commonly putting retry outside the breaker — makes the primitives undermine each other.

Practice on PipeCode

Drill the defensive-coding practice library → for the retry-policy, timeout, circuit-breaker, and bulkhead problems senior interviewers love.
Rehearse on the exception-handling practice library → for classifying transient vs fatal failures, backoff loops, and fallback paths.
Wire the patterns into real jobs on the ETL practice library → for SLA-bounded multi-hop pipelines, deadline budgets, and end-to-end resilient ingestion.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the nesting-order decision against real graded inputs.

Lock in resilience muscle memory

Docs explain the patterns. PipeCode drills explain the decision — when a naive retry loop becomes a retry storm, when a missing timeout hangs a worker, when a circuit breaker earns its place, and in what order the four primitives must nest around an idempotent call. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice defensive-coding problems →
Practice ETL problems →

DEV Community: Gowtham Potureddi

AWS Lambda for ETL: Event-Driven Data Pipelines

1. When Lambda fits ETL (and when it doesn't)

Event-driven, stateless, fifteen-minute compute — the fit is glue and transform, not heavy batch

Worked example — the fit / no-fit decision table

Worked example — the cost crossover versus a warm cluster

Worked example — the "10 GB file" streaming trap

Data engineering interview question on Lambda ETL fit

Solution Using a per-object fit split with a size-based routing function

2. Event sources & triggers — S3 / SQS / Kinesis

S3 pushes, SQS and Kinesis are polled — the event source mapping is the knob you tune

Worked example — S3 object-created trigger to a curated Parquet copy

Worked example — SQS batch consumer with partial batch response

Worked example — Kinesis shard consumer with ordering and parallelization

Data engineering interview question on event source mappings

Solution Using ReportBatchItemFailures with a tuned visibility timeout and DLQ redrive

3. Concurrency, memory & cold starts

One sandbox per event — concurrency, memory, and cold starts are the three dials that decide throughput, latency, and blast radius

Worked example — reserved concurrency to protect a downstream database

Worked example — provisioned concurrency for a latency-sensitive step

Worked example — memory power-tuning to minimise cost

Data engineering interview question on concurrency and cold starts

Solution Using reserved concurrency, connection reuse, and SQS back-pressure

4. Error handling — DLQ, retries, idempotency

At-least-once means retries are guaranteed — so handlers must be idempotent and poison pills must have somewhere to go

Worked example — async S3 invocation with retries and an on-failure destination

Worked example — idempotent S3-to-warehouse upsert with a dedupe table

Worked example — Kinesis bisect-on-error to isolate a poison record

Data engineering interview question on idempotency and DLQs

Solution Using an idempotency table with a conditional write plus a DLQ redrive

5. Patterns & limits — fan-out, Step Functions handoff

Fan-out to scale horizontally, hand off at the fifteen-minute wall — the two moves that take Lambda past its limits

Worked example — SNS fan-out to independent consumers

Worked example — Step Functions handoff for a multi-hour backfill

Worked example — Distributed Map over an S3 manifest

Data engineering interview question on scaling past Lambda limits

Solution Using a Step Functions Distributed Map handoff with bounded concurrency

Cheat sheet — AWS Lambda ETL recipes

Frequently asked questions

Is AWS Lambda good for ETL?

What is the 15-minute Lambda limit and how do I work around it?

How do S3 triggers work with Lambda?

What is the difference between reserved and provisioned concurrency?

How do I make a Lambda ETL job idempotent?

When should I use Step Functions instead of Lambda?

Practice on PipeCode

Lock in event-driven ETL muscle memory

Redis for Data Engineers: Beyond Caching

1. Redis core data structures — strings, hashes, lists, sets, sorted sets

The value type you pick is the optimization — five shapes, each with O(1) or O(log N) primitives

Worked example — atomic counters with INCR and EXPIRE

Worked example — a hash-backed session / feature-store row

Worked example — a list as a capped "last N events" buffer

Data engineering interview question on choosing the right structure

Solution Using a hash-per-session with a sliding TTL and atomic field counters

2. Redis Streams — the append-only log with consumer groups

redis streams turn Redis into a durable, replayable log with at-least-once consumer groups — the ingestion buffer you already have

Worked example — ingesting events with XADD and a capped stream

Worked example — a consumer-group worker loop with XREADGROUP and XACK

Worked example — recovering a dead consumer with XAUTOCLAIM

Data engineering interview question on Redis Streams

Solution Using a capped stream + consumer group + XAUTOCLAIM + idempotent sink

3. Sorted sets — leaderboards, time-series windows, rate limiting

sorted sets are the Swiss-army structure — one score-ordered set solves leaderboards, time-series windows, and rate limiting

Worked example — a top-N leaderboard with ZINCRBY and ZREVRANGE

Worked example — a precise sliding-window rate limiter

Worked example — time-bucketed metrics with score = timestamp

Data engineering interview question on sorted sets

Solution Using a sorted-set sliding window driven by one atomic Lua script

4. Probabilistic structures — HyperLogLog and Bloom filters

hyperloglog and Bloom filters trade exactness for constant memory — count billions of uniques in 12 KB, test membership in kilobytes

Worked example — daily unique visitors with HyperLogLog

Worked example — rolling daily HLLs into weekly/monthly with PFMERGE

Worked example — a Bloom filter as a dedup / cache-miss pre-filter

Data engineering interview question on probabilistic structures

Solution Using one HyperLogLog per day and PFMERGE rollups

5. Persistence, patterns and pitfalls

Redis is durable if you configure it — RDB, AOF, eviction, and atomicity decide whether your data survives a restart

Worked example — choosing RDB vs AOF for a durable buffer

Worked example — cache-aside with stampede protection

Worked example — atomic counters with `INCR` and `EXPIRE`

`redis streams` turn Redis into a durable, replayable log with at-least-once consumer groups — the ingestion buffer you already have

Worked example — ingesting events with `XADD` and a capped stream

Worked example — a consumer-group worker loop with `XREADGROUP` and `XACK`

Worked example — recovering a dead consumer with `XAUTOCLAIM`

`sorted sets` are the Swiss-army structure — one score-ordered set solves leaderboards, time-series windows, and rate limiting

Worked example — a top-N leaderboard with `ZINCRBY` and `ZREVRANGE`

`hyperloglog` and Bloom filters trade exactness for constant memory — count billions of uniques in 12 KB, test membership in kilobytes

Worked example — rolling daily HLLs into weekly/monthly with `PFMERGE`

Worked example — `_id`, `ObjectId`, and BSON types you will meet in the warehouse

The pipeline is an ordered conveyor — `$match` filters, `$group` folds, `$project` reshapes, and order is the program

Worked example — revenue per status with `$match` + `$group`

Worked example — `$project` vs `$addFields` and computed fields

`$lookup` is a left outer join into an array; `$unwind` flattens that array into one document per element

Worked example — basic `$lookup` from orders to customers

Worked example — `$unwind` an embedded array then group back

Worked example — correlated sub-pipeline `$lookup`

Worked example — COLLSCAN to IXSCAN with `explain()`

`$merge` upserts pipeline output incrementally; `$out` replaces a collection; sharding partitions data by a shard key

Worked example — incremental daily rollup with `$merge`

Worked example — `$out` full snapshot for export