Azure Event Hubs & Functions: Event-Driven Data on Azure

azure event hubs is the managed ingestion front door that decides whether a telemetry stream, a clickstream, or a fleet of IoT devices lands in your lake as an ordered, replayable log — or as an unrecoverable firehose that drops events the moment a consumer falls behind. It is a partitioned, append-only event broker: every producer writes into one of a fixed set of partitions, every event is stamped with an immutable offset, and every downstream reader subscribes through a consumer group that tracks its own position independently of every other reader. Get the partition count and the throughput sizing right on day one and the same hub carries you from a thousand events a second to a million; get them wrong and you inherit a rebuild, because a partition count is frozen at creation and every downstream contract hard-codes assumptions about ordering and parallelism.

This guide is the walkthrough you wished existed the first time an interviewer asked "how would you ingest a million events a second on Azure and fan them out to a warehouse and a real-time handler at the same time?" It moves in layers: the partition-and-consumer-group model that governs ordering and parallelism, the producer side where a partition key pins related events together and throughput units cap your ingress and egress, the automatic event hubs capture feature that lands raw Avro into Blob or ADLS with zero code, the azure functions trigger that turns the stream into an event-driven compute pipeline with durable checkpointing, and the patterns layer where a Kafka endpoint lets existing Kafka clients connect unchanged. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library →, rehearse on the event-processing practice library →, and sharpen the real-time axis with the real-time analytics practice library →.

---

On this page

• The Event Hubs model — partitions and consumer groups
• Producers, partition keys and throughput units
• Event Hubs Capture to storage
• Azure Functions triggers and checkpointing
• Patterns and Kafka-protocol compatibility
• Cheat sheet — Azure Event Hubs recipes
• Frequently asked questions
• Practice on PipeCode

---

1. The Event Hubs model — partitions and consumer groups

The namespace → hub → partition → consumer-group hierarchy is the whole mental model — get it and everything else follows

The one-sentence invariant: Azure Event Hubs is a partitioned append-only log where a producer writes an immutable, offset-stamped event into exactly one partition, each partition preserves total order within itself but not across partitions, and every downstream reader consumes through a consumer group that tracks its own independent position — so ordering, parallelism, retention, and delivery semantics are all decided by how many partitions you create and how many consumer groups read them, none of which you can renegotiate cheaply after the fact. The partition count you pick in the first hour becomes the parallelism ceiling you fight for years, because the maximum number of concurrent readers in a single consumer group is exactly the number of partitions, and on the standard tier that number is frozen at creation.

The hierarchy that matters.

• Namespace. The billing and networking boundary. A namespace has a tier (Basic, Standard, Premium, Dedicated), a throughput capacity (throughput units, processing units, or capacity units), a private-endpoint / firewall configuration, and one or more event hubs inside it. Think of it as the "Kafka cluster" analogue.
• Event hub. The topic-level entity — the equivalent of a Kafka topic. A hub has a partition count (fixed at creation on Standard), a message retention window (1–7 days on Standard, longer on Premium/Dedicated), and a set of consumer groups. You send to a hub; you read from a hub via a consumer group.
• Partition. The unit of ordering and parallelism. An append-only sequence of events; each event gets a monotonically increasing offset, a sequenceNumber, and an enqueuedTime. Order is guaranteed within a partition and nowhere else.
• Consumer group. An independent read view over all partitions of the hub. Each consumer group tracks its own checkpoint per partition, so a warehouse loader and a real-time alerting job read the same events at their own pace without interfering. Standard allows up to 20 consumer groups per hub; Basic allows exactly one ($Default).

The 2026 reality — Event Hubs is the default Azure ingestion tier.

• Standard is the workhorse: up to 32 partitions per hub, up to 40 throughput units (with auto-inflate), 1–7 day retention, Capture, and the Kafka endpoint. Most production streaming pipelines start here.
• Premium trades throughput units for processing units (PUs), gives isolated compute, up to 100 partitions per hub, and up to 90-day retention — the tier for noisy-neighbour-sensitive workloads.
• Dedicated is a single-tenant cluster measured in capacity units (CUs), scaling to thousands of partitions and the highest sustained throughput; it is where self-managed Kafka migrations usually land.
• Basic is the toy tier: one consumer group, no Capture, no Kafka endpoint, 1-day retention. Fine for a proof of concept, never for production fan-out.

What the partition guarantees — and what it does not.

• Guarantees. Total order within the partition; immutability (events are never updated or deleted before retention expiry); a stable coordinate system of (partitionId, offset) and (partitionId, sequenceNumber) for replay.
• Does not guarantee. Global order across partitions — two events in different partitions have no defined relative order. If you need "all events for customer 42 in order," you must route them all to the same partition with a partition key.
• Parallelism ceiling. A single consumer group can have at most one active reader per partition (Azure recommends one owner per partition). Sixteen partitions means at most sixteen parallel readers in that consumer group. This is why partition count is your parallelism budget.
• Retention, not storage. Event Hubs is a transient buffer, not a database. Events age out after the retention window whether or not anyone read them. Durable history is the job of Capture (section 3), not the hub.

What the consumer group guarantees.

• Independent offsets. Each consumer group has its own checkpoint per partition; resetting one group's position (to replay) does not touch another group's progress.
• Fan-out. N consumer groups = N independent copies of the stream, each delivered without re-reading storage N times on the producer's dime. This is the cheapest fan-out primitive Azure offers.
• Not a work queue. Within one consumer group, partitions are owned, not load-balanced per message. You do not get competing consumers pulling from a shared backlog the way a queue does; you get partition ownership. Parallelism is coarse-grained at the partition level.

What interviewers listen for.

• Do you say "order is per-partition, not global" in the first sentence when ordering comes up? — required answer.
• Do you name the partition count = parallelism ceiling rule without prompting? — senior signal.
• Do you distinguish consumer group (independent view) from partition (ordering unit) crisply? — required answer.
• Do you flag that partition count is immutable on Standard and must be sized up front? — senior signal.
• Do you describe Event Hubs as a "retention-bounded append-only log" rather than "a queue" or "a database"? — required answer.

Worked example — sizing partitions and consumer groups for a workload

Detailed explanation. The single most useful artifact for an Event Hubs interview is a partition-and-consumer-group sizing model you can derive out loud. Every capacity discussion converges on the same three inputs — peak ingress, per-partition throughput ceiling, and the number of independent downstream consumers — within the first ten minutes. Walk through building it for a hypothetical vehicle-telemetry stream.

• Peak ingress. 400,000 events/sec at peak, ~1 KB each ≈ 400 MB/s.
• Per-partition ceiling. A partition sustains roughly 1 MB/s ingress; treat ~1 MB/s per partition as the planning number.
• Downstream consumers. A warehouse loader, a real-time alerting job, and an ad-hoc analytics team — three independent read paths.
• Ordering need. All events for one vehicle_id must stay ordered.

Question. Compute a partition count and a consumer-group layout for the telemetry hub, and state the parallelism ceiling each consumer sees.

Input.

Requirement	Value
Peak ingress	400 MB/s (400k ev/s × 1 KB)
Per-partition ingress planning ceiling	~1 MB/s
Independent downstream consumers	3 (warehouse, alerting, analytics)
Ordering key	vehicle_id
Tier	Standard (auto-inflate) or Premium

Code.

# Partition + consumer-group sizing model
PEAK_MB_S            = 400      # 400k events/s x 1 KB
PARTITION_MB_S       = 1        # planning ceiling per partition (ingress)
HEADROOM             = 1.5      # 50% headroom for skew + spikes

min_partitions = PEAK_MB_S / PARTITION_MB_S            # 400
sized_partitions = int(min_partitions * HEADROOM)      # 600

print(f"Raw partitions needed:   {min_partitions:.0f}")
print(f"With 50% headroom:       {sized_partitions}")

# Standard caps at 32 partitions/hub -> this workload forces Premium/Dedicated
STANDARD_MAX = 32
if sized_partitions > STANDARD_MAX:
    print("Exceeds Standard (32) -> Premium (<=100/hub) or Dedicated (thousands)")

# Consumer groups = independent read paths (NOT a function of throughput)
consumer_groups = ["cg-warehouse", "cg-alerting", "cg-analytics"]
print(f"Consumer groups: {len(consumer_groups)} -> {consumer_groups}")

# Parallelism ceiling each consumer group sees = partition count
print(f"Max parallel readers per consumer group = {sized_partitions}")

Step-by-step explanation.

1. Peak ingress divided by the ~1 MB/s per-partition planning ceiling gives the raw partition floor — 400 here. This is a floor, not the answer, because real traffic is skewed and spiky.
2. Multiply by a headroom factor (1.5) to absorb hot-partition skew and burst. A hub running at 100% of its partition ceiling has no room for a key that gets 3× the average traffic.
3. 600 partitions blows past the Standard cap of 32, so this workload is a Premium (≤100 partitions/hub) or Dedicated (thousands) decision. Naming that constraint is the senior move — Standard simply cannot carry 400 MB/s in one hub.
4. Consumer groups are orthogonal to throughput. Three independent downstream systems means three consumer groups, each of which sees the full stream. Adding a consumer group does not add ingress capacity; it adds a read view.
5. The parallelism ceiling each consumer group can achieve equals the partition count — with 600 partitions, the warehouse loader can run up to 600 concurrent readers. This is why partition count is the load-bearing number.

Output.

Decision	Value	Reason
Raw partition floor	400	400 MB/s ÷ 1 MB/s/partition
Sized partition count	600	+50% headroom for skew/burst
Tier	Premium or Dedicated	600 > Standard's 32-partition cap
Consumer groups	3	one per independent read path
Parallelism ceiling per group	600	equals partition count

Rule of thumb. Size partitions from peak_ingress / 1 MB/s with 50% headroom, then check the tier's partition cap; size consumer groups from the count of independent downstream systems, never from throughput. Partition count is your parallelism budget — set it once, generously, because you cannot grow it on Standard.

Worked example — offset vs sequence number vs enqueued time

Detailed explanation. Every Event Hubs event carries three positional stamps, and interviewers love to check whether you know which one to use for what. Confusing offset with sequenceNumber is a classic tell that a candidate has only read the marketing page. Walk through all three and their correct uses.

• Offset. A byte-oriented, partition-scoped position marker. Opaque string; you pass it back to resume ("start after this offset"). Not contiguous, not comparable across partitions.
• Sequence number. A monotonically increasing 64-bit integer per partition. Contiguous within a partition; ideal for gap detection ("did I miss any?").
• Enqueued time. The server-side timestamp when the event was accepted. Use it for time-based replay ("give me everything since 09:00") — but never assume it is strictly ordered across partitions.

Question. Given a resume scenario, pick the right positional stamp for (a) resuming a reader, (b) detecting a dropped event, and (c) replaying the last hour.

Input.

Need	Candidate stamp	Correct?
Resume exactly where I stopped	offset	yes
Detect a missing event	sequenceNumber	yes
Replay everything since 09:00	enqueuedTime	yes
Compare order across partitions	any	no — undefined

Code.

# Reading with the Azure SDK — inspecting the three stamps
from azure.eventhub import EventHubConsumerClient

def on_event(partition_context, event):
    print(
        f"partition={partition_context.partition_id} "
        f"offset={event.offset} "
        f"seq={event.sequence_number} "
        f"enqueued={event.enqueued_time.isoformat()}"
    )
    # Gap detection uses the CONTIGUOUS sequence number, not the offset
    last = _last_seq.get(partition_context.partition_id)
    if last is not None and event.sequence_number != last + 1:
        print(f"  GAP: expected {last + 1}, got {event.sequence_number}")
    _last_seq[partition_context.partition_id] = event.sequence_number

    partition_context.update_checkpoint(event)   # persists the OFFSET

_last_seq: dict[str, int] = {}

client = EventHubConsumerClient.from_connection_string(
    conn_str="Endpoint=sb://ns.servicebus.windows.net/;...",
    consumer_group="cg-analytics",
    eventhub_name="telemetry",
)
# Time-based replay uses enqueued_time as the starting position
with client:
    client.receive(on_event=on_event, starting_position="-1")  # "-1" = start of stream

Step-by-step explanation.

1. event.offset is the opaque resume token. When a reader checkpoints, it persists the offset; on restart it asks the service to "start after this offset." You never do arithmetic on it — it is a cursor, not a counter.
2. event.sequence_number is the contiguous per-partition integer. Because it increments by exactly one, comparing current to last + 1 detects a skipped event — impossible to do reliably with the offset.
3. event.enqueued_time is the server timestamp. starting_position accepts a datetime, so "replay the last hour" is starting_position=datetime.utcnow() - timedelta(hours=1). This is time travel within the retention window.
4. None of the three is comparable across partitions. Sequence numbers restart per partition; offsets are partition-local byte positions; enqueued times can interleave. Any cross-partition ordering claim is a bug.
5. The checkpoint call persists the offset for the consumer group on that partition — the durable "I got this far" record that section 4 builds on.

Output.

Stamp	Scope	Contiguous?	Correct use
offset	partition	no	resume/checkpoint token
sequenceNumber	partition	yes	gap detection, dedupe
enqueuedTime	partition	no	time-based replay
(any) across partitions	hub	n/a	never — order undefined

Rule of thumb. Checkpoint with the offset, detect gaps with the sequence number, replay by enqueued time — and never compare any of the three across partitions. If you need cross-event ordering, the events must share a partition.

Worked example — at-least-once delivery and the duplicate window

Detailed explanation. Event Hubs delivers at least once. A consumer that processes an event and then crashes before checkpointing will reprocess that event on restart. Senior engineers state this up front and design idempotent handlers rather than pretending exactly-once exists at the transport layer. Walk through where duplicates come from.

• The checkpoint gap. Process event → (crash) → restart → reprocess from last checkpoint. Every event between the last checkpoint and the crash is redelivered.
• The rebalance gap. When partition ownership moves between hosts (scale-out, failure), the new owner resumes from the last checkpoint — anything the old owner processed but did not checkpoint is redelivered.
• The fix. Idempotency keyed on (partitionId, sequenceNumber) or a business key, so reprocessing is a no-op.

Question. Show how a naive counter double-counts on restart, and how an idempotent handler keyed on sequence number fixes it.

Input.

Scenario	Naive handler	Idempotent handler
Process 5 events, checkpoint at 3, crash	counts 5, then 2 again = 7	counts 5 (2 re-seen, skipped)
Duplicate key	double side effect	single side effect

Code.

# Idempotent handler keyed on (partition, sequence_number)
import redis

r = redis.Redis(host="cache", port=6379)

def handle(partition_id: str, event) -> None:
    dedupe_key = f"seen:{partition_id}:{event.sequence_number}"
    # SET NX returns True only the first time this key is written
    if not r.set(dedupe_key, 1, nx=True, ex=7 * 24 * 3600):
        return  # already processed this exact event -> skip
    apply_side_effect(event)   # e.g. increment a metric, write a row

def apply_side_effect(event) -> None:
    body = event.body_as_json()
    r.incrbyfloat(f"revenue:{body['store_id']}", body["amount"])

Step-by-step explanation.

1. The naive handler increments a counter for every event it sees. After processing five events and checkpointing at sequence 3, a crash forces a restart from 3 — so events 4 and 5 are seen twice, and the counter reads 7 instead of 5. This is the duplicate window in action.
2. The idempotent handler computes a dedupe key from (partition_id, sequence_number) — globally unique within the retention window because sequence numbers are contiguous per partition.
3. SET ... NX (set-if-not-exists) returns true only the first time the key is written; on a redelivery the key already exists, set returns false, and the handler returns early without applying the side effect.
4. The TTL (7 days) matches the retention window — after events age out of the hub they can never be redelivered, so the dedupe keys can safely expire too, keeping the cache bounded.
5. This turns at-least-once transport into effectively-once processing — the only honest way to get exactly-once semantics on top of a broker that guarantees at-least-once delivery.

Output.

Handler	Events processed	Counter after crash-at-3
Naive	5 then 2 re-seen	7 (wrong)
Idempotent (seq key)	5, 2 skipped	5 (correct)

Rule of thumb. Treat Event Hubs delivery as at-least-once and make every handler idempotent — dedupe on (partitionId, sequenceNumber) or a business key with a TTL that matches retention. Never claim exactly-once at the transport layer; earn it at the handler.

Data engineering interview question on the Event Hubs model

A senior interviewer often opens with: "You're ingesting 250,000 events/second of clickstream on Azure. You need per-user ordering, three independent downstream consumers (a warehouse, a fraud-scoring service, and an analytics team), and the ability to replay the last 24 hours. Walk me through the namespace, hub, partition count, consumer groups, and delivery semantics you'd design, and where duplicates can appear."

Solution Using a partition-count + consumer-group capacity model with keyed ordering

# Capacity model for a 250k ev/s clickstream on Azure Event Hubs
PEAK_EV_S        = 250_000
AVG_EVENT_KB     = 1
PEAK_MB_S        = PEAK_EV_S * AVG_EVENT_KB / 1024      # ~244 MB/s
PARTITION_MB_S   = 1
HEADROOM         = 1.5

raw_partitions   = PEAK_MB_S / PARTITION_MB_S           # ~244
sized_partitions = int(raw_partitions * HEADROOM)       # ~366 -> Premium/Dedicated

print(f"peak={PEAK_MB_S:.0f} MB/s  raw={raw_partitions:.0f}  sized={sized_partitions}")

consumer_groups = {
    "cg-warehouse": "batch loader -> ADLS/Synapse",
    "cg-fraud":     "real-time scoring via Azure Functions",
    "cg-analytics": "ad-hoc Databricks readers",
}
retention_hours = 24     # replay window
ordering_key    = "user_id"   # -> partition key so a user's events stay ordered

// Bicep-style hub definition (conceptual)
{
  "namespace":  "clickstream-prod (Dedicated, 2 CU)",
  "eventHub":   "clicks",
  "partitionCount": 400,
  "messageRetentionInDays": 1,
  "consumerGroups": ["$Default", "cg-warehouse", "cg-fraud", "cg-analytics"],
  "captureEnabled": true
}

# Producer pins ordering with a partition key = user_id
from azure.eventhub import EventHubProducerClient, EventData

producer = EventHubProducerClient.from_connection_string(conn_str, eventhub_name="clicks")
def send_click(click: dict) -> None:
    batch = producer.create_batch(partition_key=str(click["user_id"]))
    batch.add(EventData(json.dumps(click)))
    producer.send_batch(batch)   # all of one user's clicks -> same partition -> ordered

Step-by-step trace.

Step	Decision	Reasoning
Ingress	250k ev/s ≈ 244 MB/s	events × size
Raw partitions	~244	244 MB/s ÷ 1 MB/s/partition
Sized partitions	400	headroom + round up; forces Dedicated
Ordering	partition key = user_id	per-user total order
Consumer groups	4 ($Default + 3)	one per independent reader
Replay	retention 24h + enqueuedTime	time-travel within window
Duplicates	checkpoint + rebalance gaps	idempotent handlers required

After the design lands, each user's clicks hash to one partition and stay strictly ordered; the warehouse, fraud, and analytics consumer groups each read the full stream at their own pace; a 24-hour retention plus enqueued-time positioning gives replay; and every handler is idempotent because delivery is at-least-once.

Output:

Metric	Value
Partition count	400 (Dedicated)
Parallelism ceiling per consumer group	400
Consumer groups	4
Ordering guarantee	per user_id (per partition)
Replay window	24 h
Delivery semantics	at-least-once (idempotent handlers)

Why this works — concept by concept:

• Partition count as parallelism budget — 400 partitions set the maximum number of concurrent readers per consumer group and the ingress ceiling at ~1 MB/s each. Sizing it from peak throughput with headroom is the one irreversible decision, so it is made generously up front.
• Partition key = ordering scope — hashing on user_id sends all of one user's events to the same partition, which is the only place total order holds. Global order is neither offered nor needed.
• Consumer groups = independent fan-out — four groups give four independent views of the stream without re-reading storage per consumer; each tracks its own checkpoint so replaying one never disturbs the others.
• Retention + enqueued time = replay — a 24-hour retention window plus time-based starting positions turns the hub into a bounded time machine; beyond the window, Capture is the durable record.
• Cost — Dedicated capacity units for 400 partitions, O(1) append per event on the producer, and idempotency state bounded by the retention window. The alternative — under-partitioning and rebuilding later — costs a full re-ingest plus every downstream contract change. Pay once, up front.

Streaming
Topic — streaming
Streaming partition and consumer-group problems

Practice →

Event processing Topic — event-processing Event-processing ordering and delivery problems

Practice →

---

2. Producers, partition keys and throughput units

A partition key pins related events to one partition, and throughput units cap how fast the whole hub can ingest and emit

The mental model in one line: a producer either lets Event Hubs round-robin its events across partitions (maximum spread, no ordering) or supplies a partition key that is hashed to a single partition (ordering per key, at the cost of skew risk), and either way the whole hub's ingest and egress rate is bounded by the throughput units you provision — 1 TU buys ~1 MB/s or 1,000 events/second of ingress and ~2 MB/s of egress — so producer design is a joint decision about ordering, skew, and provisioned capacity that you tune together, never separately. Every senior Azure data engineer has watched a badly chosen partition key concentrate 80% of traffic on one partition while the throughput units sat mostly idle, and every one has learned to model the key distribution before shipping.

The three ways a producer targets a partition.

• Round-robin (no key, no id). The SDK spreads events evenly across partitions for maximum ingest throughput. No ordering guarantee between events. This is the default and the right choice when events are independent (e.g. stateless metrics).
• Partition key (hashed). You pass a partition_key string; the service hashes it to a partition. All events with the same key land on the same partition and are therefore ordered. This is how you get "per-device" or "per-user" ordering. The mapping of key → partition is stable but not something you control directly.
• Explicit partition id. You send straight to partition 7. Maximum control, minimum flexibility — if you ever change partition count, your routing logic breaks. Reserved for special cases (e.g. a single-partition control stream). Prefer partition keys over explicit ids.

Throughput units — the capacity currency (Standard).

• What 1 TU buys. Ingress: up to 1 MB/s or 1,000 events/second, whichever comes first. Egress: up to 2 MB/s or 4,096 events/second. Egress is shared across all consumer groups — three consumer groups each reading at 2 MB/s need 3 TU of egress, not 1.
• The whichever-first trap. A stream of tiny 100-byte events hits the 1,000 events/second limit long before the 1 MB/s byte limit. Throughput sizing must model both the byte rate and the event rate.
• Auto-inflate. Standard namespaces can auto-scale TUs up to a configured maximum (up to 40) as load rises, but they do not auto-deflate — you scale back down manually. Auto-inflate prevents throttling during bursts.
• Premium/Dedicated equivalents. Premium provisions processing units (PUs); Dedicated provisions capacity units (CUs). Same idea — a provisioned capacity ceiling — with isolation and higher limits.

Batching — the throughput multiplier.

• EventDataBatch. The SDK packs many events into one AMQP frame up to a size limit (default 1 MB, larger on Premium/Dedicated). One send_batch of 500 events costs one network round-trip, not 500.
• Batch + partition key. A batch created with a partition key sends every event in that batch to the same partition. A batch created without one round-robins the whole batch. You cannot mix keys within a single keyed batch.
• The event-count limit. Because 1 TU allows only 1,000 events/second, batching does not raise the event ceiling — but it slashes per-event overhead and network cost, so you hit the byte ceiling more efficiently.

Hot-partition skew — the failure mode.

• The symptom. One partition runs at 100% while others idle; consumers on the hot partition lag; the hub throttles even though aggregate TU usage looks low.
• The cause. A partition key with a skewed distribution — e.g. keying on country when 70% of traffic is one country, or on a tenant_id where one tenant dwarfs the rest.
• The fix. Choose a high-cardinality, evenly-distributed key (user_id, device_id, a hash of a composite). For a few whale keys, salt the key (tenant_id + ":" + random(0..N)) to spread a hot tenant across N partitions — at the cost of losing strict per-tenant order.

Worked example — partition-key hashing and skew detection

Detailed explanation. Before shipping a partition key, model its distribution. A quick histogram of key → partition over a sample of real traffic tells you whether you have even spread or a hot partition waiting to happen. Walk through simulating the hash and reading the histogram.

• The sample. 100,000 real events with their candidate keys.
• The hash. Event Hubs hashes the key to a partition; you can approximate the spread with a stable hash mod partition count.
• The read. A near-flat histogram is healthy; a spike is a hot partition.

Question. Given a candidate partition key, estimate the per-partition load and decide whether the key is safe.

Input.

Candidate key	Cardinality	Distribution	Verdict
country	~50	very skewed	unsafe
user_id	millions	~uniform	safe
tenant_id	thousands	few whales	needs salt

Code.

# Approximate Event Hubs partition assignment and check for skew
import hashlib
from collections import Counter

def assign_partition(key: str, partition_count: int) -> int:
    h = hashlib.md5(key.encode()).hexdigest()
    return int(h, 16) % partition_count

def skew_report(keys: list[str], partition_count: int) -> None:
    hist = Counter(assign_partition(k, partition_count) for k in keys)
    total = sum(hist.values())
    hottest = max(hist.values())
    ideal = total / partition_count
    print(f"partitions used: {len(hist)}/{partition_count}")
    print(f"hottest partition: {hottest} ({hottest / total:.1%} of traffic)")
    print(f"ideal per partition: {ideal:.0f}")
    print(f"skew factor (hottest / ideal): {hottest / ideal:.2f}x")

# 70% of events keyed 'US' -> a single hot partition
country_keys = ["US"] * 70_000 + ["IN"] * 15_000 + ["DE"] * 15_000
skew_report(country_keys, partition_count=16)
# hottest partition ~70% of traffic -> skew factor ~11x -> UNSAFE

Step-by-step explanation.

1. assign_partition mimics the service's stable hash: hash the key, take modulo the partition count. The real algorithm differs, but the behaviour — same key always maps to the same partition — is what matters for reasoning about skew.
2. skew_report builds a histogram of partition assignments over the sample and compares the hottest partition to the ideal (uniform) load.
3. Keying on country with 70% US traffic puts ~70% of events on one partition — a skew factor around 11× the ideal. That one partition throttles at ~1 MB/s while fifteen others sit nearly idle.
4. The skew factor (hottest ÷ ideal) is the single number to watch. Below ~2× is comfortable; above ~4× the key is unsafe and you need higher cardinality or salting.
5. Swapping to user_id — millions of distinct values — flattens the histogram; the skew factor drops near 1.0 and the whole partition budget is usable.

Output.

Key	Partitions used	Hottest share	Skew factor	Verdict
country	3/16	~70%	~11×	unsafe
user_id	16/16	~6.5%	~1.05×	safe
tenant_id (salted)	16/16	~8%	~1.3×	safe

Rule of thumb. Model the partition-key histogram on real traffic before shipping; keep the skew factor (hottest ÷ ideal) under ~2×. High-cardinality keys (user_id, device_id) are safe; low-cardinality keys (country, status) are hot-partition generators. Salt whale keys when you must key on a skewed field.

Worked example — sizing throughput units for byte rate and event rate

Detailed explanation. Throughput-unit sizing has to satisfy two ceilings at once — the byte ceiling (1 MB/s per TU) and the event ceiling (1,000 events/s per TU) — plus egress fan-out across consumer groups. The binding constraint is whichever needs more TUs. Walk through the math for a small-event, multi-consumer stream.

• Ingress. 250,000 events/s at 200 bytes each = 50 MB/s.
• Egress. Three consumer groups each read the full 50 MB/s = 150 MB/s aggregate egress.
• The two ceilings. Byte: 50 MB/s ÷ 1 MB/s = 50 TU. Event: 250,000 ÷ 1,000 = 250 TU. Event rate binds.

Question. Compute the throughput units required for both ingress ceilings and the egress fan-out, and identify the binding constraint.

Input.

Metric	Value
Event rate	250,000 ev/s
Event size	200 bytes
Ingress bytes	50 MB/s
Consumer groups reading full stream	3

Code.

# TU sizing must satisfy BOTH the byte ceiling and the event ceiling
EV_S          = 250_000
EVENT_BYTES   = 200
CG_READING    = 3          # consumer groups reading the full stream

ingress_mb_s  = EV_S * EVENT_BYTES / (1024 * 1024)     # ~47.7 MB/s
egress_mb_s   = ingress_mb_s * CG_READING              # ~143 MB/s

# Ingress ceilings: 1 TU = 1 MB/s OR 1000 ev/s (whichever binds)
tu_ingress_bytes  = ingress_mb_s / 1.0                 # ~48
tu_ingress_events = EV_S / 1000.0                       # 250  <-- binds
# Egress: 1 TU = 2 MB/s
tu_egress         = egress_mb_s / 2.0                   # ~72

tu_needed = max(tu_ingress_bytes, tu_ingress_events, tu_egress)
print(f"ingress bytes -> {tu_ingress_bytes:.0f} TU")
print(f"ingress events -> {tu_ingress_events:.0f} TU  (binding)")
print(f"egress -> {tu_egress:.0f} TU")
print(f"TU required = {tu_needed:.0f}  (Standard caps at 40 -> Premium/Dedicated)")

Step-by-step explanation.

1. Ingress bytes: 250,000 × 200 bytes ≈ 48 MB/s, which alone needs ~48 TU on the byte ceiling.
2. Ingress events: 250,000 events/s ÷ 1,000 = 250 TU on the event ceiling. Because the events are tiny (200 bytes), the event ceiling binds far harder than the byte ceiling — the classic small-event trap.
3. Egress: three consumer groups each read ~48 MB/s = ~143 MB/s aggregate; at 2 MB/s per TU that is ~72 TU. Egress fan-out is a real cost that candidates routinely forget.
4. The requirement is the max of all three — 250 TU here. Standard caps at 40 TU, so this workload is Premium (processing units) or Dedicated (capacity units) territory.
5. The lesson: never size TUs on byte rate alone. Small events blow the event ceiling; many consumer groups blow the egress ceiling. Model all three and take the max.

Output.

Ceiling	TU needed	Binding?
Ingress bytes (1 MB/s)	~48	no
Ingress events (1000/s)	250	yes
Egress (2 MB/s × 3 CGs)	~72	no
Required (max)	250	Premium/Dedicated

Rule of thumb. Size throughput units as max(byte-rate TUs, event-rate TUs, egress-fan-out TUs). Small events bind on the 1,000 events/s ceiling; wide fan-out binds on egress. Enable auto-inflate for bursts, and cross to Premium/Dedicated the moment the number exceeds Standard's 40-TU cap.

Worked example — a batched, keyed producer with retry and backpressure

Detailed explanation. A production producer batches for throughput, keys for ordering, retries on transient throttling, and applies backpressure when the hub pushes back. Walk through a robust producer that does all four.

• Batch. Pack events into EventDataBatch until it is full, then send.
• Key. One partition key per batch for per-key ordering.
• Retry. On ServiceBusyError/throttling, exponential backoff — the SDK does this, but you set the policy.
• Backpressure. Bound the in-flight send queue so a slow hub does not exhaust producer memory.

Question. Implement a keyed, batched producer that fills batches to capacity and backs off on throttling.

Input.

Parameter	Value
Partition key	device_id
Batch target	fill to size limit
Retry policy	exponential backoff, 5 tries
Backpressure	max 10 in-flight batches

Code.

# Robust keyed, batched producer
from azure.eventhub import EventHubProducerClient, EventData
from azure.eventhub.exceptions import EventHubError
import json, time

producer = EventHubProducerClient.from_connection_string(
    conn_str,
    eventhub_name="telemetry",
    retry_total=5,                 # SDK exponential backoff on throttling
    retry_backoff_factor=0.8,
)

def send_device_events(device_id: str, events: list[dict]) -> None:
    batch = producer.create_batch(partition_key=device_id)
    for e in events:
        data = EventData(json.dumps(e))
        try:
            batch.add(data)                     # raises when batch is full
        except ValueError:                      # batch full -> flush, start new
            _send_with_backoff(batch)
            batch = producer.create_batch(partition_key=device_id)
            batch.add(data)
    if len(batch) > 0:
        _send_with_backoff(batch)

def _send_with_backoff(batch) -> None:
    for attempt in range(5):
        try:
            producer.send_batch(batch)
            return
        except EventHubError as ex:
            if not ex.error.startswith("quota") and attempt == 4:
                raise
            time.sleep(0.8 * (2 ** attempt))    # 0.8, 1.6, 3.2, 6.4s

Step-by-step explanation.

1. The producer client is configured once with retry_total and retry_backoff_factor — the SDK's built-in exponential backoff kicks in on transient throttling (ServiceBusyError) before your own loop ever runs.
2. create_batch(partition_key=device_id) opens a batch pinned to one partition; every event added inherits that key, so all of one device's telemetry stays ordered.
3. batch.add raises ValueError when the batch reaches the size limit. Catching that is the signal to flush the full batch and start a fresh one — this fills each batch to capacity for maximum per-round-trip efficiency.
4. _send_with_backoff adds an application-level retry with exponential sleep (0.8s, 1.6s, 3.2s, …) as a belt-and-braces guard for quota errors that survive the SDK's own retries.
5. The final if len(batch) > 0 flush ensures the trailing partial batch is not dropped — a classic off-by-one that silently loses the last few events per device.

Output.

Behaviour	Result
Events per network round-trip	up to ~1 MB per batch (hundreds–thousands)
Ordering	per device_id (keyed batch)
Throttling response	exponential backoff, up to 5 tries
Trailing events	flushed (no drop)

Rule of thumb. Fill batches to the size limit, one partition key per batch, and rely on the SDK's backoff for throttling with a small application-level guard on top. Never send one event per call at scale — you will hit the 1,000-events/s-per-TU ceiling on overhead alone.

Data engineering interview question on producers and throughput units

A senior interviewer might ask: "You have 500,000 IoT events/second at ~150 bytes each, you need per-device ordering, and two consumer groups will each read the full stream. Design the producer (keying, batching), size the throughput units against both ceilings and the egress fan-out, and explain how you'd detect and fix a hot partition."

Solution Using a keyed batched producer with two-ceiling TU sizing

# 1. TU sizing against both ceilings + egress fan-out
EV_S        = 500_000
EVENT_BYTES = 150
CG_READING  = 2

ingress_mb  = EV_S * EVENT_BYTES / (1024 * 1024)   # ~71.5 MB/s
tu_bytes    = ingress_mb / 1.0                       # ~72
tu_events   = EV_S / 1000.0                           # 500  <-- binds
tu_egress   = (ingress_mb * CG_READING) / 2.0         # ~72
tu_required = max(tu_bytes, tu_events, tu_egress)     # 500 -> Dedicated

# 2. Keyed batched producer (per-device ordering, filled batches)
producer = EventHubProducerClient.from_connection_string(
    conn_str, eventhub_name="iot", retry_total=5, retry_backoff_factor=0.8)

def publish(device_id: str, events: list[dict]) -> None:
    batch = producer.create_batch(partition_key=device_id)
    for e in events:
        try:
            batch.add(EventData(json.dumps(e)))
        except ValueError:
            producer.send_batch(batch)
            batch = producer.create_batch(partition_key=device_id)
            batch.add(EventData(json.dumps(e)))
    if len(batch) > 0:
        producer.send_batch(batch)

# 3. Hot-partition detection on live metrics (per-partition incoming bytes)
def hot_partition_alert(per_partition_bytes: dict[str, float]) -> None:
    total = sum(per_partition_bytes.values())
    ideal = total / len(per_partition_bytes)
    for pid, b in per_partition_bytes.items():
        if b > 2 * ideal:                       # >2x ideal = hot
            print(f"HOT partition {pid}: {b/ideal:.1f}x ideal -> re-key or salt")

Step-by-step trace.

Step	Value	Reasoning
Ingress bytes	~72 MB/s	500k × 150 B
TU on byte ceiling	~72	72 MB/s ÷ 1
TU on event ceiling	500	500k ÷ 1000 — binds
TU on egress	~72	(72 × 2) ÷ 2
TU required	500	max of the three → Dedicated
Ordering	device_id key	per-device order
Skew guard	alert > 2× ideal	re-key/salt hot partitions

After the design, tiny 150-byte events force the event ceiling to bind at 500 TU-equivalent — a Dedicated cluster — not the ~72 the byte rate alone suggests; the keyed batched producer keeps each device ordered and fills batches for efficiency; and a per-partition bytes alert catches skew before it throttles the hub.

Output:

Metric	Value
Binding ceiling	event rate (500k/s)
Capacity	500 TU-equivalent → Dedicated
Ordering	per device_id
Egress fan-out	2 consumer groups × ~72 MB/s
Skew alert threshold	> 2× ideal per partition

Why this works — concept by concept:

• Two-ceiling TU sizing — capacity is max(byte TUs, event TUs, egress TUs). Small events make the 1,000-events/s ceiling bind at 500, far above what byte rate implies; missing this under-provisions and throttles.
• Keyed batches — one partition key per batch gives per-device ordering while filled batches amortise the per-event overhead that would otherwise waste the event ceiling.
• Egress fan-out is real capacity — two consumer groups each reading the full stream double the egress demand; egress at 2 MB/s per TU is a separate line item from ingress.
• Skew detection on per-partition bytes — a live "> 2× ideal" alert catches a hot partition before it throttles the whole hub; the fix is a higher-cardinality key or salting the whale key.
• Cost — a Dedicated cluster sized to the event ceiling, O(1) append per event, and a cheap per-partition metric watch. Under-sizing on byte rate alone would silently throttle at 1/7th the needed capacity — the most expensive kind of mistake because it looks fine in a demo.

Streaming
Topic — streaming
Streaming producer and partitioning problems

Practice →

Real-time analytics Topic — real-time-analytics Real-time analytics throughput-sizing problems

Practice →

---

3. Event Hubs Capture to storage

event hubs capture writes the raw stream to Blob or ADLS as Avro automatically — the zero-code cold path that turns a transient buffer into durable history

The mental model in one line: event hubs capture is a built-in, no-code feature that continuously writes every event flowing through a hub to Azure Blob Storage or ADLS Gen2 as Avro files, flushing on whichever comes first of a time window (minimum 60 seconds) or a size window (minimum 10 MB), organised into a partition- and date-templated folder path — so you get a durable, replayable, lake-native copy of the stream for the batch/cold path without ever writing or operating a consumer, and without paying an egress throughput-unit cost for the capture itself. Every senior Azure lakehouse uses Capture as the bronze layer: the hub handles the hot path, Capture handles the durable history, and the two never fight over the same throughput budget.

What Capture is — and why it exists.

• The problem it solves. Event Hubs retention is transient — 1 to 7 days on Standard. Anything you want to keep forever (audit, reprocessing, ML training sets) needs a durable landing zone. Writing that consumer yourself means checkpointing, scaling, and file-rolling logic.
• What Capture does. The service itself reads every partition and writes Avro files to your storage account on a schedule. No consumer, no compute, no checkpoint store to operate. It is the single most under-used feature in the product.
• Cost model. Capture is billed per throughput unit (a Capture add-on), not per consumer-group egress — it does not consume your egress TU budget the way a downstream reader would. The hot path keeps its full egress ceiling.

The window — time OR size, whichever first.

• Time window. Minimum 60 seconds, maximum 900 seconds (15 minutes). Capture flushes a file at least this often even on a slow stream.
• Size window. Minimum 10 MB, maximum 500 MB. Capture flushes when the accumulated data for a partition reaches this size.
• Whichever first. A busy partition hits the size window and rolls large files frequently; a quiet partition hits the time window and rolls small files on a timer. You tune the pair to balance file size against latency-to-lake.
• Empty windows. A toggle controls whether Capture emits an (empty) file when a window elapses with no events. Turning it off avoids a litter of zero-row files on sparse partitions.

The file path template — partition and date tokens.

• Default pattern. {Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second} with a .avro file per window. This yields a Hive-style, date-partitioned layout that Spark, Synapse, and Databricks read natively.
• Customisable. You can reorder tokens — e.g. put {Year}/{Month}/{Day} before {PartitionId} so downstream jobs prune by date first. The {PartitionId} token is mandatory; date tokens are optional but strongly recommended for pruning.
• Avro schema. Each file holds records with SequenceNumber, Offset, EnqueuedTimeUtc, SystemProperties, Properties, and Body (the raw event bytes). Your actual payload lives in Body, so downstream you parse Body as JSON/binary yourself.

Reading Capture downstream.

• Spark / Databricks. spark.read.format("avro").load(path) reads the files; you then decode Body (often CAST(Body AS STRING) then from_json).
• Synapse / Serverless SQL. OPENROWSET(..., FORMAT='AVRO') queries the files in place.
• Idempotent reprocessing. Because every record carries SequenceNumber and Offset, a reprocessing job can dedupe against a previous run — Capture files are the replay source when retention has already expired in the hub.

Worked example — enabling Capture with a tuned window and path

Detailed explanation. The canonical Capture setup: target an ADLS Gen2 container, pick a time/size window that balances file size against freshness, choose a date-first path for pruning, and disable empty-window files. Walk through the configuration.

• Target. ADLS Gen2 container bronze, folder prefix eventhubs.
• Window. 300 seconds OR 100 MB — files roughly every 5 minutes or 100 MB.
• Path. Date-first so batch jobs prune by day.
• Empty files. Off — no zero-row litter.

Question. Provide the Capture configuration for the telemetry hub landing into ADLS Gen2.

Input.

Setting	Value
Destination	ADLS Gen2 bronze container
Time window	300 s
Size window	100 MB
Path format	date-first, partition token included
Emit empty files	false

Code.

// Event Hubs Capture configuration (ARM/Bicep-style, conceptual)
{
  "captureDescription": {
    "enabled": true,
    "encoding": "Avro",
    "intervalInSeconds": 300,          // time window (60..900)
    "sizeLimitInBytes": 104857600,     // 100 MB size window (10MB..500MB)
    "skipEmptyArchives": true,         // no zero-row files
    "destination": {
      "name": "EventHubArchive.AzureBlockBlob",
      "properties": {
        "storageAccountResourceId": "/subscriptions/.../bronzelake",
        "blobContainer": "bronze",
        // date-first path so Spark prunes by day before partition
        "archiveNameFormat":
          "eventhubs/{EventHub}/{Year}/{Month}/{Day}/{Hour}/{PartitionId}/{Minute}_{Second}"
      }
    }
  }
}

# Resulting file layout in ADLS Gen2
bronze/eventhubs/telemetry/2026/09/05/09/0/12_30.avro
bronze/eventhubs/telemetry/2026/09/05/09/0/17_30.avro
bronze/eventhubs/telemetry/2026/09/05/09/1/12_30.avro
...
# One folder tree per hour, then per partition; ~5-min or 100 MB files

Step-by-step explanation.

1. enabled: true with encoding: Avro turns Capture on — the only two settings strictly required. Everything else is tuning.
2. intervalInSeconds: 300 and sizeLimitInBytes: 100 MB set the dual window. A busy partition rolls a ~100 MB file well before 300 seconds; a quiet one rolls whatever accumulated at the 5-minute mark.
3. skipEmptyArchives: true suppresses empty files on windows where a partition saw no events — important on sparse partitions to avoid thousands of zero-row Avro files that slow down listing.
4. archiveNameFormat puts the date tokens before {PartitionId}, so a downstream job filtering "just 2026-09-05" prunes at the date directory level before ever touching partition folders — a large I/O saving on big lakes.
5. The result is a Hive-style, date-partitioned Avro dataset that Spark, Synapse Serverless, and Databricks Auto Loader all read without any custom parsing of the folder structure.

Output.

Setting	Effect
300 s / 100 MB window	~5-min or 100 MB files
skipEmptyArchives	no zero-row files
date-first path	day-level pruning
Avro encoding	native lake read

Rule of thumb. Enable Capture on every production hub, tune the window to your target file size (aim for 64–256 MB files to keep Spark happy), put date tokens before the partition token for pruning, and skip empty archives. Capture is nearly free insurance against retention expiry.

Worked example — parsing the Capture Avro Body in Spark

Detailed explanation. Capture Avro records wrap your payload in a Body column of raw bytes alongside metadata columns. Downstream you read the Avro, cast Body to a string, and parse your JSON schema out of it. Walk through the Databricks/Spark read.

• Read. format("avro") over the date-partitioned path.
• Decode. Body is bytes → cast to string → from_json with your schema.
• Keep metadata. Retain SequenceNumber, EnqueuedTimeUtc for ordering and dedupe.

Question. Read one day of Capture files and flatten the telemetry payload into typed columns.

Input.

Column (Capture Avro)	Meaning
SequenceNumber	per-partition contiguous id
Offset	resume token
EnqueuedTimeUtc	server timestamp
Body	raw event bytes (your JSON)

Code.

# Databricks / Spark — read Capture Avro and flatten Body
from pyspark.sql.functions import col, from_json
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, LongType

payload_schema = StructType([
    StructField("device_id", StringType()),
    StructField("temp_c",    DoubleType()),
    StructField("ts",        LongType()),
])

path = "abfss://[e-mail]/eventhubs/telemetry/2026/09/05/*"

raw = spark.read.format("avro").load(path)

flat = (
    raw.select(
        col("SequenceNumber").alias("seq"),
        col("Offset").alias("offset"),
        col("EnqueuedTimeUtc").alias("enqueued"),
        from_json(col("Body").cast("string"), payload_schema).alias("p"),
    )
    .select("seq", "offset", "enqueued", "p.device_id", "p.temp_c", "p.ts")
)

# Dedupe across reprocessing runs using the partition-contiguous seq
deduped = flat.dropDuplicates(["device_id", "seq"])
deduped.write.mode("overwrite").saveAsTable("silver.telemetry")

Step-by-step explanation.

1. spark.read.format("avro").load(path) reads the day's Capture files; the wildcard over the hour folders pulls all partitions for that date.
2. Each row has the Capture metadata columns plus Body, which is your raw event as bytes. Casting Body to string and applying from_json with an explicit schema turns the opaque blob into typed fields.
3. Selecting p.device_id, p.temp_c, p.ts flattens the parsed struct into first-class columns — the silver-layer shape downstream analytics expects.
4. Retaining SequenceNumber lets a reprocessing run dedupe: dropDuplicates(["device_id", "seq"]) removes any records a prior run already wrote, because the sequence number is stable and contiguous per partition.
5. Writing to a silver.telemetry table completes the bronze (Capture Avro) → silver (typed, deduped) hop — the lambda architecture's cold path, built entirely on Capture output with no streaming consumer.

Output.

seq	device_id	temp_c	enqueued
40001	dev-7	22.4	2026-09-05T09:00:12Z
40002	dev-7	22.6	2026-09-05T09:00:17Z
40003	dev-9	19.1	2026-09-05T09:00:18Z

Rule of thumb. Treat Capture Avro as bronze: read with the Avro reader, decode Body with an explicit from_json schema, keep SequenceNumber for dedupe, and land a typed silver table. The metadata columns Capture adds are exactly what you need to make reprocessing idempotent.

Worked example — Capture vs a hand-rolled consumer for the cold path

Detailed explanation. Teams often reach for a custom "archiver" consumer before discovering Capture does the same job for free. Walk through the comparison so you can defend the Capture choice in an interview.

• Hand-rolled. A consumer group + code that batches events and writes files, with checkpointing, scaling, and file-rolling you own.
• Capture. The service does all of it, billed as a TU add-on, no egress consumed.
• When custom still wins. Non-Avro format, custom partitioning, or on-the-fly transformation — rare for a pure cold-path archive.

Question. Compare Capture against a hand-rolled archiver on operational cost, and state when each wins.

Input.

Dimension	Hand-rolled archiver	Event Hubs Capture
Code to write	consumer + file roller + checkpoints	none
Egress TU cost	consumes a consumer group's egress	none (Capture add-on)
Scaling	you own it	managed
Format	any	Avro only

Code.

Cold-path decision
==================
Q1. Do you need a durable, replayable copy of the raw stream?      -> yes: keep reading
Q2. Is Avro an acceptable landing format?                          -> yes: use Capture
                                                                   -> no:  hand-roll or transcode later
Q3. Do you need per-event transformation before landing?           -> yes: hot-path consumer
                                                                   -> no:  Capture
Q4. Do you need a custom folder scheme Capture cannot express?     -> yes: hand-roll
                                                                   -> no:  Capture (date+partition tokens)

Default for a pure archive/bronze layer: Capture.

Step-by-step explanation.

1. If you need durable history at all, the only real question is Avro-vs-not and transform-vs-not. For a raw bronze archive the answer is almost always "Avro is fine, no transform," which points straight at Capture.
2. The hand-rolled archiver consumes a consumer group's egress TUs — real capacity you now cannot give to a hot-path reader. Capture is billed as a separate add-on and does not touch egress. That alone often decides it.
3. The operational burden of a custom archiver — checkpoint store, scaling to partition count, file-rolling, poison handling — is exactly the burden Capture removes. Every line of that code is a line you now maintain and page on.
4. Custom archivers win only for a non-Avro landing format, an on-the-fly transformation, or a folder scheme the token template cannot express. These are the exception, not the rule, for a cold path.
5. The senior move is to default to Capture for the bronze layer and reserve custom consumers for the hot path where transformation and low latency actually justify the code.

Output.

Scenario	Winner
Raw bronze archive, Avro OK	Capture
Egress budget tight	Capture (no egress cost)
Per-event transform before land	hand-rolled hot path
Custom non-Avro format	hand-rolled

Rule of thumb. Default the cold path to Capture; it removes the consumer, the checkpoints, the scaling, and the egress cost. Hand-roll an archiver only when you need a non-Avro format or an inline transformation — and even then, land raw via Capture first and transform in a downstream batch job.

Data engineering interview question on Event Hubs Capture

A senior interviewer might ask: "You need a durable, replayable bronze layer for a 100,000 events/second hub whose hub retention is only 3 days, feeding a nightly Spark job and occasional 30-day reprocessing. Design the Capture configuration, the storage layout for efficient pruning, the downstream read, and how reprocessing stays idempotent after hub retention has expired."

Solution Using Capture as the durable bronze layer with idempotent reprocessing

// 1. Capture -> ADLS Gen2, tuned for ~128 MB files and day-level pruning
{
  "captureDescription": {
    "enabled": true,
    "encoding": "Avro",
    "intervalInSeconds": 600,
    "sizeLimitInBytes": 134217728,     // 128 MB target file
    "skipEmptyArchives": true,
    "destination": {
      "name": "EventHubArchive.AzureBlockBlob",
      "properties": {
        "blobContainer": "bronze",
        "archiveNameFormat":
          "telemetry/{Year}/{Month}/{Day}/{Hour}/{PartitionId}/{Minute}_{Second}"
      }
    }
  }
}

# 2. Nightly Spark job reads yesterday's Capture files, dedupes, writes silver
from pyspark.sql.functions import col, from_json
day = "2026/09/04"
raw = spark.read.format("avro").load(
    f"abfss://[e-mail]/telemetry/{day}/*/*")
flat = raw.select(
    col("SequenceNumber").alias("seq"),
    col("PartitionId").alias("pid"),
    from_json(col("Body").cast("string"), payload_schema).alias("p"),
)
# 3. Idempotent write: dedupe on (partition, seq) which is stable forever
(flat.dropDuplicates(["pid", "seq"])
     .write.mode("overwrite")
     .partitionBy("pid")
     .saveAsTable("silver.telemetry"))

# 4. 30-day reprocess works even after 3-day hub retention expired,
#    because Capture already durably stored every event in ADLS.
spark.read.format("avro").load(".../telemetry/2026/08/*/*/*")   # last month

Step-by-step trace.

Concern	Answer	Reasoning
Retention gap	hub 3 days vs 30-day reprocess	Capture holds it durably in ADLS
File size	128 MB target	600 s OR 128 MB window
Pruning	date-first path	day-level directory prune
Dedupe key	(PartitionId, SequenceNumber)	stable, contiguous, forever
Silver write	overwrite + partitionBy pid	idempotent reprocessing
Cost	Capture add-on, no egress	hot path keeps full egress

After the design, the hub's 3-day retention no longer bounds reprocessing — Capture has already written every event to ADLS as Avro, so a 30-day rerun just reads last month's folders. The nightly job dedupes on (PartitionId, SequenceNumber), so an overlapping or re-run window produces the exact same silver table.

Output:

Metric	Value
Durable history	unlimited (ADLS, beyond hub retention)
Target file size	~128 MB
Pruning granularity	per day
Reprocessing	idempotent (dedupe on pid, seq)
Egress TU cost of Capture	none

Why this works — concept by concept:

• Capture decouples retention from history — the hub keeps only 3 days of hot data, but Capture writes every event durably to ADLS, so replay and reprocessing are bounded by storage, not by the hub's retention window.
• Dual window tuned to file size — a 600 s / 128 MB window rolls files near the sweet spot for Spark, avoiding both tiny-file overhead and oversized files that hurt parallelism.
• Date-first path enables pruning — putting year/month/day before the partition token lets the reader skip whole days at the directory level, the single biggest I/O saving on a large lake.
• (PartitionId, SequenceNumber) dedupe = idempotence — these stamps are stable forever in the Avro records, so an overwrite-mode reprocess always converges to the same silver table regardless of overlap.
• Cost — a Capture TU add-on with zero egress consumption, O(events) storage in cheap ADLS, and O(day) pruning per job. The alternative — a hand-rolled archiver — costs a consumer group's egress plus all the checkpoint/scaling/file-rolling code you would then operate.

Event processing
Topic — event-processing
Event-processing archive and replay problems

Practice →

Real-time analytics Topic — real-time-analytics Real-time analytics cold-path and lakehouse problems

Practice →

---

4. Azure Functions triggers and checkpointing

The azure functions Event Hub trigger maps one function instance per partition, processes events in batches, and persists progress with checkpointing to a blob store

The mental model in one line: the azure functions Event Hub trigger is a serverless consumer where the runtime assigns at most one function instance per partition, invokes your handler with a batch of events, and — after a successful invocation — writes a checkpoint (the last processed offset per partition) into a blob checkpoint store, so scale is bounded by partition count, delivery is at-least-once (an invocation that crashes before checkpointing is retried), and correctness depends entirely on your handler being idempotent. Every senior event-driven Azure design leans on this trigger for the hot path, and every one treats the checkpoint-after-batch behaviour as the source of both its scaling model and its duplicate window.

How the trigger binds to the hub.

• One instance per partition. The Functions scale controller assigns partition ownership so that at most one active instance processes a given partition at a time within the consumer group. This is why the partition count is also the Functions parallelism ceiling.
• Batch by default. The trigger hands your function an array of events per invocation (cardinality: many), sized by maxBatchSize. Batching amortises invocation overhead and is the throughput-friendly default; single-event (cardinality: one) is available but slower.
• Consumer group. The trigger reads through a named consumer group — always create a dedicated consumer group for the Functions app rather than sharing $Default, so its checkpoints stay independent of other readers.
• Checkpoint store. The runtime persists offsets to a blob container (the AzureWebJobsStorage account by default). One checkpoint blob per partition records "processed through offset X."

The checkpoint lifecycle — where at-least-once comes from.

• When it checkpoints. After your function returns successfully for a batch, the runtime advances the checkpoint to the last event in that batch. It does not checkpoint per event.
• The duplicate window. If the function processes events but the host crashes before the post-batch checkpoint is written, the new owner resumes from the previous checkpoint and reprocesses the whole batch. That is the at-least-once guarantee made concrete.
• Rebalance. On scale-out/scale-in, partition ownership moves; the new owner resumes from the last durable checkpoint. Same duplicate window as a crash.
• The contract. Because the trigger is at-least-once, your handler must be idempotent. There is no configuration flag that makes it exactly-once — you earn exactly-once by deduping in the handler.

Scaling — the target-based scaler.

• Scale signal. The Functions runtime measures per-partition event backlog (unprocessed events) and scales instances toward the partition count. More backlog → more instances, up to one per partition.
• The ceiling. You can never usefully run more instances than partitions in one consumer group — the (partitions + 1)th instance has no partition to own and sits idle. Partition count caps useful concurrency.
• maxBatchSize and prefetchCount. maxBatchSize sets how many events one invocation gets; prefetchCount sets how many the client pulls ahead into memory. Larger values raise throughput per instance at the cost of memory and a bigger reprocess-on-crash window.
• Premium/Dedicated plans. For predictable low latency and no cold starts, the Functions Premium plan (pre-warmed instances) or a Dedicated plan beats Consumption for high-throughput event processing.

Poison events and error handling.

• Whole-batch failure. If your handler throws, the entire batch is retried from the last checkpoint — including the good events. A single poison event can wedge a partition.
• The fix. Wrap per-event processing in try/except inside the batch handler; route failures to a dead-letter (a Service Bus queue or a storage table) and continue, so one bad event does not block the partition's checkpoint.
• Ordering caveat. Dead-lettering a mid-batch event breaks strict order for that event; accept it, because the alternative is a stuck partition.

Worked example — a batch trigger with per-event error isolation

Detailed explanation. The canonical Functions Event Hub consumer takes a batch, processes each event idempotently, isolates per-event failures to a dead-letter, and lets the runtime checkpoint after the batch returns. Walk through the handler.

• Trigger. Cardinality many; dedicated consumer group; maxBatchSize tuned.
• Idempotency. Dedupe on (partition, sequenceNumber).
• Isolation. Per-event try/except → dead-letter → continue.

Question. Implement a Python Functions Event Hub trigger that processes a batch idempotently with per-event error isolation.

Input.

Setting	Value
Cardinality	many (batch)
Consumer group	cg-functions
Dedupe key	partition + sequenceNumber
Failure route	dead-letter table

Code.

# Azure Functions (Python v2) Event Hub batch trigger
import azure.functions as func
import json, logging

app = func.FunctionApp()

@app.event_hub_message_trigger(
    arg_name="events",
    event_hub_name="telemetry",
    connection="EVENTHUB_CONN",
    consumer_group="cg-functions",
    cardinality=func.Cardinality.MANY,   # batch of events per invocation
)
def process(events: list[func.EventHubEvent]) -> None:
    for event in events:
        pid = event.partition_key or event.metadata["PartitionContext"]["PartitionId"]
        seq = event.sequence_number
        try:
            if already_processed(pid, seq):      # idempotency guard
                continue
            body = json.loads(event.get_body().decode("utf-8"))
            handle_one(body)                     # the actual side effect
            mark_processed(pid, seq)
        except Exception as ex:                  # isolate poison events
            logging.error(f"poison event pid={pid} seq={seq}: {ex}")
            dead_letter(pid, seq, event.get_body())   # route + continue
    # Function returns -> runtime advances the checkpoint for this batch

Step-by-step explanation.

1. The trigger declares cardinality=MANY, so events is a list — one invocation covers a whole batch. This amortises the per-invocation cost and is the throughput-friendly default.
2. A dedicated consumer_group="cg-functions" keeps this app's checkpoints independent of any other reader on the hub — never point two different consumers at $Default.
3. already_processed(pid, seq) is the idempotency guard: because delivery is at-least-once, a redelivered batch re-enters the loop, but the guard makes reprocessing a no-op. mark_processed records the sequence number after the side effect.
4. The per-event try/except isolates poison events: a bad event is logged and dead-lettered, and the loop continues. Without this, one malformed event would throw, fail the whole batch, and the runtime would retry the entire batch forever — a wedged partition.
5. Crucially, the runtime checkpoints only after the function returns successfully. By catching per-event errors and returning normally, the batch checkpoints and the partition advances — the good events are not blocked by the one bad one.

Output.

Event in batch	Result
new, valid	processed, marked
duplicate (redelivered)	skipped (idempotent)
poison (bad JSON)	dead-lettered, batch continues
batch return	checkpoint advances

Rule of thumb. Process Event Hub batches with a per-event try/except that dead-letters failures and continues, guard side effects with a (partition, sequenceNumber) idempotency check, and let the runtime checkpoint after the batch. Never let one poison event throw out of the handler — it wedges the whole partition.

Worked example — tuning maxBatchSize, prefetch, and the checkpoint window

Detailed explanation. host.json controls batch size, prefetch, and how the trigger balances throughput against the reprocess-on-crash window. Larger batches raise throughput but widen the duplicate window on failure. Walk through the trade-off.

• maxEventBatchSize. Events per invocation. Higher = fewer invocations, more throughput, bigger reprocess window.
• prefetchCount. Events pulled into memory ahead of processing. Higher = smoother throughput, more memory.
• batchCheckpointFrequency. How many batches between checkpoints (host-level). 1 = checkpoint every batch (smallest duplicate window); higher = fewer storage writes but a wider window.

Question. Choose host.json settings for a high-throughput, duplicate-tolerant workload and for a low-latency, duplicate-sensitive one.

Input.

Workload	Priority	Batch	Checkpoint freq
High-throughput ETL	throughput	large	every few batches
Low-latency alerting	small duplicate window	small	every batch

Code.

// host.json — Event Hubs extension tuning
{
  "version": "2.0",
  "extensions": {
    "eventHubs": {
      "maxEventBatchSize": 256,        // events per invocation
      "prefetchCount": 512,            // pull-ahead buffer (>= batch size)
      "batchCheckpointFrequency": 1,   // checkpoint after EVERY batch
      "initialOffsetOptions": {
        "type": "fromStart"            // or "fromEnd" / "fromEnqueuedTime"
      }
    }
  }
}

# Two profiles
High-throughput ETL:
  maxEventBatchSize = 512
  prefetchCount     = 1024
  batchCheckpointFrequency = 5    # fewer storage writes; wider reprocess window
  -> more events/invocation, tolerate re-doing up to 5 batches on crash

Low-latency alerting:
  maxEventBatchSize = 32
  prefetchCount     = 64
  batchCheckpointFrequency = 1    # checkpoint every batch; smallest duplicate window
  -> react fast, minimise reprocessing on failure

Step-by-step explanation.

1. maxEventBatchSize is the throughput lever: 512 events per invocation means far fewer invocations than 32, so per-invocation overhead is amortised — good for bulk ETL where a wider reprocess window is acceptable.
2. prefetchCount should be at least the batch size (often 2×) so the client always has the next batch buffered; too low starves the handler, too high wastes memory.
3. batchCheckpointFrequency trades storage writes against the duplicate window. Frequency 1 checkpoints after every batch — the smallest reprocess window on a crash, at the cost of one blob write per batch. Frequency 5 checkpoints every fifth batch — fewer writes, but a crash can redo up to five batches.
4. Low-latency alerting picks small batches and frequency 1: react to events quickly and, if the host crashes, reprocess at most one small batch — minimising both latency and the duplicate blast radius.
5. High-throughput ETL picks large batches and a higher frequency: maximise events per invocation and reduce checkpoint-write overhead, accepting that a crash reprocesses several batches (safe because the handler is idempotent).

Output.

Profile	Batch	Prefetch	Checkpoint freq	Duplicate window
ETL	512	1024	5	up to 5 batches
Alerting	32	64	1	1 batch

Rule of thumb. Tune maxEventBatchSize for throughput, keep prefetchCount ≥ batch size, and set batchCheckpointFrequency to trade storage writes against the reprocess window — 1 for low-latency/duplicate-sensitive, higher for high-throughput. Idempotency is what makes a wider window safe.

Worked example — computing Functions parallelism and lag

Detailed explanation. To reason about whether a Functions app can keep up, compute the per-instance throughput and compare the aggregate to the ingress rate. If ingress exceeds partitions × per-instance throughput, lag grows without bound. Walk through the model.

• Instances. At most one per partition = partition count.
• Per-instance throughput. batch size ÷ handler latency per batch.
• Keep-up condition. partitions × per_instance_eps ≥ ingress_eps.

Question. For a 16-partition hub at 40,000 events/s, decide whether the current Functions config keeps up, and if not, what to change.

Input.

Parameter	Value
Partitions	16
Ingress	40,000 ev/s
maxEventBatchSize	256
Handler latency per batch	200 ms

Code.

# Can the Functions app keep up?
PARTITIONS        = 16
INGRESS_EPS       = 40_000
BATCH_SIZE        = 256
BATCH_LATENCY_S   = 0.200

per_instance_eps  = BATCH_SIZE / BATCH_LATENCY_S      # 1280 ev/s per instance
aggregate_eps     = PARTITIONS * per_instance_eps      # 20,480 ev/s

print(f"per-instance: {per_instance_eps:.0f} ev/s")
print(f"aggregate (16 instances): {aggregate_eps:.0f} ev/s")
print(f"ingress: {INGRESS_EPS} ev/s")

if aggregate_eps < INGRESS_EPS:
    # Options: (a) faster handler, (b) bigger batch, (c) more partitions
    needed_per_instance = INGRESS_EPS / PARTITIONS      # 2500 ev/s
    needed_latency = BATCH_SIZE / needed_per_instance   # 0.1024 s
    print(f"LAG GROWS. Need <= {needed_latency*1000:.0f} ms/batch "
          f"OR more partitions ({INGRESS_EPS/per_instance_eps:.0f} needed)")

Step-by-step explanation.

1. Per-instance throughput is batch size ÷ batch latency: 256 events ÷ 0.2 s = 1,280 events/s per instance.
2. Aggregate throughput is that times the instance ceiling (partition count): 16 × 1,280 = 20,480 events/s.
3. Ingress is 40,000 events/s — nearly double the aggregate the app can process. Backlog grows without bound; end-to-end latency climbs until the hub retention window is threatened.
4. To keep up at 16 partitions, each instance must hit 2,500 events/s, i.e. ≤ 102 ms per 256-event batch — halve the handler latency (optimise the side effect, parallelise I/O within the batch).
5. The alternative is more partitions: at 1,280 events/s per instance, keeping up needs ⌈40,000 ÷ 1,280⌉ = 32 partitions. But partition count is fixed at creation on Standard — another reason to over-provision partitions up front.

Output.

Metric	Value
Per-instance throughput	1,280 ev/s
Aggregate (16 instances)	20,480 ev/s
Ingress	40,000 ev/s
Verdict	lag grows — under-provisioned
Fixes	≤102 ms/batch, or 32 partitions

Rule of thumb. Check the keep-up condition partitions × (batch ÷ latency) ≥ ingress before shipping a Functions consumer. If it fails, the levers are a faster handler, a bigger batch, or more partitions — and since partitions are frozen at creation, over-provision them early.

Data engineering interview question on Azure Functions and checkpointing

A senior interviewer might ask: "Design an Azure Functions consumer for a 24-partition Event Hub at 60,000 events/second that must be exactly-once at the effect level, survive poison events without wedging a partition, and never fall behind. Cover the trigger config, the checkpoint model, idempotency, poison handling, the host.json tuning, and the scaling math."

Solution Using a checkpointed, idempotent, poison-isolating Functions consumer

# 1. Batch trigger: dedicated consumer group, idempotent, poison-isolated
import azure.functions as func
import json, logging

app = func.FunctionApp()

@app.event_hub_message_trigger(
    arg_name="events", event_hub_name="orders",
    connection="EVENTHUB_CONN", consumer_group="cg-functions",
    cardinality=func.Cardinality.MANY)
def process(events: list[func.EventHubEvent]) -> None:
    for e in events:
        pid = e.metadata["PartitionContext"]["PartitionId"]
        seq = e.sequence_number
        try:
            if claim_once(pid, seq):             # atomic set-if-absent -> idempotent
                handle(json.loads(e.get_body().decode()))
        except Exception as ex:
            logging.error(f"poison pid={pid} seq={seq}: {ex}")
            dead_letter(pid, seq, e.get_body())   # continue -> batch still checkpoints

// 2. host.json — throughput with a bounded duplicate window
{
  "version": "2.0",
  "extensions": {
    "eventHubs": {
      "maxEventBatchSize": 256,
      "prefetchCount": 512,
      "batchCheckpointFrequency": 1
    }
  }
}

# 3. Scaling math — does 24 partitions keep up at 60k ev/s?
PARTITIONS, INGRESS = 24, 60_000
BATCH, LATENCY_S = 256, 0.080                  # 80 ms/batch handler
per_instance = BATCH / LATENCY_S                # 3200 ev/s
aggregate = PARTITIONS * per_instance           # 76,800 ev/s >= 60k -> keeps up

Step-by-step trace.

Layer	Choice	Purpose
Trigger	cardinality many, cg-functions	batch throughput, isolated checkpoints
Idempotency	claim_once(pid, seq)	at-least-once → effect-once
Poison	per-event try/except → dead-letter	one bad event never wedges partition
Checkpoint	batchCheckpointFrequency 1	smallest reprocess window
Scaling	24 × 3200 = 76,800 ev/s	≥ 60k ingress → keeps up
Ceiling	≤ 24 instances	one per partition

After the design, each of the 24 partitions is owned by one instance processing 256-event batches in 80 ms (3,200 events/s each), for 76,800 events/s aggregate — comfortably above the 60,000 ingress. claim_once makes redelivered batches no-ops (effect-once), per-event isolation dead-letters poison events so the batch still checkpoints, and checkpoint-every-batch keeps the duplicate window to one small batch.

Output:

Metric	Value
Aggregate throughput	76,800 ev/s (≥ 60k)
Parallelism ceiling	24 (= partitions)
Delivery	at-least-once → effect-once
Duplicate window	1 batch
Poison handling	dead-letter, no wedge

Why this works — concept by concept:

• One instance per partition — the runtime caps useful concurrency at the partition count, so 24 partitions is both the parallelism budget and the scaling target; the keep-up math is done against that ceiling.
• claim_once idempotency — an atomic set-if-absent on (partition, sequenceNumber) turns at-least-once delivery into effect-once processing, the only honest way to get exactly-once on top of this trigger.
• Per-event poison isolation — catching and dead-lettering inside the loop lets the batch return successfully so the checkpoint advances; a thrown exception would retry the whole batch forever and wedge the partition.
• batchCheckpointFrequency = 1 — checkpointing after every batch keeps the reprocess-on-crash window to a single small batch, minimising duplicates at the cost of one blob write per batch.
• Cost — up to 24 warm instances, one checkpoint blob write per batch per partition, and a bounded idempotency store. The alternative — no idempotency or no poison isolation — trades a little saved code for double-counting and wedged partitions in production, the two failure modes interviewers probe hardest.

Event processing
Topic — event-processing
Event-processing checkpoint and idempotency problems

Practice →

Streaming Topic — streaming Streaming serverless-consumer scaling problems

Practice →

---

5. Patterns and Kafka-protocol compatibility

Fan-out with consumer groups, split hot and cold paths, and point existing Kafka clients straight at the Event Hubs endpoint on port 9093

The mental model in one line: the durable Event Hubs patterns are fan-out (N consumer groups = N independent read views), the lambda split (Capture feeds the cold/batch path while Azure Functions or Stream Analytics feed the hot/real-time path), and replay (rewind a consumer group within retention or reread from Capture beyond it) — and layered on top of all of them is Kafka-protocol compatibility, where a namespace exposes a Kafka endpoint on port 9093 with SASL so any existing Kafka client or Kafka Connect connector produces and consumes against Event Hubs unchanged, letting you retire a self-managed Kafka cluster without rewriting a single producer. Every senior Azure migration off self-managed Kafka rides this compatibility, and every mature Event Hubs deployment uses fan-out and the lambda split together.

Fan-out — one stream, many independent consumers.

• The primitive. Each consumer group is an independent view with its own checkpoints. A warehouse loader, a real-time scorer, and an analytics team each get a consumer group and read the full stream at their own pace.
• Why it beats re-publishing. You do not pay to write the stream N times; the producer writes once, and fan-out is a read-side construct. Egress capacity (throughput units) is the only cost that scales with consumer count.
• Isolation. A slow or broken consumer group cannot stall the others — each tracks its own offset. This is the operational safety net that makes multi-team streaming sane.

The lambda split — hot path and cold path from one hub.

• Cold path. Capture writes Avro to ADLS for batch analytics, ML training, and reprocessing (section 3). Latency: minutes. Durability: unlimited.
• Hot path. Azure Functions (section 4), Stream Analytics, or Databricks Structured Streaming consume the hub directly for sub-second reactions. Latency: milliseconds to seconds.
• Why both. The warehouse wants complete, deduped history it can rerun; the alerting service wants the newest event now. One hub, two consumer paths, no duplicated ingestion.

Replay — rewind within retention, reread beyond it.

• Within retention. Reset a consumer group's checkpoint to an earlier offset or enqueued time and reprocess — bounded by the hub's retention window (1–7 days Standard).
• Beyond retention. The events have aged out of the hub, but Capture still holds them in ADLS. Replay becomes a batch read of Capture Avro (section 3). This is why Capture is the true system of record, not the hub.
• Per-group replay. Because offsets are per consumer group, you can replay one consumer's view without disturbing any other — reprocess the warehouse feed while alerting keeps moving.

Kafka-protocol compatibility — the drop-in endpoint.

• What it is. Standard tier and above expose a Kafka endpoint at <namespace>.servicebus.windows.net:9093 speaking the Kafka protocol (1.0+). A Kafka producer/consumer or Kafka Connect connector points at it with SASL/PLAIN auth (username $ConnectionString, password = the namespace connection string) and works unchanged.
• The mapping. A Kafka topic maps to an Event Hub; a Kafka partition maps to an Event Hubs partition; a Kafka consumer group maps to an Event Hubs consumer group. The mental models line up one-to-one.
• What is supported. Produce, consume, consumer groups, and Kafka Connect. Transactions and log compaction are not supported the same way; idempotent producer semantics are limited. Basic tier has no Kafka endpoint.
• Why it matters. You migrate off a self-managed Kafka cluster by changing bootstrap servers and auth in config — no code rewrite — then adopt Capture, Functions, and managed scaling incrementally.

Worked example — a Kafka client producing to Event Hubs unchanged

Detailed explanation. The migration test: take an existing Kafka producer, change only the bootstrap servers and SASL config, and watch it produce to Event Hubs. Walk through the config that makes a stock kafka-python (or Java) client work against the Kafka endpoint.

• Endpoint. <namespace>.servicebus.windows.net:9093.
• Auth. SASL/PLAIN over TLS; username $ConnectionString, password = namespace connection string.
• Code. Unchanged producer API.

Question. Configure a standard Kafka producer to write to an Event Hub via the Kafka endpoint, changing only connection settings.

Input.

Setting	Value
bootstrap.servers	ns.servicebus.windows.net:9093
security.protocol	SASL_SSL
sasl.mechanism	PLAIN
username	$ConnectionString
password	Endpoint=sb://ns...;SharedAccessKey=...

Code.

# Stock kafka-python producer pointed at the Event Hubs Kafka endpoint
from kafka import KafkaProducer
import json

NAMESPACE_CONN = "Endpoint=sb://ns.servicebus.windows.net/;SharedAccessKeyName=..."

producer = KafkaProducer(
    bootstrap_servers="ns.servicebus.windows.net:9093",
    security_protocol="SASL_SSL",
    sasl_mechanism="PLAIN",
    sasl_plain_username="$ConnectionString",
    sasl_plain_password=NAMESPACE_CONN,
    value_serializer=lambda v: json.dumps(v).encode("utf-8"),
    key_serializer=lambda k: k.encode("utf-8"),
)

# The Kafka 'topic' name is the Event Hub name; the key pins the partition
producer.send("telemetry", key="device-42", value={"temp_c": 22.4})
producer.flush()
# Same API, same key-based partitioning, now landing in Event Hubs

Step-by-step explanation.

1. The only changes from a vanilla Kafka setup are bootstrap_servers (the namespace on port 9093), security_protocol=SASL_SSL, and the SASL/PLAIN credentials — everything else is the standard Kafka producer API.
2. The username is the literal string $ConnectionString; the password is the namespace's connection string. This is the Event Hubs convention for Kafka SASL auth — no separate Kafka user management.
3. The Kafka topic argument ("telemetry") maps to the Event Hub of the same name. Creating the topic on the Kafka side corresponds to creating the event hub in the namespace.
4. The Kafka message key ("device-42") is hashed to a partition exactly like an Event Hubs partition key — so the ordering guarantees you relied on in Kafka carry over unchanged.
5. The result: an existing Kafka codebase produces to Event Hubs by editing config, not code. This is the mechanism that makes "retire the self-managed Kafka cluster" a config change rather than a rewrite.

Output.

Kafka concept	Event Hubs mapping
topic	event hub
partition	partition
consumer group	consumer group
message key	partition key
bootstrap:9093 + SASL	namespace endpoint + connection string

Rule of thumb. Migrate Kafka producers/consumers to Event Hubs by changing bootstrap servers to namespace:9093, security to SASL_SSL/PLAIN, and credentials to $ConnectionString + the namespace connection string — no code changes. Topic→hub, partition→partition, group→group map one-to-one.

Worked example — consumer-group fan-out for three independent teams

Detailed explanation. Fan-out gives three teams the same stream with independent progress. Walk through wiring three consumer groups so a slow team never blocks a fast one.

• Groups. cg-warehouse, cg-realtime, cg-ml.
• Independence. Each has its own checkpoints; replay one without touching others.
• Egress cost. Three groups reading the full stream = 3× egress TUs.

Question. Configure three consumers so each reads the full stream independently, and state the egress capacity implication.

Input.

Consumer group	Purpose	Latency need
cg-warehouse	batch load to Synapse	minutes
cg-realtime	Functions scorer	sub-second
cg-ml	feature extraction	seconds

Code.

# Three independent consumers, one hub, three consumer groups
from azure.eventhub import EventHubConsumerClient

def make_consumer(group: str):
    return EventHubConsumerClient.from_connection_string(
        conn_str, consumer_group=group, eventhub_name="telemetry")

warehouse = make_consumer("cg-warehouse")   # own offsets
realtime  = make_consumer("cg-realtime")    # own offsets
ml        = make_consumer("cg-ml")          # own offsets

# Replaying the warehouse feed does NOT disturb realtime or ml:
#   reset cg-warehouse to an earlier position; the others keep their offsets.

# Egress capacity: each group reads the full stream.
INGRESS_MB_S = 20
egress_needed_mb_s = INGRESS_MB_S * 3        # 60 MB/s across 3 groups
tu_egress = egress_needed_mb_s / 2.0         # 30 TU of egress

Step-by-step explanation.

1. Three EventHubConsumerClients point at the same hub but different consumer groups. Each group maintains its own per-partition checkpoints, so the three read positions are fully independent.
2. cg-realtime (a Functions app) reacts in sub-second time; cg-warehouse batches every few minutes; cg-ml sits in between. None of them constrains the others' pace.
3. Replay isolation is the headline benefit: resetting cg-warehouse to reprocess yesterday does not move cg-realtime or cg-ml. Per-group offsets make surgical replay possible.
4. The cost is egress: each group reads the full 20 MB/s stream, so aggregate egress is 60 MB/s, needing ~30 TU of egress capacity (2 MB/s per TU). Fan-out scales egress, not ingress.
5. If a fourth team appears, add a fourth consumer group — up to 20 on Standard — and budget another 20 MB/s of egress. The producer side is untouched.

Output.

Group	Independent offsets	Replay isolation	Egress share
cg-warehouse	yes	yes	20 MB/s
cg-realtime	yes	yes	20 MB/s
cg-ml	yes	yes	20 MB/s
Total egress	—	—	60 MB/s (~30 TU)

Rule of thumb. Give every independent downstream system its own consumer group for pace and replay isolation, and budget egress throughput units at ingress × number_of_groups ÷ 2 MB/s. Fan-out is a read-side pattern — it scales egress capacity, never the producer.

Worked example — the throughput-unit + partition scaling playbook

Detailed explanation. Scaling Event Hubs means moving two dials together — throughput units (or PUs/CUs) for rate, and partition count for parallelism — while respecting that partition count is frozen on Standard. Walk through the playbook for a hub outgrowing its tier.

• Rate dial. Raise TUs (or enable auto-inflate) up to the tier cap.
• Parallelism dial. Partition count — set at creation, immutable on Standard, so provision ahead.
• The migration. When either dial hits the tier cap, move tiers (Standard → Premium → Dedicated).

Question. A Standard hub at 32 partitions and 40 TU is saturating both. Lay out the scaling decision.

Input.

Symptom	Dial	Standard cap
throttling on ingress rate	throughput units	40 TU
consumer parallelism maxed	partition count	32 partitions
both saturated	tier	move to Premium/Dedicated

Code.

Event Hubs scaling playbook
===========================
1. Rate pressure (throttling, high TU usage)?
     -> raise throughput units / enable auto-inflate (Standard cap 40 TU)
     -> if at cap: move to Premium (PUs) or Dedicated (CUs)

2. Parallelism pressure (consumers can't keep up, one-instance-per-partition maxed)?
     -> need more partitions
     -> Standard: partition count is FIXED at creation -> cannot grow in place
        -> create a new hub with more partitions + dual-write/migrate consumers
     -> Premium/Dedicated: higher partition ceilings (100+/hub, thousands total)

3. Both saturated at Standard caps (32 partitions AND 40 TU)?
     -> migrate the namespace to Premium or Dedicated
     -> re-provision the hub with a generous partition count UP FRONT

Golden rule: over-provision partitions at creation. Rate you can raise later;
partitions on Standard you cannot.

Step-by-step explanation.

1. Rate pressure — throttling or sustained high TU usage — is the easy dial: raise throughput units or enable auto-inflate, up to the Standard cap of 40 TU. Beyond that, the tier itself must change.
2. Parallelism pressure — consumers maxed at one instance per partition and still lagging — needs more partitions. This is the hard dial on Standard because partition count is immutable after creation.
3. Growing partitions on Standard means creating a new hub with more partitions and migrating producers/consumers (often dual-writing during cutover) — a real project, not a config tweak. This is the cost of under-provisioning partitions early.
4. When both dials hit the Standard caps (32 partitions and 40 TU), the answer is a tier move: Premium (processing units, ≤100 partitions/hub) or Dedicated (capacity units, thousands of partitions). Re-provision the hub with a generous partition count during the migration.
5. The golden rule falls out: over-provision partitions at creation because you can always raise the rate dial later, but you cannot grow partitions in place on Standard. Extra partitions are cheap insurance; a partition-count rebuild is expensive.

Output.

Pressure	Action	In place?
Rate (TU)	raise TU / auto-inflate	yes (to 40 TU)
Parallelism (partitions)	more partitions	no on Standard — new hub
Both at cap	Premium/Dedicated migration	tier change

Rule of thumb. Scale rate with throughput units (raise anytime up to the tier cap) and scale parallelism with partition count (set once, generously, at creation). Over-provision partitions early — on Standard they are immutable, so a partition-count change is a hub rebuild and consumer migration, not a slider.

Data engineering interview question on patterns and Kafka compatibility

A senior interviewer might ask: "You're migrating a self-managed Kafka cluster to Azure with minimal code change, then adding a durable lake copy and a real-time handler, and three teams need independent access. Design the Event Hubs setup: the Kafka-endpoint migration, the fan-out with consumer groups, the lambda split with Capture and Functions, and the scaling story."

Solution Using Kafka-protocol ingestion + fan-out + lambda split

# 1. Existing Kafka producers/consumers: config-only migration to :9093
producer = KafkaProducer(
    bootstrap_servers="ns.servicebus.windows.net:9093",
    security_protocol="SASL_SSL", sasl_mechanism="PLAIN",
    sasl_plain_username="$ConnectionString",
    sasl_plain_password=NAMESPACE_CONN)
# topic -> event hub, partition -> partition, group -> consumer group (no code change)

# 2. Fan-out: one hub, four consumer groups (independent offsets + replay)
telemetry (hub, 400 partitions)
 ├── cg-kafka-legacy   (existing Kafka consumers, unchanged)
 ├── cg-warehouse      (batch)
 ├── cg-realtime       (Azure Functions hot path)
 └── cg-ml             (feature extraction)

// 3. Lambda split: Capture (cold) + Functions (hot) from the same hub
{
  "capture": { "enabled": true, "encoding": "Avro",
               "intervalInSeconds": 600, "sizeLimitInBytes": 134217728,
               "destination": "bronze/telemetry/{Year}/{Month}/{Day}/{Hour}/{PartitionId}" },
  "hotPath": { "consumer": "AzureFunctions", "consumerGroup": "cg-realtime",
               "checkpointStore": "blob", "idempotent": true }
}

Step-by-step trace.

Layer	Component	Purpose
Migration	Kafka clients → :9093 SASL	retire self-managed Kafka, no rewrite
Ingestion	1 hub, 400 partitions	shared source of truth
Fan-out	4 consumer groups	independent pace + replay
Cold path	Capture → Avro in ADLS	durable history, reprocessing
Hot path	Functions on cg-realtime	sub-second reactions, checkpointed
Scaling	TUs for rate, partitions for parallelism	rate raisable; partitions pre-sized

After the migration, the legacy Kafka apps talk to Event Hubs by config alone; the hub fans out to four consumer groups that never block each other; Capture lands durable Avro for the warehouse and reprocessing while Azure Functions handle the real-time path with checkpointed, idempotent consumers; and scaling is rate-via-TUs plus a generously pre-sized partition count.

Output:

Downstream	Access	Latency	Durability
Legacy Kafka apps	Kafka endpoint :9093	unchanged	hub retention
Warehouse	cg-warehouse + Capture	minutes	unlimited (ADLS)
Real-time	cg-realtime (Functions)	sub-second	checkpointed
ML	cg-ml	seconds	hub + Capture

Why this works — concept by concept:

• Kafka endpoint = config-only migration — pointing existing clients at namespace:9093 with SASL maps topic→hub, partition→partition, group→group, so a self-managed Kafka cluster retires without a code rewrite.
• Consumer-group fan-out — four independent views share one ingestion; each tracks its own offsets, so a slow or replaying group never stalls the others, and only egress capacity scales with consumer count.
• Lambda split — Capture serves the cold path (durable Avro, unlimited history, reprocessing) while Functions serve the hot path (sub-second, checkpointed, idempotent) from the same hub, with no duplicated ingestion.
• Two-dial scaling — throughput units scale rate (raisable up to the tier cap) and partition count scales parallelism (frozen on Standard, so pre-sized generously); knowing which dial is reversible is the senior insight.
• Cost — one hub with a generous partition count, TUs/PUs/CUs for rate, egress scaling with consumer-group count, and a Capture add-on for the cold path. The alternative — re-publishing the stream per consumer or rewriting Kafka producers — costs far more in both capacity and engineering time than fan-out plus a config-only endpoint swap.

Streaming
Topic — streaming
Streaming Kafka-compatibility and migration problems

Practice →

Real-time analytics
Topic — real-time-analytics
Real-time analytics fan-out and lambda-pattern problems

Practice →

---

Cheat sheet — Azure Event Hubs recipes

• The hierarchy. Namespace (tier + capacity + networking) → event hub (topic; fixed partition count on Standard) → partition (ordering + parallelism unit; offset/sequenceNumber/enqueuedTime) → consumer group (independent read view, own checkpoints). Order is per-partition, never global; a single consumer group runs at most one active reader per partition, so partition count is your parallelism ceiling.
• Partition sizing. partitions ≈ (peak_ingress_MB_s / 1 MB/s) × 1.5 headroom, then check the tier cap (Standard 32, Premium ≤100/hub, Dedicated thousands). Partition count is immutable on Standard — over-provision at creation; growing it means a new hub and a consumer migration.
• Consumer-group sizing. One consumer group per independent downstream system (Standard cap 20). Adding a group adds a read view, not ingress capacity; it does add egress load (ingress × groups ÷ 2 MB/s TUs).
• Throughput-unit math. 1 TU = 1 MB/s or 1,000 events/s ingress (whichever binds), 2 MB/s egress shared across all consumer groups. Size as max(byte-rate TUs, event-rate TUs, egress-fan-out TUs). Tiny events bind on the event ceiling; wide fan-out binds on egress. Enable auto-inflate for bursts (Standard cap 40 TU).
• Partition key vs round-robin. No key → round-robin (max spread, no order). partition_key → hashed to one partition (per-key order, skew risk). Explicit partition id → full control, brittle. Keep the skew factor (hottest ÷ ideal) under ~2×; salt whale keys (key:rand(0..N)) to spread a hot tenant.
• Capture config. enabled + encoding=Avro, time window 60–900 s, size window 10–500 MB (whichever first), skipEmptyArchives=true, path {...}/{Year}/{Month}/{Day}/{Hour}/{PartitionId}/... (date tokens before partition for pruning). Bills as a TU add-on, consumes no egress. Aim for 64–256 MB files.
• Reading Capture. spark.read.format("avro"), then decode the Body bytes with from_json + explicit schema; keep SequenceNumber/Offset/EnqueuedTimeUtc. Dedupe reprocessing on (PartitionId, SequenceNumber) — stable forever, so overwrite-mode reruns are idempotent.
• Functions trigger. Event Hub trigger, cardinality=many (batch), dedicated consumer group, blob checkpoint store. One instance per partition = parallelism ceiling. Checkpoint fires after a successful batch → at-least-once delivery → duplicates on crash/rebalance. Make handlers idempotent (claim_once(partition, sequenceNumber)).
• host.json tuning. maxEventBatchSize for throughput, prefetchCount ≥ batch size, batchCheckpointFrequency trades storage writes vs the reprocess window (1 = smallest duplicate window). Isolate poison events per-event (try/except → dead-letter → continue) so one bad event never wedges a partition's checkpoint.
• Functions keep-up check. partitions × (maxEventBatchSize ÷ batch_latency) ≥ ingress_eps, or lag grows unbounded. Fix with a faster handler, bigger batch, or more partitions (and remember partitions are fixed on Standard).
• Kafka endpoint. Standard+ exposes namespace:9093, SASL_SSL/PLAIN, username $ConnectionString, password = namespace connection string. topic→hub, partition→partition, group→group. Produce/consume/Connect supported; transactions and log compaction are not fully. Migrate self-managed Kafka by config, not code.
• Patterns. Fan-out (N consumer groups = N views), lambda split (Capture cold + Functions/Stream Analytics hot), replay (rewind a group within retention; reread Capture beyond it). Scale rate with TUs (reversible) and parallelism with partitions (pre-size, Standard-immutable). Basic tier = 1 consumer group, no Capture, no Kafka — never for production.

Frequently asked questions

What is Azure Event Hubs in one sentence?

Azure Event Hubs is a fully-managed, partitioned, append-only event-ingestion service — the Azure equivalent of a managed Kafka topic — where producers write immutable, offset-stamped events into a fixed set of partitions, each partition preserves total order within itself, and any number of downstream systems read the stream independently through consumer groups that track their own position. It is a retention-bounded buffer (1–7 days on Standard), not a database, so durable history comes from event hubs capture writing the raw stream to Blob/ADLS. It is the default front door for event-driven and streaming pipelines on Azure, feeding Azure Functions, Stream Analytics, Databricks, and Synapse.

Partitions vs consumer groups — what's the difference?

A partition is the unit of ordering and parallelism: an append-only sequence where events are totally ordered and stamped with a contiguous sequence number, and where a single consumer group can run at most one active reader — so the partition count is the parallelism ceiling. A consumer group is an independent read view of the whole hub: each group has its own per-partition checkpoints, so a warehouse loader, a real-time scorer, and an analytics team can each consume the full stream at their own pace without interfering, and you can replay one group's position without disturbing the others. Partitions are set at creation (immutable on Standard); consumer groups can be added later (up to 20 on Standard). Rule of thumb: size partitions from throughput, size consumer groups from the number of independent downstream systems.

How does Event Hubs Capture work?

event hubs capture is a built-in, no-code feature that continuously writes every event flowing through a hub to Azure Blob Storage or ADLS Gen2 as Avro files. It flushes a file per partition on whichever comes first of a time window (60–900 seconds) or a size window (10–500 MB), into a partition- and date-templated folder path. Capture is billed as a throughput-unit add-on and — critically — consumes no consumer-group egress, so the hot path keeps its full egress budget. Each Avro record carries SequenceNumber, Offset, EnqueuedTimeUtc, and your payload in a Body column, which you decode downstream with from_json. Capture is the durable cold-path/bronze layer: it turns the transient hub into unlimited, replayable history and is the true system of record once hub retention has expired.

How does checkpointing work with Azure Functions?

The Azure Functions Event Hub trigger reads through a consumer group and, after your handler returns successfully for a batch of events, writes a checkpoint — the last processed offset per partition — into a blob checkpoint store. Because it checkpoints per batch (not per event), delivery is at-least-once: if the host crashes or partition ownership rebalances before the checkpoint is written, the new owner resumes from the previous checkpoint and reprocesses the batch, producing duplicates. There is no flag that makes this exactly-once; you earn effect-once by making the handler idempotent — typically deduping on (partitionId, sequenceNumber) with an atomic set-if-absent. Tune the duplicate window with batchCheckpointFrequency (1 = checkpoint every batch, smallest window) and isolate poison events per-event so one failure never wedges a partition.

Event Hubs vs Kafka — do I need both?

Usually not — Event Hubs exposes a Kafka-protocol endpoint on port 9093 (Standard tier and above) that speaks the Kafka protocol, so existing Kafka producers, consumers, and Kafka Connect connectors work against it by changing only the bootstrap servers and SASL credentials ($ConnectionString + the namespace connection string). Kafka concepts map one-to-one: topic → event hub, partition → partition, consumer group → consumer group. This lets you retire a self-managed Kafka cluster without rewriting application code, then adopt managed features like Capture, auto-inflate, and Azure Functions incrementally. The caveats: transactions, full idempotent-producer semantics, and log compaction are not supported the same way as Apache Kafka, and the Basic tier has no Kafka endpoint at all. If you need those specific Kafka features, evaluate carefully; for the vast majority of produce/consume workloads, Event Hubs replaces Kafka.

How many throughput units and partitions do I need?

Size them from different inputs. Partitions come from throughput and parallelism: (peak_ingress_MB_s / 1 MB/s) × 1.5 headroom, capped by the tier (Standard 32, Premium ≤100/hub, Dedicated thousands) — and because partition count is immutable on Standard, over-provision it at creation. Throughput units come from rate: max(byte-rate TUs at 1 MB/s each, event-rate TUs at 1,000 events/s each, egress-fan-out TUs at 2 MB/s each). Small events bind on the 1,000-events/s-per-TU ceiling (a 200-byte-event stream needs far more TUs than its byte rate suggests), and every extra consumer group adds egress load. Enable auto-inflate so bursts do not throttle. The asymmetry is the key insight: you can raise throughput units anytime, but you cannot grow partitions in place on Standard, so pre-size partitions generously and tune TUs as you go.

Practice on PipeCode

• Drill the streaming practice library → for the partitioning, consumer-group, throughput-sizing, and Kafka-migration problems senior interviewers love.
• Rehearse on the event-processing practice library → for checkpointing, idempotency, at-least-once delivery, and archive/replay patterns.
• Sharpen the real-time axis with the real-time analytics practice library → for the lambda-split, hot/cold-path, and fan-out scenarios.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the Event Hubs partition/consumer-group model against real graded inputs.

Lock in event-driven Azure muscle memory

Docs explain the knobs. PipeCode drills explain the decision — when to add a partition versus a throughput unit, why timestamp order is per-partition, when Capture beats a hand-rolled archiver, and how at-least-once delivery forces idempotent Functions handlers. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs event-driven data engineers actually face.

Practice streaming problems →
Practice event-processing problems →