Amazon Kinesis Deep Dive: Data Streams, Firehose & Analytics

amazon kinesis is the AWS-native answer to the question "how does a business move millions of events per second from where they happen to everywhere they need to land, in seconds, without dropping or reordering a single one" — and it is the single service family that senior data engineers get asked to reason about most often, because "just use Kinesis" hides three genuinely different products behind one brand. A clickstream event, an IoT temperature reading, a change-data-capture row, a payment authorization — each has to reach the real-time dashboard, the fraud model, the data lake, and the warehouse without re-reading a source of truth, without losing ordering within a customer's activity, and without a human ever hand-tuning a cluster. The engineering trade-off is not "should we stream" — every event-driven stack needs a durable pipe — but which Kinesis service you point at each downstream, and how you size the unit of throughput underneath it.

This guide is the deep dive you wished existed the first time an interviewer asked "walk me through Kinesis Data Streams versus Firehose and when you'd pick each," or "your producer is throwing ProvisionedThroughputExceededException on one shard while the rest sit idle — what's wrong?", or "explain enhanced fan-out and why it costs more than the default consumer." It walks through the whole family — Data Streams as the durable, replayable, ordered log built from shards; the partition key that routes every record to a shard and fixes its ordering; the two consumer models (shared-throughput reads versus enhanced fan-out with HTTP/2 push) and the checkpointing that makes them resumable; Firehose as the fully-managed load-to-lake pipe that buffers and delivers to S3 and Redshift; kinesis analytics (now Managed Service for Apache Flink) for windowed stream compute; and the resharding operations that split hot shards and merge cold ones. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library →, rehearse on the real-time analytics practice library →, and sharpen the event side with the event processing practice library →.

---

On this page

• The Kinesis family — Data Streams, Firehose, and Analytics
• Data Streams — shards, partition keys, and ordering
• Consumers and enhanced fan-out
• Firehose — buffered delivery to S3 and Redshift
• Analytics, scaling, and resharding
• Cheat sheet — Amazon Kinesis recipes
• Frequently asked questions
• Practice on PipeCode

---

1. The Kinesis family — Data Streams, Firehose, and Analytics

One brand, three services — pick the wrong one and you rebuild the pipeline six months later

The one-sentence invariant: Amazon Kinesis is not one product but a family — Kinesis Data Streams is a durable, replayable, ordered log you read yourself; Amazon Data Firehose is a fully-managed delivery pipe with no replay that loads to S3, Redshift, and OpenSearch; and Managed Service for Apache Flink (formerly Kinesis Data Analytics) is stream compute that runs windowed SQL or Flink over a stream — and the interview signal is knowing which one owns ordering, which one owns replay, and which one you never have to scale by hand.

The mistake juniors make is treating "Kinesis" as a single noun. The mistake seniors avoid is pointing Firehose at a workload that needs replay, or standing up Data Streams plus a hand-written consumer for a workload that only ever needs "dump these logs into S3 as Parquet." The choice binds the downstream: a consumer that assumes ordered, replayable records cannot be retrofitted onto Firehose, and a team that picked Firehose for its zero-ops delivery cannot suddenly ask it to rewind three hours.

The three services at a glance.

• Kinesis Data Streams (KDS). A durable log partitioned into shards. Producers PutRecord/PutRecords; consumers read by shard and checkpoint their position. Ordering is guaranteed within a shard; records persist for 24 hours by default (extendable to 365 days), so consumers can replay. You own the consumer application (KCL, Lambda, Flink, or Firehose-as-consumer). This is the choice when you need low latency, multiple independent consumers, or replay.
• Amazon Data Firehose. A fully-managed delivery stream. You configure a source and a destination (S3, Redshift, OpenSearch, Splunk, generic HTTP), plus a buffer, and Firehose handles batching, retry, and delivery. There are no shards to manage and no replay — once a record is delivered it is gone from Firehose. This is the choice for load-to-lake / load-to-warehouse ETL where near-real-time (seconds to minutes) is fine.
• Managed Service for Apache Flink (Kinesis Data Analytics). Stream compute. It consumes a stream (usually KDS or MSK), runs tumbling/sliding/session windowed aggregations in SQL or Apache Flink, and writes results to a sink. This is the choice when you need to compute over the stream — running counts, top-N, anomaly scores — not just move it.

The four axes interviewers actually probe.

• Ordering and replay. Data Streams guarantees per-shard ordering (by sequence number) and lets consumers replay within the retention window. Firehose provides neither — it is a one-way delivery pipe. If the question mentions "replay," "reprocess," or "ordered per user," the answer is Data Streams, not Firehose.
• Unit of throughput. Data Streams throughput is measured in shards (1 MB/s or 1000 records/s ingress, 2 MB/s egress each) in provisioned mode, or auto-scaled in on-demand mode. Firehose throughput is a soft account limit you request increases against; there is no shard. Naming the shard as the atom of Data Streams throughput is a required senior answer.
• Latency. Data Streams is sub-second producer-to-consumer (≈200 ms typical; ≈70 ms with enhanced fan-out). Firehose is buffer-bound — seconds to minutes depending on buffer size/interval. Flink adds its own windowing latency. Match the freshness requirement to the service.
• Who manages scaling. Firehose and on-demand Data Streams auto-scale. Provisioned Data Streams makes you choose shard count and reshard. Managed Flink scales by KPUs. "Who owns the capacity decision" is the operational axis.

The 2026 reality — the family is complementary, not competitive.

• Data Streams is the backbone for anything with multiple consumers or replay: fraud scoring + dashboard + lake, all reading the same stream independently. It is also the source many teams put in front of Firehose.
• Firehose is the default load-to-lake pipe. Point it at a stream (or send to it directly), let it buffer to Parquet, and it lands partitioned objects in S3 or COPYs into Redshift with zero consumer code.
• Managed Flink is the compute layer when the answer is an aggregate, not a copy — sessionization, windowed counts, streaming joins.
• A very common production topology is producers → Data Streams → (a) a Flink app for real-time metrics, (b) a Firehose consumer for the S3/Redshift lake, and (c) a Lambda consumer for alerts — three consumers, one ordered replayable log.

What interviewers listen for.

• Do you name all three services and say which owns ordering and replay? — senior signal.
• Do you say "Firehose has no replay and no shards" the moment Firehose comes up? — required answer.
• Do you describe the shard as the unit of throughput and ordering for Data Streams? — required answer.
• Do you push back on "just use Kinesis" by asking "which one — do you need replay, or just delivery?" — senior signal.
• Do you describe a topology as one stream, many independent consumers rather than "a Kinesis pipe"? — senior signal.

Worked example — the three-service decision matrix

Detailed explanation. The single most useful artifact for a Kinesis interview is a memorised 3×N comparison of the three services against the axes that decide the pick. Every senior Kinesis discussion converges on this within the first ten minutes; having it in your head separates a fluent answer from a stumbling one. Walk through building it for a hypothetical clickstream that must feed a live dashboard, a fraud model, and a Parquet data lake.

• Source. A web/mobile SDK emitting ~50,000 click events/second at peak, each ~400 bytes.
• Downstream 1. Real-time dashboard — needs per-user ordering and sub-second freshness.
• Downstream 2. Fraud model — needs independent replayable access to the same events.
• Downstream 3. Data lake — needs the events landed in S3 as partitioned Parquet, minutes-fresh is fine.

Question. Assign a Kinesis service to each downstream and justify each pick against ordering, replay, latency, and who scales it.

Input.

Service	Ordering	Replay	Latency	Scaling unit
Data Streams	per shard	yes (24h–365d)	~200 ms (~70 ms EFO)	shard (or on-demand)
Firehose	none guaranteed	no	seconds–minutes	managed (soft limit)
Managed Flink	event-time windows	via source stream	window-bound	KPU

Code.

Topology decision (clickstream → 3 downstreams)
===============================================

Producers (SDK, 50k rec/s)
        │  PutRecords, partition key = user_id
        ▼
  Kinesis Data Streams  "clicks"   (ordered, replayable log)
        ├──► Managed Flink app      → live dashboard  (windowed counts)
        ├──► Lambda / KCL consumer  → fraud model     (independent replay)
        └──► Firehose (stream source) → S3 Parquet lake (buffered delivery)

Why one stream, three consumers:
  • Each consumer reads independently at its own position.
  • Fraud model can replay yesterday without touching the dashboard.
  • Firehose owns the S3 batching so no one writes lake plumbing.

Step-by-step explanation.

1. All three downstreams read the same Data Streams "clicks" stream. This is the core pattern: Data Streams is the durable ordered log, and each consumer maintains its own position, so they never interfere. The dashboard being slow does not slow the fraud model.
2. The dashboard downstream is a Managed Flink app because it needs a computed answer — clicks-per-minute per page — not the raw events. Flink runs a tumbling window over the stream and emits aggregates to the dashboard's store.
3. The fraud model reads via a KCL or Lambda consumer because it needs the raw ordered events and occasional replay (re-score yesterday after a model change). Data Streams' 24h+ retention makes that replay a position rewind, not a re-ingest.
4. The data lake downstream is Firehose configured with the stream as its source. Firehose buffers records, converts to Parquet, and lands partitioned objects in S3 — no consumer code, no S3 batching logic. Firehose's lack of replay is fine here because the lake is the durable copy.
5. The partition key is user_id so all of a user's clicks land on one shard in order — the dashboard and fraud model both depend on per-user ordering. Section 2 covers why that choice also risks a hot shard.

Output.

Downstream	Service	Why
Live dashboard	Managed Flink	needs windowed compute, not raw copy
Fraud model	Data Streams + KCL/Lambda	needs raw ordered events + replay
Data lake (S3 Parquet)	Firehose (stream source)	managed buffered delivery, no code
Backbone	Data Streams "clicks"	one ordered replayable log for all

Rule of thumb. Never pick a Kinesis service by name recognition. Pick Data Streams when you need ordering, replay, or multiple independent consumers; pick Firehose when you need managed delivery to a store and can live without replay; pick Managed Flink when the deliverable is a computed aggregate. Most real systems use Data Streams as the backbone and layer Firehose and Flink as consumers.

Worked example — what interviewers actually probe about the family

Detailed explanation. The senior Kinesis interview has a predictable arc: an ambiguous opener ("how would you stream events into our lake?"), then progressive narrowing to test whether you know which service owns which guarantee. Candidates who name the service and its guarantee score highest; candidates who say "a Kinesis pipeline" score lowest. Walk through the grading rubric.

• Ambiguous opener. "How do we get events into the warehouse in near-real-time?" — invites you to name Firehose or Data Streams + a consumer.
• Follow-up 1. "What if two teams need the same events?" — probes the multiple-consumers axis (Data Streams).
• Follow-up 2. "What if one team needs to reprocess last week?" — probes replay (Data Streams retention).
• Follow-up 3. "Who scales it at 3am?" — probes on-demand vs provisioned vs Firehose managed.

Question. Draft a two-minute answer that names the right service for each follow-up without being asked twice.

Input.

Interview signal	Weak answer	Senior answer
Service named	"a Kinesis stream"	"Data Streams for the log, Firehose for lake delivery"
Two consumers	"add another Lambda"	"each consumer reads the stream independently at its own position"
Replay last week	"re-ingest from source"	"extend retention; rewind the shard iterator to a timestamp"
Who scales	"we bump the shard count"	"on-demand mode auto-scales, or reshard in provisioned mode"

Code.

Two-minute Kinesis family answer
================================

"I'd land raw events in Kinesis Data Streams — it's the durable, ordered,
 replayable log. Ordering is per shard, keyed by partition key.

 Two teams needing the same events isn't a problem: each is an independent
 consumer reading the stream at its own checkpointed position, so they don't
 contend — with enhanced fan-out each even gets a dedicated 2 MB/s pipe.

 Reprocessing last week means extending retention (up to 365 days) and
 rewinding the shard iterator to AT_TIMESTAMP — no re-ingest from source.

 For the warehouse/lake, I'd attach a Firehose delivery stream with the
 Data Stream as its source: it buffers, converts to Parquet, and lands
 partitioned objects in S3 (or COPYs into Redshift) with no consumer code.

 Scaling: on-demand Data Streams auto-scales with traffic; if we're on
 provisioned, we UpdateShardCount or reshard. Firehose scales itself."

Step-by-step explanation.

1. The answer names Data Streams as the backbone in the first sentence and immediately attaches the two guarantees interviewers grade — ordered and replayable — plus the mechanism, partition key and shard.
2. The multiple-consumers follow-up is pre-empted with "each is an independent consumer reading at its own position." Mentioning enhanced fan-out here signals you know the throughput isolation story, not just the concept.
3. The replay follow-up is answered with the concrete lever — retention extension plus AT_TIMESTAMP iterator — instead of the weak "re-ingest from source," which throws away the whole point of a durable log.
4. The delivery half of the answer hands the lake to Firehose as a stream consumer, naming buffering, Parquet conversion, and S3 partitioning — the three things Firehose does so you don't write them.
5. The scaling close distinguishes on-demand (auto), provisioned + reshard (manual), and Firehose managed (auto) — three different owners of the capacity decision, which is exactly the operational axis the interviewer is testing.

Output.

Grading criterion	Weak score	Senior score
Names service + guarantee	rare	mandatory
Multiple independent consumers	occasional	mandatory
Replay via retention + iterator	rare	senior signal
Firehose owns lake batching	occasional	required
Names the scaling owner	rare	senior signal

Rule of thumb. The senior Kinesis answer is a two-minute monologue that assigns each requirement — ordering, replay, delivery, compute, scaling — to the service that owns it, without waiting for the follow-ups. Rehearse it once; deploy it every time the interviewer says "Kinesis."

Data engineering interview question on the Kinesis family

A senior interviewer often opens with: "A payments team streams ~30,000 authorization events/second. They need (a) a fraud service scoring every event in order per card, (b) an independent analytics team computing per-merchant approval rates in one-minute windows, and (c) every raw event landed in S3 as Parquet for the lake. Design the Kinesis topology, name the service for each need, and justify ordering, replay, and who scales it."

Solution Using Data Streams as the backbone with Flink and Firehose consumers

# 1. Backbone — one ordered, replayable log
Kinesis Data Streams  "payments-auth"
  partition key = card_id        # per-card ordering for fraud
  mode          = on-demand      # auto-scales with traffic spikes
  retention     = 7 days         # replay for reprocessing / audits

# 2. Fraud consumer — KCL/Lambda, per-card ordering, checkpointed
def handle_records(records, checkpointer):
    """Lambda/KCL handler: score each auth in shard order."""
    for r in records:                     # records arrive in-shard-order
        auth = json.loads(r["data"])
        score = fraud_model.score(auth)   # ordered per card_id
        if score > THRESHOLD:
            emit_alert(auth, score)
    checkpointer.checkpoint(records[-1]["sequenceNumber"])  # durable progress

-- 3. Analytics consumer — Managed Flink SQL, 1-minute tumbling window
CREATE TABLE auth_src (
    merchant_id  STRING,
    approved     BOOLEAN,
    event_time   TIMESTAMP(3),
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH ('connector'='kinesis', 'stream'='payments-auth', ...);

SELECT merchant_id,
       window_start,
       AVG(CASE WHEN approved THEN 1.0 ELSE 0.0 END) AS approval_rate
FROM TABLE(TUMBLE(TABLE auth_src, DESCRIPTOR(event_time), INTERVAL '1' MINUTE))
GROUP BY merchant_id, window_start, window_end;

// 4. Lake consumer — Firehose delivery stream with the Data Stream as source
{
  "DeliveryStreamName": "payments-auth-lake",
  "KinesisStreamSourceConfiguration": { "KinesisStreamARN": "arn:...:stream/payments-auth" },
  "ExtendedS3DestinationConfiguration": {
    "BucketARN": "arn:aws:s3:::payments-lake",
    "BufferingHints": { "SizeInMBs": 128, "IntervalInSeconds": 60 },
    "DataFormatConversionConfiguration": { "Enabled": true },   // JSON → Parquet
    "Prefix": "auth/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/"
  }
}

Step-by-step trace.

Step	Component	What happens
Ingest	Producers → payments-auth	PutRecords with partition key = card_id; on-demand mode absorbs the spike
Routing	Partition key	each card_id hashes to one shard → per-card order preserved
Fraud	KCL/Lambda consumer	reads shard in order, scores, checkpoints sequenceNumber
Analytics	Managed Flink	independent read; 1-min tumbling window on event_time
Lake	Firehose (stream source)	buffers 128 MB / 60 s, converts to Parquet, lands partitioned S3 objects
Replay	7-day retention	any consumer rewinds to AT_TIMESTAMP without re-ingest

After deployment, one ordered log feeds three independent consumers: the fraud service sees every authorization for a card in commit order, the analytics team computes per-merchant approval rates in one-minute windows without touching the fraud path, and Firehose lands raw Parquet in S3 — all replayable for seven days.

Output:

Need	Service	Ordering	Replay	Scaling
Fraud scoring	Data Streams + KCL/Lambda	per card_id	yes (7d)	on-demand
Per-merchant windows	Managed Flink	event-time window	via source	KPU
S3 Parquet lake	Firehose (stream source)	n/a	no (lake is durable copy)	managed
Backbone log	Data Streams payments-auth	per shard	yes	on-demand

Why this works — concept by concept:

• Data Streams as the backbone — a single durable, ordered, replayable log that every consumer reads independently at its own position. One ingest path, many fan-outs; no consumer contends with another.
• Partition key = card_id — routes every authorization for a card to the same shard, so the fraud model sees them in order. Ordering in Kinesis is per shard, and the partition key is the only lever that decides the shard.
• On-demand mode — the stream auto-scales shard count with traffic, so a payments spike does not throw ProvisionedThroughputExceededException. You trade a per-GB premium for zero capacity planning.
• Firehose stream-source delivery — Firehose consumes the same stream and owns all the lake plumbing: buffering, Parquet conversion, S3 partitioning. No hand-written batching, no replay expectation (the lake is the durable copy).
• Cost — one on-demand stream (per-GB + per-shard-hour equivalent), one Flink app (KPUs), one Firehose stream (per-GB delivered + conversion). O(1) producer cost per record; consumers scale independently. Compared to three separate ingest pipelines, this is one log read three ways.

Streaming
Topic — streaming
Streaming architecture and Kinesis topology problems

Practice →

Real-time Topic — real-time-analytics Real-time analytics fan-out problems

Practice →

---

2. Data Streams — shards, partition keys, and ordering

The shard is the atom of throughput and ordering — and the partition key decides which shard every record lands on

The mental model in one line: a Kinesis Data Stream is a set of shards, each an independent append-only log with fixed capacity (1 MB/s or 1000 records/s in, 2 MB/s out) and a guaranteed ordering by sequence number; the partition key you attach to every record is MD5-hashed to a 128-bit integer and mapped into a shard's hash-key range, so the partition key alone decides which shard a record lands on — which means ordering is per-shard, throughput scales by adding shards, and a skewed partition key concentrates traffic onto one hot shard while the rest sit idle. Every senior data engineer has sized a stream by shard count; every senior data engineer has debugged a hot shard; and getting the partition-key design right is what separates a stream that scales linearly from one that throttles at 5% of its paid capacity.

Shard capacity — the numbers you must have memorised.

• Ingress. Each shard accepts 1 MB/second or 1,000 records/second, whichever limit it hits first. A stream of 3,000 tiny 100-byte records/second is record-limited (needs 3 shards for the record cap even though it's only 300 KB/s); a stream of 5 MB/s of large records is byte-limited (needs 5 shards).
• Egress. Each shard serves 2 MB/second of reads in the shared-throughput model, shared across all classic consumers. Enhanced fan-out gives each registered consumer its own dedicated 2 MB/s per shard (section 3).
• API limits. GetRecords can be called up to 5 times/second per shard and returns up to 10 MB or 10,000 records per call. PutRecords batches up to 500 records or 5 MB per call. These per-shard caps are why throughput scales by shard count, not by hammering one shard harder.
• Stream capacity = shard count × per-shard capacity. Twenty shards = 20 MB/s or 20,000 records/s ingress. To go faster, add shards (reshard); there is no "bigger shard."

The partition key — the single most important design decision.

• What it does. Every PutRecord carries a PartitionKey string. Kinesis computes MD5(partition_key) → a 128-bit integer, and routes the record to whichever shard owns that value in its hash-key range. Same partition key → same shard → ordered together.
• Ordering guarantee. Records with the same partition key are delivered to a consumer in the order they were written (by SequenceNumber, which is monotonically increasing per shard). There is no cross-shard ordering — two records with different partition keys may land on different shards and be read in any interleaving.
• Cardinality matters. A high-cardinality, evenly-distributed key (user_id, device_id, order_id) spreads load across shards. A low-cardinality or skewed key (country, event_type, a constant) concentrates load — the classic hot-shard bug.
• ExplicitHashKey escape hatch. You can bypass the MD5 hash by supplying an ExplicitHashKey, pinning a record to a specific shard's range directly. Useful for deterministic shard assignment; rarely needed.

Hot shards and ProvisionedThroughputExceededException.

• The symptom. Producers get ProvisionedThroughputExceededException on writes, or a consumer's IteratorAgeMilliseconds climbs on one shard, while stream-level metrics show plenty of spare capacity. That is a hot shard: one shard is saturated because too many records share partition keys that hash into its range.
• The cause. Skewed partition keys. partition_key = "US" when 80% of traffic is US-based sends 80% of records to one shard regardless of how many shards you have.
• The fixes. (a) Redesign the partition key to a higher-cardinality field, or salt it (user_id + "#" + random(0..N)) if you can relax per-key ordering; (b) reshard — split the hot shard so its hash range is divided (section 5); (c) switch to on-demand mode, which redistributes and splits automatically based on observed traffic.

Retention, capacity modes, and durability.

• Retention. Default 24 hours; extendable up to 365 days (extended retention beyond 7 days is billed extra). Retention is what makes replay possible — a consumer can rewind its iterator within the window.
• Provisioned mode. You set the shard count and pay per shard-hour + per PUT payload unit. You own resharding. Predictable cost; manual scaling.
• On-demand mode. No shard management; the stream scales up to 2× the previous 30-day peak automatically, and you pay per GB ingested/retrieved. Simpler ops; higher per-GB cost. AWS still exposes shards under the hood (they matter for ordering and for consumers), but you don't size them.
• Durability. Every record is synchronously replicated across three Availability Zones before the PutRecord returns success. A record acknowledged is a record that survives an AZ loss.

Worked example — sizing a stream by shards

Detailed explanation. The canonical first question: given a throughput requirement, how many shards do you provision? The trap is forgetting the record-count limit and sizing only by bytes. Walk through both limits for an IoT telemetry stream.

• Traffic. 8,000 messages/second at peak, each ~250 bytes.
• Byte rate. 8,000 × 250 B = 2.0 MB/s.
• Record rate. 8,000 records/s.
• Headroom. Size for peak × 1.25 so a burst doesn't throttle.

Question. Compute the minimum shard count for the byte limit and the record limit, take the max, then add headroom.

Input.

Parameter	Value
Peak messages/sec	8,000
Bytes/message	250
Shard byte limit	1 MB/s
Shard record limit	1,000 rec/s
Headroom factor	1.25

Code.

# Shard sizing — take the max of the byte-bound and record-bound counts
import math

peak_rps      = 8000
bytes_per_rec = 250
HEADROOM      = 1.25

# Byte-bound: each shard = 1 MB/s = 1_048_576 bytes/s
byte_rate      = peak_rps * bytes_per_rec           # 2_000_000 B/s
shards_by_byte = math.ceil(byte_rate / 1_048_576)   # ceil(1.907) = 2

# Record-bound: each shard = 1000 records/s
shards_by_rec  = math.ceil(peak_rps / 1000)         # ceil(8.0)   = 8

# The stream needs the larger of the two, plus headroom
base_shards = max(shards_by_byte, shards_by_rec)     # 8
shards      = math.ceil(base_shards * HEADROOM)      # ceil(10.0)  = 10

print(f"byte-bound={shards_by_byte}  record-bound={shards_by_rec}  provision={shards}")
# → byte-bound=2  record-bound=8  provision=10

Step-by-step explanation.

1. The byte-bound calculation says 2 MB/s ÷ 1 MB/s per shard = 2 shards would carry the bytes. If you sized on bytes alone you'd provision 2 shards — and immediately throttle.
2. The record-bound calculation is the binding constraint here: 8,000 records/s ÷ 1,000 records/s per shard = 8 shards. Small records make this stream record-limited, not byte-limited. Missing this is the single most common shard-sizing mistake.
3. Taking max(2, 8) = 8 respects both limits. A shard throttles when either limit is exceeded, so you must satisfy the tighter one.
4. Applying 1.25 headroom → 10 shards. Streams see bursts above the average peak; provisioning at exactly the peak means every burst throttles a producer and inflates IteratorAge.
5. If traffic were unpredictable, the alternative is on-demand mode: no shard math, auto-scaling to 2× the trailing peak, at a higher per-GB rate. For steady, well-understood traffic, provisioned + this sizing is cheaper.

Output.

Bound	Formula	Shards
Byte	ceil(2.0 MB/s ÷ 1 MB/s)	2
Record	ceil(8000 ÷ 1000)	8
Max of both	max(2, 8)	8
With 1.25 headroom	ceil(8 × 1.25)	10

Rule of thumb. Always size a shard count against both the 1 MB/s byte limit and the 1,000 records/s record limit, take the larger, then add 20–30% headroom. Small-record streams are almost always record-bound; large-payload streams are byte-bound. When in doubt or when traffic is spiky, use on-demand mode and skip the arithmetic.

Worked example — partition-key hashing and ordering

Detailed explanation. To reason about ordering and hot shards you must see how a partition key becomes a shard assignment. Kinesis MD5-hashes the key to a 128-bit integer and matches it against each shard's StartingHashKey/EndingHashKey range. Walk through hashing three keys against a two-shard stream.

• Stream. 2 shards. The 128-bit hash space (0 … 2^128−1) is split in half: shard-0 owns [0, 2^127−1], shard-1 owns [2^127, 2^128−1].
• Keys. user-42, user-99, user-42 again.
• Goal. Show that same key → same shard → ordered, and different keys may split.

Question. Compute which shard each key routes to and confirm the two user-42 records stay ordered on one shard.

Input.

Record	Partition key	Order written
r1	user-42	1
r2	user-99	2
r3	user-42	3

Code.

# How Kinesis maps a partition key to a shard (illustrative)
import hashlib

NUM_SHARDS = 2
HASH_SPACE = 1 << 128                      # 2^128 possible hash values
SHARD_WIDTH = HASH_SPACE // NUM_SHARDS     # each shard owns an equal slice

def shard_for(partition_key: str) -> int:
    digest = hashlib.md5(partition_key.encode()).hexdigest()
    hash_val = int(digest, 16)             # 128-bit integer
    return min(hash_val // SHARD_WIDTH, NUM_SHARDS - 1)

for key in ["user-42", "user-99", "user-42"]:
    print(f"{key:8s} -> shard-{shard_for(key)}")
# → user-42  -> shard-0
# → user-99  -> shard-1
# → user-42  -> shard-0     (same key, same shard, stays ordered)

Step-by-step explanation.

1. MD5("user-42") produces a fixed 128-bit integer. Because MD5 is deterministic, every record tagged user-42 hashes to the identical value and therefore the identical shard — this is the mechanism behind the ordering guarantee.
2. That hash value falls into shard-0's range, so both user-42 records (r1 and r3) route to shard-0. Within shard-0 they are assigned monotonically increasing SequenceNumbers, so a consumer reads r1 before r3 — order preserved.
3. user-99 hashes into shard-1's range and routes there. r2 is now on a different shard from r1/r3. Kinesis makes no guarantee about the relative order of r1 (shard-0) and r2 (shard-1) as seen across shards — cross-shard ordering does not exist.
4. This is exactly why per-entity ordering requires putting the entity id in the partition key: to keep a user's events ordered, the partition key must be user_id, so all of that user's events share a shard.
5. The flip side: if 80% of traffic used the same key, 80% would pile onto one shard. Even ordering-correct keys can be hot if their value distribution is skewed — the ordering requirement and the load-balancing requirement can conflict, and section 5's resharding plus key-salting resolve it.

Output.

Record	Key	Shard	Sequence order on shard
r1	user-42	shard-0	1st on shard-0
r3	user-42	shard-0	2nd on shard-0 (after r1)
r2	user-99	shard-1	1st on shard-1

Rule of thumb. Put the entity you need ordered (user, device, account) in the partition key so its records share a shard and stay ordered. Verify the key's value distribution is high-cardinality and even before shipping — an ordering-correct but skewed key is a hot shard waiting to happen.

Worked example — diagnosing a hot shard

Detailed explanation. A 12-shard stream is provisioned for 12 MB/s but producers throw ProvisionedThroughputExceededException while stream-level IncomingBytes reads only ~3 MB/s. That contradiction — throttling below paid capacity — is the signature of a hot shard. Walk through the diagnosis using per-shard CloudWatch metrics.

• Symptom. Write throttling + rising GetRecords.IteratorAgeMilliseconds on one shard.
• Evidence. Enable enhanced (shard-level) monitoring; compare IncomingBytes and IncomingRecords per shard.
• Root cause. The partition key is event_country; the "US" value carries most of the traffic and hashes into one shard's range.

Question. Use per-shard metrics to confirm the hot shard, then pick a remediation that preserves the ordering the app needs.

Input.

Shard	IncomingBytes/s	% of stream
shard-000000 (US)	950 KB/s	~78% (throttling)
shard-000001	60 KB/s	~5%
shard-000002…011	~30 KB/s each	~17% total

Code.

# Salt a skewed partition key while preserving per-entity ordering where it matters
import random

# BEFORE — every US event collides on one shard
def key_before(event):
    return event["country"]                    # "US" for ~78% of traffic → hot shard

# AFTER — keep ordering per user (the real ordering need), not per country
def key_after(event):
    return event["user_id"]                    # high-cardinality → even spread

# If ordering must stay per-country but load must spread, salt with N buckets
# (accepts N-way interleaving within a country; consumer re-sorts if needed)
N_SALT = 16
def key_salted(event):
    bucket = random.randint(0, N_SALT - 1)
    return f'{event["country"]}#{bucket}'      # "US#0".."US#15" → 16 shards

# Confirm with the AWS CLI — per-shard metrics after enhanced monitoring is on
aws cloudwatch get-metric-statistics \
  --namespace AWS/Kinesis --metric-name IncomingBytes \
  --dimensions Name=StreamName,Value=events Name=ShardId,Value=shardId-000000000000 \
  --statistics Sum --period 60 --start-time ... --end-time ...

Step-by-step explanation.

1. Stream-level metrics hid the problem: 3 MB/s across a 12 MB/s stream looks healthy. The fix is enabling shard-level (enhanced) monitoring, which exposes IncomingBytes/IncomingRecords per shard and reveals that shard-000000 carries ~78% of the load.
2. The root cause is the partition key country: "US" is one string, so MD5("US") always lands in shard-000000's range. No amount of extra shards helps, because the hash of a constant string never moves.
3. The best fix is usually to change the key to the field that actually needs ordering — here user_id. Per-country ordering was never a real requirement; per-user was. user_id is high-cardinality and spreads evenly.
4. If per-country ordering genuinely matters, salt the key: append a random bucket (US#0…US#15) to spread "US" across 16 shards, accepting that events within a country are now interleaved across buckets and a consumer must re-sort if it needs strict per-country order.
5. Resharding (split shard-000000) alone does not fix a constant key: splitting divides the hash range, but MD5("US") still falls in exactly one of the two halves. Resharding helps only once the key distribution is fixed. This is the subtle senior insight.

Output.

Remediation	Fixes hot shard?	Ordering preserved
Add shards, same key	no (constant key stays on one shard)	n/a
Split hot shard, same key	no (MD5 of constant still one range)	n/a
Repartition on user_id	yes (high-cardinality)	per user (the real need)
Salt country#bucket	yes (spreads across N shards)	per country only after re-sort
Switch to on-demand	partial (auto-splits by load)	per key still applies

Rule of thumb. A hot shard is almost always a partition-key problem, not a shard-count problem. Turn on shard-level monitoring, find the skewed key value, and fix the key (repartition or salt) before you reshard — splitting a shard whose skew comes from a constant key changes nothing.

Data engineering interview question on Data Streams sizing and keys

A senior interviewer might ask: "You run a 20-shard provisioned stream keyed on store_id. A flagship store's Black Friday traffic saturates its shard and producers throttle, while the stream sits at 40% overall. The app requires per-store ordering. Walk me through the diagnosis, a remediation that keeps per-store ordering, the shard math, and how you'd prevent it next year."

Solution Using shard-level monitoring, a targeted split, and a composite key

# 1. Confirm the hot shard from per-shard metrics (enhanced monitoring ON)
#    shardId-...0007 carries ~55% of IncomingRecords → the flagship store's shard

# 2. Keep per-store ordering but relieve the ONE hot store with a composite key.
#    Normal stores: key = store_id  (unchanged ordering & spread)
#    Flagship store: key = store_id + "#" + (order_id % K)   (K sub-streams)
FLAGSHIP = "store-0007"
K = 8
def partition_key(event):
    sid = event["store_id"]
    if sid == FLAGSHIP:
        return f'{sid}#{event["order_id"] % K}'   # spread flagship across K shards
    return sid                                     # everyone else unchanged

# 3. Split the flagship's hot shard so the new composite-keyed traffic has room.
#    Find the shard's hash range, split at its midpoint to create two children.
aws kinesis split-shard \
  --stream-name orders \
  --shard-to-split shardId-000000000007 \
  --new-starting-hash-key <midpoint-of-parent-range>

# 4. Consumers must drain the PARENT shard to SHARD_END before reading children,
#    so no record is read out of lineage order (KCL handles this automatically).

# 5. Downstream re-merge: the flagship's K sub-keys interleave, so if the
#    consumer needs strict per-store order, buffer by store and sort by
#    (event_time, sequence_number) within a small window before processing.
def ordered_by_store(batch):
    from collections import defaultdict
    buckets = defaultdict(list)
    for r in batch:
        buckets[r["store_id"]].append(r)
    for sid, rows in buckets.items():
        rows.sort(key=lambda r: (r["event_time"], r["sequence_number"]))
        yield sid, rows

Step-by-step trace.

Step	Action	Result
Diagnose	shard-level IncomingRecords	shard-...0007 = 55% of stream (flagship)
Key	composite store#bucket for flagship only	flagship spreads across K=8 shard slots
Split	split-shard on the hot shard	parent → two children; hash range halved
Lineage	drain parent to SHARD_END first	children read only after parent done → no reorder
Re-merge	consumer sorts flagship by (event_time, seq)	strict per-store order restored downstream

After the change, the flagship store's Black Friday load fans across eight logical sub-keys and two physical child shards, the 19 normal stores keep their simple store_id key and per-store ordering untouched, and a small consumer-side sort restores strict flagship ordering. Overall stream utilisation flattens from a 55%/idle split to an even spread.

Output:

Metric	Before	After
Hot shard share	~55% (throttling)	~15% (even)
Producer throttling	frequent on flagship	none
Per-store ordering	native	native (normal) / re-sorted (flagship)
Shards touched	1 saturated	1 split into 2
Blast radius of one store	whole shard	bounded to K sub-keys

Why this works — concept by concept:

• Shard-level monitoring — stream-level metrics average away the skew; enhanced per-shard IncomingRecords/IncomingBytes is the only way to see which shard is hot. Diagnosis precedes remediation.
• Composite partition key — appending order_id % K to only the flagship's key spreads its load across K logical sub-streams without touching the other 19 stores' clean ordering. Surgical, not global.
• split-shard — dividing the hot shard's hash range gives the newly-spread traffic physical room. Splitting after fixing the key works because the composite key now hashes across the new child ranges.
• Parent → child lineage to SHARD_END — a consumer must finish the parent shard before reading its children, or records could be processed out of lineage order. KCL enforces this automatically; a hand-rolled consumer must too.
• Cost — one extra shard (the split) plus a small consumer-side sort buffer for the flagship. O(1) per record on the producer; the re-sort is O(m log m) over a tiny per-window batch. Compared to over-provisioning the whole stream for one store's peak, this is targeted and cheap.

Streaming
Topic — streaming
Shard sizing and partition-key problems

Practice →

Events Topic — event-processing Event ordering and keyed-partition problems

Practice →

---

3. Consumers and enhanced fan-out

Shared throughput splits one 2 MB/s pipe across every consumer; enhanced fan-out gives each consumer its own — and checkpointing makes both resumable

The mental model in one line: a Kinesis consumer reads a shard either in the shared throughput model (classic GetRecords polling, where all consumers of a shard split a single 2 MB/s egress budget and 5 calls/second) or with enhanced fan-out, where each registered consumer gets a dedicated 2 MB/s per shard delivered via an HTTP/2 SubscribeToShard push at ~70 ms latency — and either way the consumer's durable progress lives in a checkpointing record (the last processed SequenceNumber) so a restart resumes exactly where it left off instead of replaying from the start. Every senior data engineer has hit the "second consumer starved the first" wall that enhanced fan-out solves, and every one has debugged a consumer that reprocessed a day of data because checkpointing was wrong.

Shared throughput — the default polling model.

• How it reads. A consumer gets a ShardIterator, then calls GetRecords in a loop. Each shard serves 2 MB/second of egress shared across all shared-throughput consumers and allows 5 GetRecords calls/second. Two consumers on one shard each effectively get ~1 MB/s.
• The contention problem. Add a third and fourth consumer and they all fight over the same 2 MB/s and the same 5 calls/s. GetRecords starts returning ProvisionedThroughputExceededException (read-side throttling) and latency climbs. Shared throughput does not isolate consumers.
• Latency. Polling-bound: with 5 calls/s the best-case poll cadence is ~200 ms, and under contention it's worse. Fine for batch-ish consumers (Firehose, a lake loader); painful for latency-sensitive ones.
• Cost. No per-consumer charge — you pay for the stream (shards or on-demand GB). Cheapest when you have one or two consumers.

Enhanced fan-out (EFO) — dedicated per-consumer throughput.

• How it reads. A consumer registers against the stream (RegisterStreamConsumer) and calls SubscribeToShard, which opens an HTTP/2 connection. Kinesis then pushes records to that consumer at a dedicated 2 MB/second per shard, independent of every other consumer.
• Isolation. Each registered EFO consumer gets its own 2 MB/s per shard. Five EFO consumers on a 10-shard stream each get 10 × 2 MB/s = 20 MB/s, with no contention. Up to 20 registered consumers per stream can use EFO.
• Latency. ~70 ms typical, because records are pushed as they arrive rather than polled. This is the reason to pay for EFO on a latency-sensitive consumer.
• Cost. Billed per consumer-shard-hour plus per GB retrieved via EFO. It is genuinely more expensive; you justify it when you have ≥3 consumers or a strict latency SLA.

The KCL, leases, and checkpointing — how a consumer stays correct across restarts.

• The Kinesis Client Library (KCL). The standard consumer framework. It assigns one worker to each shard, coordinates ownership through a lease table in DynamoDB, and calls your record processor. When shards split/merge, KCL rebalances leases automatically.
• Checkpointing. Your processor periodically calls checkpoint(sequenceNumber). KCL writes that sequence number into the DynamoDB lease row for the shard. On restart (or when another worker takes the lease), processing resumes from the checkpoint — not from the beginning.
• Delivery semantics. Kinesis + KCL is at-least-once: if a worker dies after processing but before checkpointing, the next owner reprocesses from the last checkpoint. Consumers must be idempotent (dedupe by a record id) to get effectively-once behaviour.
• Checkpoint cadence trade-off. Checkpoint too often → DynamoDB write cost and throttling on the lease table. Checkpoint too rarely → more reprocessing on failure. Typical practice: checkpoint per batch, not per record.

Iterator types — where a consumer starts reading.

• TRIM_HORIZON — start at the oldest record still in the shard (full replay of the retention window).
• LATEST — start at the newest record; only new arrivals. Skips history.
• AT_SEQUENCE_NUMBER / AFTER_SEQUENCE_NUMBER — start at (or after) a specific sequence number — the checkpoint-resume mechanism.
• AT_TIMESTAMP — start at a wall-clock time within retention — the "replay from 2pm yesterday" lever.

Worked example — a KCL consumer with per-batch checkpointing

Detailed explanation. The canonical resumable consumer: a KCL record processor that handles a batch of records, does idempotent work, and checkpoints once per batch. The DynamoDB lease table makes it resumable and rebalance-safe. Walk through the processor.

• Framework. KCL v2 (Java/Python via the MultiLangDaemon).
• Checkpoint policy. Once per batch, after the batch's work is durable.
• Idempotency. Dedupe by record_id so at-least-once redelivery is harmless.

Question. Write the process_records handler that processes a batch idempotently and checkpoints the last sequence number.

Input.

Element	Value
Library	KCL v2
Lease/checkpoint store	DynamoDB
Checkpoint cadence	per batch
Delivery semantics	at-least-once → idempotent consumer

Code.

# KCL v2 record processor — per-batch idempotent processing + checkpoint
class OrderProcessor:
    def initialize(self, init_input):
        self.shard_id = init_input.shard_id

    def process_records(self, process_records_input):
        records = process_records_input.records
        last_seq = None
        for r in records:
            payload = json.loads(r.binary_data)
            # Idempotency: skip if we've already applied this record_id
            if not already_applied(payload["record_id"]):
                apply_effect(payload)                 # the real work
                mark_applied(payload["record_id"])    # dedupe marker (durable)
            last_seq = r.sequence_number

        # Checkpoint ONCE per batch, after the batch's effects are durable.
        # KCL writes last_seq into the shard's DynamoDB lease row.
        if last_seq is not None:
            process_records_input.checkpointer.checkpoint(last_seq)

    def lease_lost(self, lease_lost_input):
        pass   # another worker took the shard; do NOT checkpoint here

    def shard_ended(self, shard_ended_input):
        # Parent shard fully drained (after a split/merge) — checkpoint SHARD_END
        shard_ended_input.checkpointer.checkpoint()

Step-by-step explanation.

1. KCL hands the processor a batch of records already in shard order. The loop processes each, tracking last_seq — the highest sequence number in the batch — which is what we'll checkpoint.
2. Each record is applied only if already_applied(record_id) is false. Because Kinesis is at-least-once, the same record can be redelivered after a failed checkpoint; the dedupe marker turns at-least-once into effectively-once.
3. The checkpoint happens once, after the batch loop, not per record. Per-record checkpointing would hammer the DynamoDB lease table and throttle. Per-batch is the standard cadence — it bounds reprocessing to one batch on failure.
4. checkpoint(last_seq) writes the sequence number into the shard's lease row in DynamoDB. If this worker dies, the next worker to claim the lease resumes with AFTER_SEQUENCE_NUMBER = last_seq, replaying at most the un-checkpointed tail.
5. shard_ended checkpoints SHARD_END when a parent shard is fully drained after a reshard — this signals KCL that the parent is done and its child shards may now be read (the lineage rule from section 2).

Output.

Event	Checkpoint written	Resume point after crash
Batch of 500 processed	last_seq of batch	AFTER_SEQUENCE_NUMBER(last_seq)
Crash mid-batch (no checkpoint)	none	previous checkpoint → whole batch replays
Redelivered record	n/a	dedupe marker skips it
Parent shard drained	SHARD_END	children become readable

Rule of thumb. Checkpoint once per batch after the batch's effects are durable, make every consumer idempotent (dedupe by a record id) because Kinesis is at-least-once, and let KCL own the DynamoDB lease table. Per-record checkpointing throttles DynamoDB; never-checkpointing replays the world on every restart.

Worked example — shared throughput vs enhanced fan-out throughput budget

Detailed explanation. The decision between shared and enhanced fan-out is a throughput-and-latency budget. Walk through a 10-shard stream that grows from one consumer to four to see exactly where shared throughput starves and EFO earns its cost.

• Stream. 10 shards → 20 MB/s total egress in the shared model.
• Consumers. Start with 1 (a lake loader), add a dashboard, a fraud model, and an alerting service.
• Question at each step. Does each consumer get the throughput and latency it needs?

Question. Compute per-consumer throughput under shared vs EFO as consumers scale from 1 to 4, and decide the switch point.

Input.

Consumers	Shared per-consumer	EFO per-consumer	Shared latency	EFO latency
1	20 MB/s	20 MB/s	~200 ms	~70 ms
2	10 MB/s	20 MB/s	~200 ms+	~70 ms
3	~6.7 MB/s	20 MB/s	rising	~70 ms
4	~5 MB/s	20 MB/s	throttling	~70 ms

Code.

# Throughput each consumer actually gets, shared vs enhanced fan-out
SHARDS      = 10
PER_SHARD   = 2               # MB/s egress per shard
STREAM_EGRESS = SHARDS * PER_SHARD    # 20 MB/s shared budget

def shared_per_consumer(n_consumers):
    # ALL shared consumers split the SAME 20 MB/s
    return STREAM_EGRESS / n_consumers

def efo_per_consumer(n_consumers):
    # EACH EFO consumer gets a dedicated 2 MB/s PER shard
    return SHARDS * PER_SHARD          # 20 MB/s, independent of others

for n in (1, 2, 3, 4):
    print(f"{n} consumers: shared={shared_per_consumer(n):.1f} MB/s  "
          f"efo={efo_per_consumer(n):.1f} MB/s each")
# → 1 consumers: shared=20.0 MB/s  efo=20.0 MB/s each
# → 2 consumers: shared=10.0 MB/s  efo=20.0 MB/s each
# → 3 consumers: shared= 6.7 MB/s  efo=20.0 MB/s each
# → 4 consumers: shared= 5.0 MB/s  efo=20.0 MB/s each

Step-by-step explanation.

1. With one consumer, shared and EFO are identical in throughput (20 MB/s). The only difference is latency: EFO's push gives ~70 ms vs polling's ~200 ms. If latency doesn't matter, one consumer should stay shared and save the EFO fee.
2. With two consumers, shared throughput halves each to 10 MB/s because they split the same 20 MB/s stream budget. EFO consumers still each get their own 20 MB/s. If both consumers need close to full throughput, shared already starves them.
3. By three or four consumers, shared per-consumer throughput falls to ~6.7 and ~5 MB/s, and GetRecords read-throttling appears as consumers compete for the 5-calls/s-per-shard limit. This is the starvation wall.
4. EFO is flat at 20 MB/s per consumer regardless of consumer count (up to the 20-consumer registration cap), because each has a dedicated pipe. The trade is a per-consumer-shard-hour + per-GB-retrieved charge.
5. The switch point is typically ≥3 consumers or a strict latency SLA. Below that, shared is cheaper and adequate; at or above it, EFO's isolation is worth the money because the alternative is throttling and lag.

Output.

Consumers	Recommended model	Reason
1 (latency-tolerant)	shared	cheapest; full throughput
1 (latency SLA ~70 ms)	EFO	push latency
2	shared or EFO	shared halves throughput; EFO isolates
3+	EFO	shared starves; read-throttling

Rule of thumb. Stay on shared throughput for one or two latency-tolerant consumers; switch to enhanced fan-out at three or more consumers or whenever a consumer has a sub-100 ms latency SLA. EFO buys per-consumer isolation and ~70 ms push latency at a per-consumer-hour + per-GB price — worth it exactly when contention or latency would otherwise bite.

Worked example — replay with iterator types

Detailed explanation. A model change requires re-scoring every event from 2pm yesterday forward, while the live consumer keeps running on new data. The lever is a separate consumer started with an AT_TIMESTAMP iterator, reading the retained history independently. Walk through the replay.

• Requirement. Reprocess from 2026-09-04T14:00:00Z without disturbing the live consumer.
• Retention. Stream retention ≥ the replay distance (extend to 7 days if needed).
• Mechanism. A new consumer (or a reset checkpoint) using AT_TIMESTAMP.

Question. Start a replay consumer at a timestamp and explain why it doesn't affect the live one.

Input.

Element	Value
Replay start	2026-09-04T14:00:00Z
Iterator type	AT_TIMESTAMP
Live consumer	untouched, own checkpoint
Isolation	EFO (separate registered consumer)

Code.

# Start a replay reader at a timestamp — independent of the live consumer
import boto3, datetime as dt

kinesis = boto3.client("kinesis")

def replay_shard(stream, shard_id, since: dt.datetime):
    it = kinesis.get_shard_iterator(
        StreamName=stream,
        ShardId=shard_id,
        ShardIteratorType="AT_TIMESTAMP",      # start at wall-clock time
        Timestamp=since,
    )["ShardIterator"]

    while it:
        resp = kinesis.get_records(ShardIterator=it, Limit=10000)
        for r in resp["Records"]:
            rescore(json.loads(r["Data"]))     # re-run the NEW model
        it = resp.get("NextShardIterator")     # None at SHARD_END
        if not resp["Records"] and resp["MillisBehindLatest"] == 0:
            break                              # caught up to now → stop replay

replay_shard("payments-auth", "shardId-000000000000",
             dt.datetime(2026, 9, 4, 14, 0, tzinfo=dt.timezone.utc))

Step-by-step explanation.

1. The replay uses ShardIteratorType="AT_TIMESTAMP" with Timestamp=2026-09-04T14:00Z. Kinesis returns an iterator positioned at the first record on/after that time — provided it is still within the retention window (extend retention first if the replay reaches further back than current retention).
2. This is a separate consumer with its own iterator/checkpoint. Data Streams lets any number of consumers read the same shard at different positions, so the replay reader and the live reader coexist — the live consumer's checkpoint is never touched.
3. MillisBehindLatest tells the replay how far behind "now" it is. When it returns records-empty and MillisBehindLatest == 0, the replay has caught up to the live tip and can stop.
4. If the two consumers would contend on the shared 2 MB/s budget, register the replay as an EFO consumer so it gets its own dedicated pipe and the live consumer's throughput is unaffected.
5. Because processing is at-least-once and the rescore is idempotent (keyed by event id + model version), the replay can safely overlap events the live consumer already scored under the old model — the new score simply supersedes by version.

Output.

Reader	Iterator	Position	Effect on live
Live consumer	AFTER_SEQUENCE_NUMBER	its checkpoint	none
Replay consumer	AT_TIMESTAMP(2pm)	2pm yesterday → now	none (own position)
Isolation	EFO registration	dedicated 2 MB/s	no throughput contention
Stop condition	MillisBehindLatest==0	caught up	replay ends

Rule of thumb. Replay is a position operation, not a re-ingest: extend retention to cover the window, start a separate consumer with AT_TIMESTAMP (or reset a checkpoint to AFTER_SEQUENCE_NUMBER), register it for enhanced fan-out so it doesn't steal the live consumer's throughput, and rely on idempotency to make overlapping reprocessing safe.

Data engineering interview question on consumers and fan-out

A senior interviewer might ask: "A 16-shard stream feeds one lake loader today. Product wants to add a real-time dashboard (sub-100 ms) and a fraud model, and next quarter a fifth consumer. Reads are already occasionally throttling. Walk me through shared vs enhanced fan-out, the checkpointing model, the delivery semantics, and how you'd guarantee the fraud model never reprocesses a scored event."

Solution Using enhanced fan-out registration plus KCL checkpointing and idempotency

# 1. Register latency/throughput-sensitive consumers for enhanced fan-out.
#    Each gets a dedicated 2 MB/s per shard, HTTP/2 push (~70ms), no contention.
import boto3
kinesis = boto3.client("kinesis")
for name in ["dashboard", "fraud-model", "alerting"]:
    kinesis.register_stream_consumer(
        StreamARN="arn:aws:kinesis:...:stream/events",
        ConsumerName=name,             # up to 20 EFO consumers per stream
    )
# The lake loader stays on shared throughput (latency-tolerant, saves cost).

# 2. Fraud model — KCL processor with per-batch checkpoint + idempotency
class FraudProcessor:
    def process_records(self, inp):
        last_seq = None
        for r in inp.records:
            ev = json.loads(r.binary_data)
            key = (ev["event_id"], MODEL_VERSION)          # idempotency key
            if not seen.putIfAbsent(key):                  # atomic dedupe
                score = model.score(ev)                    # exactly-once effect
                publish(ev, score)
            last_seq = r.sequence_number
        if last_seq:
            inp.checkpointer.checkpoint(last_seq)          # once per batch

# 3. Delivery semantics contract
#    Kinesis + KCL = at-least-once. A crash after processing but before
#    checkpoint replays the tail batch. The (event_id, MODEL_VERSION) dedupe
#    marker makes reprocessing a no-op → effectively-once for the fraud effect.

Step-by-step trace.

Step	Action	Result
Register	EFO for dashboard, fraud, alerting	each gets dedicated 2 MB/s/shard, ~70 ms
Lake loader	stays shared	latency-tolerant; no EFO fee
Read throttling	gone for EFO consumers	dedicated pipes remove contention
Fraud checkpoint	per batch → DynamoDB lease	resume = AFTER_SEQUENCE_NUMBER
Crash mid-batch	tail replays	dedupe by (event_id, version) → no double score

After the change, the dashboard hits its sub-100 ms SLA on a dedicated EFO pipe, the fraud model and alerting each get isolated throughput with no read-throttling, the lake loader stays on cheap shared throughput, and the fraud effect is effectively-once because at-least-once redelivery is absorbed by the idempotency marker.

Output:

Consumer	Model	Latency	Reprocess safety
Dashboard	EFO	~70 ms	n/a (read-only)
Fraud model	EFO + KCL	~70 ms	effectively-once via dedupe
Alerting	EFO	~70 ms	idempotent alerts
Lake loader	shared	~200 ms	idempotent S3 keys

Why this works — concept by concept:

• Enhanced fan-out registration — each latency-sensitive consumer gets a dedicated 2 MB/s per shard over an HTTP/2 push, so adding a third, fourth, and fifth consumer never starves the others. Up to 20 registered consumers per stream.
• Shared throughput for the tolerant consumer — the lake loader doesn't need ~70 ms, so it stays on the free shared model and saves the per-consumer-hour EFO charge. Right-size the model per consumer.
• Per-batch KCL checkpointing — durable progress in the DynamoDB lease table bounds reprocessing to one batch on failure, without the write-throttling of per-record checkpoints.
• Idempotency by (event_id, model_version) — turns Kinesis's at-least-once delivery into an effectively-once effect, so a crash-and-replay never double-scores an event. Semantics are a consumer responsibility, not a Kinesis guarantee.
• Cost — EFO adds per-consumer-shard-hour + per-GB-retrieved for three consumers; the lake loader stays free-tier shared. O(1) per record; the dedupe check is O(1) against a keyed store. Compared to over-provisioning shards to fake isolation, EFO is the correct, bounded spend.

Events
Topic — event-processing
Consumer checkpointing and delivery-semantics problems

Practice →

Streaming Topic — streaming Enhanced fan-out and throughput problems

Practice →

---

4. Firehose — buffered delivery to S3 and Redshift

Firehose is the zero-ops delivery pipe — it buffers records by size or time, optionally transforms and converts them, then lands them in S3, Redshift, or OpenSearch

The mental model in one line: Amazon Data firehose is a fully-managed delivery stream with no shards and no replay — you configure a source and a destination, a buffer defined by size (1–128 MB) and interval (60–900 s), an optional Lambda transform, and optional record-format conversion to Parquet/ORC, and Firehose batches incoming records until the buffer fills by size or age (whichever first), then delivers the batch to S3, Redshift (via a COPY through S3), OpenSearch, Splunk, or an HTTP endpoint — retrying failures and writing un-deliverable records to an error prefix, all without a single line of consumer code. Every senior data engineer reaches for Firehose the moment the requirement is "land these events in the lake/warehouse as files" rather than "give me a replayable ordered log."

The buffer — the one knob that defines Firehose latency and file size.

• Two limits, OR semantics. Firehose accumulates records until buffer size (1–128 MB for S3) or buffer interval (60–900 seconds) is reached — whichever comes first. High-throughput streams flush on size; low-throughput streams flush on interval.
• The freshness/file-size trade-off. A small buffer (1 MB / 60 s) means fresher data but many tiny files (the "small-files problem" that murders downstream query performance). A large buffer (128 MB / 900 s) means efficient, query-friendly files but up to 15-minute-old data. You tune the buffer to the freshness SLA and the query engine's ideal file size.
• Dynamic sizing. Firehose can automatically raise the buffer size to improve delivery efficiency during high throughput; you set the baseline.
• No shards. There is nothing to reshard. Firehose scales its own throughput; you request account-limit increases if you exceed the default rate.

Transformation and format conversion — the in-flight processing.

• Lambda transform. You attach a Lambda that receives a batch of records and returns them transformed (enriched, filtered, reformatted). Each returned record is marked Ok, Dropped, or ProcessingFailed. Failed/oversized transforms go to the error prefix.
• Record-format conversion. Firehose can convert incoming JSON to Parquet or ORC columnar formats using a Glue table schema — the single biggest win for a lake destination, because columnar Parquet is what Athena/Redshift Spectrum/Spark want.
• Dynamic partitioning. Firehose can partition S3 objects by values extracted from the record (e.g. customer_id, or a date parsed from the payload) using JQ expressions, writing to prefixes like s3://bucket/cust=123/dt=2026-09-05/. This makes downstream partition-pruning queries cheap.
• Compression. GZIP/Snappy/ZIP for raw formats; Parquet/ORC carry their own compression.

Destinations — where Firehose lands data.

• S3. The most common: buffered objects written to a prefix, optionally Parquet, optionally dynamically partitioned. The Prefix supports timestamp namespacing (year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/).
• Redshift. Firehose stages the batch in S3 first, then issues a Redshift COPY to load it. You supply the cluster, table, and COPY options. The intermediate S3 bucket is part of the delivery, not optional.
• OpenSearch / Splunk / HTTP. Direct delivery to search and observability sinks, and to any HTTP endpoint (third-party SaaS like Datadog, New Relic) with buffered batching and retry.
• Error handling. Records that repeatedly fail delivery are written to a configurable S3 error/backup prefix so nothing is silently lost; you reprocess from there.

Firehose vs Data Streams — the decision that comes up every time.

• Firehose = delivery, no replay, no ordering guarantee, no code, seconds-to-minutes latency. Pick it when the destination is the durable copy and near-real-time is enough.
• Data Streams = replayable ordered log, you write the consumer, sub-second. Pick it when you need replay, ordering, or multiple independent consumers.
• They compose. A very common pattern is Data Streams as the backbone with a Firehose delivery stream attached as a consumer — Data Streams gives replay/ordering to real-time consumers, Firehose gives zero-ops lake delivery, from the same events.

Worked example — tuning the buffer for freshness vs file size

Detailed explanation. The buffer is the whole game for Firehose-to-S3. Walk through choosing size/interval for a lake feed that Athena queries, balancing a 5-minute freshness target against Athena's preference for 128 MB+ files.

• Throughput. ~4 MB/s steady.
• Freshness SLA. Data queryable within ~5 minutes.
• Query engine. Athena — hates thousands of tiny files; loves ~128 MB Parquet.

Question. Pick buffer size and interval that meet the 5-minute SLA without producing tiny files, and compute the resulting file cadence.

Input.

Parameter	Value
Throughput	4 MB/s
Freshness SLA	~5 min (300 s)
Ideal file size	~128 MB
Buffer size range	1–128 MB
Buffer interval range	60–900 s

Code.

# Which buffer limit fires first at 4 MB/s?
THROUGHPUT_MBPS = 4
BUFFER_SIZE_MB  = 128       # max for S3
BUFFER_INT_S    = 300       # 5-minute freshness SLA

time_to_fill_by_size = BUFFER_SIZE_MB / THROUGHPUT_MBPS   # 128/4 = 32 s
# 32 s << 300 s, so at 4 MB/s the SIZE limit fires first every ~32 s.

# Resulting file cadence and size:
if time_to_fill_by_size <= BUFFER_INT_S:
    flush_every_s = time_to_fill_by_size     # 32 s
    file_size_mb  = BUFFER_SIZE_MB           # 128 MB
else:
    flush_every_s = BUFFER_INT_S             # interval-bound
    file_size_mb  = THROUGHPUT_MBPS * BUFFER_INT_S

print(f"flush every {flush_every_s:.0f}s, ~{file_size_mb:.0f} MB files")
# → flush every 32s, ~128 MB files   (well within the 300s SLA)

Step-by-step explanation.

1. At 4 MB/s, a 128 MB buffer fills in 32 seconds — far under the 300 s interval. So the size limit is the one that fires, every ~32 seconds, producing ~128 MB files. That's ideal for Athena and comfortably inside the 5-minute freshness SLA.
2. The interval (300 s) is effectively a safety net here: it only fires if throughput drops so low that 128 MB doesn't accumulate in 5 minutes. Setting it to 300 s guarantees data is never more than 5 minutes stale even during a lull.
3. If freshness had to be ~1 minute instead, you couldn't wait for 128 MB at low throughput — you'd lower the interval to 60 s and accept smaller files during quiet periods, then compact them later with a scheduled job.
4. The anti-pattern is a 1 MB / 60 s buffer "for freshness" — at 4 MB/s that still flushes on size every 0.25 s worth of data... no, it flushes when 1 MB accumulates (~0.25 s), producing thousands of 1 MB files an hour. Athena then spends more time listing/opening files than scanning data.
5. Enabling Parquet conversion on top makes each 128 MB object columnar, so Athena scans only the columns a query needs — compounding the file-size win.

Output.

Config	Flush trigger	File size	Freshness	Athena-friendly
128 MB / 300 s @ 4 MB/s	size (~32 s)	~128 MB	≤32 s	yes
1 MB / 60 s @ 4 MB/s	size (~0.25 s)	~1 MB	seconds	no (tiny files)
128 MB / 60 s @ 0.5 MB/s	interval (60 s)	~30 MB	≤60 s	acceptable

Rule of thumb. Set buffer size toward the max (128 MB) so high-throughput streams produce query-friendly files, and set the interval to your freshness SLA as a safety net for low-throughput periods. If you need both sub-minute freshness and large files, accept small files from Firehose and compact them in a scheduled downstream job — don't starve the buffer.

Worked example — Parquet conversion with dynamic partitioning

Detailed explanation. The highest-leverage Firehose-to-S3 config for a lake is JSON→Parquet conversion plus dynamic partitioning by a record field, so Athena/Spark can prune partitions and scan columns. Walk through the config for a clickstream landing partitioned by event date and customer.

• Input. JSON click events with customer_id and an epoch event_ts.
• Conversion. JSON → Parquet using a Glue table schema.
• Partitioning. cust=<customer_id>/dt=<yyyy-MM-dd>/ extracted from the record.

Question. Configure Firehose to convert to Parquet and dynamically partition by customer_id and event date.

Input.

Setting	Value
Source format	JSON
Target format	Parquet (via Glue schema)
Partition keys	customer_id, event date
Prefix	cust=!{partitionKeyFromQuery:cust}/dt=!{partitionKeyFromQuery:dt}/

Code.

{
  "ExtendedS3DestinationConfiguration": {
    "BucketARN": "arn:aws:s3:::clicks-lake",
    "BufferingHints": { "SizeInMBs": 128, "IntervalInSeconds": 120 },

    "DataFormatConversionConfiguration": {
      "Enabled": true,
      "InputFormatConfiguration":  { "Deserializer": { "OpenXJsonSerDe": {} } },
      "OutputFormatConfiguration": { "Serializer": { "ParquetSerDe": {} } },
      "SchemaConfiguration": {
        "DatabaseName": "lake", "TableName": "clicks", "Region": "us-east-1"
      }
    },

    "DynamicPartitioning": { "Enabled": true },
    "ProcessingConfiguration": {
      "Enabled": true,
      "Processors": [{
        "Type": "MetadataExtraction",
        "Parameters": [
          { "ParameterName": "MetadataExtractionQuery",
            "ParameterValue": "{cust:.customer_id, dt:(.event_ts/1000|strftime(\"%Y-%m-%d\"))}" },
          { "ParameterName": "JsonParsingEngine", "ParameterValue": "JQ-1.6" }
        ]
      }]
    },

    "Prefix": "cust=!{partitionKeyFromQuery:cust}/dt=!{partitionKeyFromQuery:dt}/",
    "ErrorOutputPrefix": "errors/!{firehose:error-output-type}/"
  }
}

Step-by-step explanation.

1. DataFormatConversionConfiguration turns each buffered batch of JSON into a Parquet object using the Glue lake.clicks table schema. Firehose reads column types from Glue, so the schema must exist and match the JSON fields.
2. DynamicPartitioning.Enabled = true plus the MetadataExtraction processor runs a JQ query over each record to extract partition values: cust from customer_id, and dt by dividing the epoch-millis event_ts by 1000 and formatting as %Y-%m-%d.
3. The Prefix template references those extracted keys with !{partitionKeyFromQuery:cust} and !{partitionKeyFromQuery:dt}, so records land under s3://clicks-lake/cust=123/dt=2026-09-05/...parquet. Athena's partition pruning can now skip entire prefixes.
4. ErrorOutputPrefix routes any record that fails conversion or partitioning to an errors/ prefix keyed by error type — so a malformed record never blocks the batch and is recoverable.
5. Downstream, an Athena/Glue crawler (or partition projection) registers the cust/dt partitions; a query filtering WHERE dt = '2026-09-05' AND cust = '123' scans one small Parquet prefix instead of the whole lake.

Output.

Aspect	Result
Object format	Parquet (columnar, compressed)
S3 layout	cust=/dt=/*.parquet
Athena scan	partition-pruned + column-pruned
Bad records	errors// prefix (recoverable)
Buffer	128 MB / 120 s (size-bound)

Rule of thumb. For any Firehose lake feed, enable Parquet conversion (needs a Glue schema) and dynamic partitioning by the fields queries filter on (date + a high-cardinality dimension). Always set an ErrorOutputPrefix so malformed records are quarantined, not lost — and verify the Glue schema matches the JSON before go-live.

Worked example — Redshift delivery via S3 COPY

Detailed explanation. Firehose-to-Redshift is not a direct insert — Firehose stages each batch in an intermediate S3 bucket, then runs a Redshift COPY to bulk-load it. Understanding the two-hop path explains the config and the failure modes. Walk through a delivery stream that loads an events table.

• Path. Firehose → intermediate S3 bucket → COPY into Redshift table.
• COPY options. Format, compression, and column mapping supplied in config.
• Failure handling. Failed COPYs and the manifest land in an S3 error prefix.

Question. Configure Firehose to load Redshift via S3 and explain why the S3 hop is mandatory.

Input.

Setting	Value
Intermediate S3	s3://events-fh-staging
Redshift table	analytics.events
COPY options	JSON 'auto', GZIP
Buffer	16 MB / 300 s

Code.

{
  "RedshiftDestinationConfiguration": {
    "ClusterJDBCURL": "jdbc:redshift://cluster.xxx.us-east-1.redshift.amazonaws.com:5439/prod",
    "Username": "firehose_loader",
    "CopyCommand": {
      "DataTableName": "analytics.events",
      "CopyOptions": "JSON 'auto' GZIP TIMEFORMAT 'epochmillisecs'"
    },
    "S3Configuration": {
      "BucketARN": "arn:aws:s3:::events-fh-staging",
      "BufferingHints": { "SizeInMBs": 16, "IntervalInSeconds": 300 },
      "CompressionFormat": "GZIP",
      "ErrorOutputPrefix": "redshift-errors/"
    },
    "RetryOptions": { "DurationInSeconds": 3600 }
  }
}

-- What Firehose effectively runs under the hood, per delivered batch:
COPY analytics.events
FROM   's3://events-fh-staging/2026/09/05/manifest'
IAM_ROLE 'arn:aws:iam::...:role/firehose-redshift'
JSON 'auto' GZIP TIMEFORMAT 'epochmillisecs';

Step-by-step explanation.

1. Firehose buffers records (16 MB / 300 s) and writes each batch as a GZIP object into the intermediate S3 bucket — this staging step is mandatory because Redshift's high-throughput ingest path is COPY from S3, not row-by-row INSERT.
2. Firehose then issues the COPY into analytics.events using the configured CopyOptions (JSON 'auto' maps JSON keys to columns; GZIP matches the staged compression; TIMEFORMAT parses epoch-millis timestamps).
3. The S3 hop is why Redshift delivery has two buffers to reason about: the S3 buffering hints control the staged-file size, and the COPY frequency follows the S3 flush. Bigger staged files → fewer, larger COPYs → less commit overhead on Redshift.
4. RetryOptions.DurationInSeconds = 3600 tells Firehose to keep retrying a failed COPY for up to an hour (e.g. during a Redshift maintenance window). If it still fails, the batch and a manifest are written to redshift-errors/ for manual replay.
5. Because the staged objects persist in S3, a failed or skipped COPY is fully recoverable — you can re-run the COPY from the manifest by hand. The S3 hop doubles as a durable landing zone, not just a transport detail.

Output.

Stage	Mechanism	On failure
Buffer	16 MB / 300 s	n/a
Stage	GZIP object → S3 staging	retained in S3
Load	Redshift COPY from S3	retry ≤ 3600 s
Give up	write manifest + batch	redshift-errors/ prefix
Recover	manual COPY from manifest	replayable

Rule of thumb. Firehose-to-Redshift always goes through S3 and a COPY — size the S3 buffer for efficient COPY batches (tens of MB, not 1 MB), set a generous retry duration to survive Redshift maintenance, and point ErrorOutputPrefix at a monitored bucket so failed loads are replayable rather than lost. The intermediate S3 bucket is a durable landing zone, treat it as one.

Data engineering interview question on Firehose delivery

A senior interviewer might ask: "You must land ~6 MB/s of JSON telemetry in an S3 data lake as Parquet, partitioned by device type and date, queryable within 3 minutes, and also load a 1% sample into Redshift for a BI dashboard. No consumer code. Design the Firehose delivery, the buffer settings, the transform/conversion, the partitioning, and the error handling."

Solution Using a Parquet-converting Firehose to S3 plus a sampling transform to Redshift

// 1. Lake delivery — Firehose to S3, Parquet, dynamic partitioning, 3-min SLA
{
  "ExtendedS3DestinationConfiguration": {
    "BucketARN": "arn:aws:s3:::telemetry-lake",
    "BufferingHints": { "SizeInMBs": 128, "IntervalInSeconds": 180 },
    "DataFormatConversionConfiguration": { "Enabled": true,
      "OutputFormatConfiguration": { "Serializer": { "ParquetSerDe": {} } },
      "SchemaConfiguration": { "DatabaseName": "lake", "TableName": "telemetry" } },
    "DynamicPartitioning": { "Enabled": true },
    "ProcessingConfiguration": { "Enabled": true, "Processors": [{
      "Type": "MetadataExtraction",
      "Parameters": [
        { "ParameterName": "MetadataExtractionQuery",
          "ParameterValue": "{dev:.device_type, dt:(.ts/1000|strftime(\"%Y-%m-%d\"))}" },
        { "ParameterName": "JsonParsingEngine", "ParameterValue": "JQ-1.6" } ] }] },
    "Prefix": "dev=!{partitionKeyFromQuery:dev}/dt=!{partitionKeyFromQuery:dt}/",
    "ErrorOutputPrefix": "errors/!{firehose:error-output-type}/"
  }
}

# 2. Sampling transform Lambda (for the SECOND Firehose → Redshift)
#    Keeps ~1% of records; drops the rest so only a sample reaches Redshift.
import base64, hashlib, json

def handler(event, _ctx):
    out = []
    for r in event["records"]:
        payload = json.loads(base64.b64decode(r["data"]))
        h = int(hashlib.md5(payload["device_id"].encode()).hexdigest(), 16)
        keep = (h % 100 == 0)                         # deterministic 1% sample
        out.append({
            "recordId": r["recordId"],
            "result": "Ok" if keep else "Dropped",    # Dropped = not delivered
            "data": r["data"],
        })
    return {"records": out}

// 3. BI sample delivery — Firehose to Redshift via S3 COPY
{
  "RedshiftDestinationConfiguration": {
    "CopyCommand": { "DataTableName": "bi.telemetry_sample",
                     "CopyOptions": "JSON 'auto' GZIP TIMEFORMAT 'epochmillisecs'" },
    "S3Configuration": { "BucketARN": "arn:aws:s3:::telemetry-fh-staging",
                         "BufferingHints": { "SizeInMBs": 16, "IntervalInSeconds": 300 },
                         "CompressionFormat": "GZIP", "ErrorOutputPrefix": "rs-errors/" },
    "ProcessingConfiguration": { "Enabled": true, "Processors": [
      { "Type": "Lambda", "Parameters": [
        { "ParameterName": "LambdaArn", "ParameterValue": "arn:...:function:sample-1pct" } ] } ] },
    "RetryOptions": { "DurationInSeconds": 3600 }
  }
}

Step-by-step trace.

Layer	Component	Purpose
Lake	Firehose → S3	128 MB / 180 s buffer → ~128 MB Parquet, 3-min SLA
Partition	JQ MetadataExtraction	dev=/dt=/ prefixes
Sample	Lambda transform	deterministic 1% keep; rest Dropped
BI	Firehose → Redshift via S3	COPY the 1% into bi.telemetry_sample
Errors	S3 error prefixes	conversion + COPY failures recoverable

After deployment, the full 6 MB/s stream lands as partition-pruned, column-pruned Parquet in the lake within 3 minutes, a deterministic 1% device sample loads into Redshift for the BI dashboard through the S3-COPY path, malformed records are quarantined in error prefixes, and not one line of consumer code was written.

Output:

Destination	Format	Buffer	Freshness	Volume
S3 lake	Parquet, partitioned	128 MB / 180 s	≤3 min	100% (~6 MB/s)
Redshift BI	rows via COPY	16 MB / 300 s	≤5 min	~1% sample
Error prefixes	raw/JSON	n/a	on failure	malformed only

Why this works — concept by concept:

• Buffer 128 MB / 180 s — at 6 MB/s the size limit fires in ~21 s, producing query-friendly ~128 MB Parquet files, with the 180 s interval as a low-throughput safety net inside the 3-minute SLA.
• Parquet conversion + dynamic partitioning — columnar files under dev=/dt= prefixes let Athena prune partitions and scan only needed columns, turning a full-lake scan into a targeted one.
• Deterministic sampling transform — hashing device_id % 100 keeps a stable 1% (the same devices every time), and marking the rest Dropped means Firehose simply doesn't deliver them — sampling with zero extra infrastructure.
• Redshift via S3 COPY — the sample is staged as GZIP in S3 then bulk-loaded with COPY, the only high-throughput Redshift ingest path; the staging bucket doubles as a replayable landing zone.
• Cost — two Firehose streams (per-GB delivered + Parquet conversion on the lake stream), one small Lambda, one Redshift COPY workload on 1% of volume. O(1) per record; no shards, no consumer fleet. Compared to a hand-built Spark lake loader plus a Redshift loader, this is configuration, not code.

Real-time
Topic — real-time-analytics
Streaming delivery and lake-loading problems

Practice →

Streaming Topic — streaming Buffering, batching, and format-conversion problems

Practice →

---

5. Analytics, scaling, and resharding

Managed Flink computes windowed aggregates over the stream; resharding splits hot shards and merges cold ones to rebalance throughput

The mental model in one line: kinesis analytics — now Amazon Managed Service for Apache Flink — is the stream-compute layer that reads a Kinesis stream and runs tumbling, sliding, or session **windows over event-time to emit running aggregates, while resharding is the throughput-management operation that splits a shard (dividing its hash-key range into two children to add capacity) or merges two adjacent shards (combining their ranges into one child to cut cost) — and the senior insight tying them together is that both windowed compute and resharding must respect the parent→child shard lineage: a consumer drains a parent shard to SHARD_END before reading its children, so ordering and window correctness survive a scaling event.** Every senior data engineer has had to add capacity to a live stream without dropping or reordering data, and every one has had to explain how a windowed aggregate stays correct across a reshard.

Managed Flink / Kinesis Analytics — windowed stream compute.

• What it is. A managed Apache Flink runtime (or the legacy SQL engine) that consumes a stream, runs continuous queries with windowing and streaming joins, and writes results to a sink (another stream, Firehose, S3, a database). You write Flink SQL or a Flink app; AWS runs and scales it.
• Window types. Tumbling (fixed, non-overlapping — "count per 1-minute bucket"), sliding/hop (fixed size, overlapping — "5-minute count updated every minute"), and session (gap-based — "group events until a 30-minute inactivity gap"). Choosing the window is choosing the aggregation semantics.
• Event time and watermarks. Real streams arrive out of order; Flink uses event-time (a timestamp in the record, ROWTIME in the SQL engine) plus watermarks to decide when a window is complete despite late/out-of-order arrivals. Processing-time windows are simpler but wrong for late data.
• KPUs. Capacity is measured in Kinesis Processing Units (each ≈ 1 vCPU + 4 GB). You scale parallelism by KPU count; the service can auto-scale KPUs with load.

Resharding — changing shard count on a live stream.

• Shard split. SplitShard takes one shard and a split point in its hash-key range and produces two child shards, each owning half the parent's range. This adds capacity (each child now has its own 1 MB/s / 2 MB/s budget) — the fix for a hot shard once the key distribution is sound.
• Shard merge. MergeShards takes two adjacent shards and combines their ranges into a single child. This removes capacity to cut cost when shards are under-utilised.
• Parent → child lineage. After a split/merge, the parent shard stops receiving new records but still holds un-read data. Consumers must read the parent to SHARD_END before reading the children, or records straddling the reshard could be processed out of order. KCL enforces this; hand-rolled consumers must implement it.
• UpdateShardCount. A higher-level API that scales a stream to a target shard count with uniform scaling (it does the splits/merges for you, keeping shards evenly sized). Simpler than manual split/merge for a general capacity change; rate-limited (you can roughly double/halve per call).

On-demand mode — resharding you don't do.

• Auto-scaling. On-demand streams monitor traffic and split/merge shards automatically, scaling up to 2× the trailing 30-day peak. You never call SplitShard.
• When to use. Spiky or unpredictable traffic, or teams that don't want to own capacity. The cost is a higher per-GB rate than a well-sized provisioned stream.
• Ordering still per shard. On-demand hides shard management, not shard semantics — partition keys still route to shards, ordering is still per shard, and a consumer still sees splits/merges (and must honour lineage).

Killing a hot shard — the full playbook (ties sections 2 and 5 together).

• Step 1 — fix the key. If the hot shard comes from a skewed/constant partition key, repartition or salt the key first (section 2). Splitting a shard whose skew is a constant key does nothing.
• Step 2 — split. With a sound key, SplitShard the hot shard so the redistributed traffic has physical room, or UpdateShardCount up for a uniform increase.
• Step 3 — drain lineage. Consumers finish the parent to SHARD_END, then fan out to the children — no reordering.
• Step 4 — consider on-demand. For recurring, unpredictable hotspots, on-demand mode auto-handles the split/merge cycle.

Worked example — a tumbling-window aggregation in Managed Flink

Detailed explanation. The canonical stream-compute task: count events per key per fixed time bucket. Walk through a Flink SQL tumbling window that computes per-page clicks per minute on event-time, tolerant of late arrivals via a watermark.

• Input. A clicks stream with page_id and event_time.
• Window. Tumbling, 1 minute, on event-time.
• Lateness. 5-second watermark tolerance for out-of-order arrivals.

Question. Write the Flink SQL that emits clicks-per-page-per-minute and explain how the watermark closes each window.

Input.

Element	Value
Source	Kinesis stream clicks
Grain	page_id × 1-minute bucket
Time	event-time (event_time)
Watermark	event_time − 5 s

Code.

-- Source table bound to the Kinesis stream, with an event-time watermark
CREATE TABLE clicks (
    page_id     STRING,
    user_id     STRING,
    event_time  TIMESTAMP(3),
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND   -- tolerate 5s lateness
) WITH (
    'connector' = 'kinesis',
    'stream'    = 'clicks',
    'aws.region'= 'us-east-1',
    'scan.stream.initpos' = 'LATEST',
    'format'    = 'json'
);

-- 1-minute tumbling count per page, on event-time
SELECT
    page_id,
    window_start,
    window_end,
    COUNT(*) AS clicks
FROM TABLE(
    TUMBLE(TABLE clicks, DESCRIPTOR(event_time), INTERVAL '1' MINUTE)
)
GROUP BY page_id, window_start, window_end;

Step-by-step explanation.

1. The WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND declaration tells Flink to treat the stream's clock as "the max event_time seen, minus 5 seconds." A record whose event_time is older than the current watermark is late and (by default) dropped from its window.
2. TUMBLE(..., DESCRIPTOR(event_time), INTERVAL '1' MINUTE) assigns each record to exactly one non-overlapping 1-minute bucket based on its event_time (e.g. 12:00:00–12:00:59). Tumbling windows never overlap, so every click is counted once.
3. GROUP BY page_id, window_start, window_end produces one output row per page per minute. The count accumulates as records arrive.
4. The window emits when the watermark passes window_end. Because the watermark trails the max event_time by 5 seconds, the 12:00 window fires once event_times reach ~12:01:05 — giving out-of-order records 5 seconds of grace to still be counted in the right minute.
5. Switching TUMBLE to HOP (sliding) would emit overlapping windows (e.g. a 5-minute count every minute); switching to SESSION would group by inactivity gap. The window function is the only thing that changes — the watermark machinery is identical.

Output.

page_id	window_start	window_end	clicks
/home	12:00:00	12:01:00	1,204
/home	12:01:00	12:02:00	1,318
/pricing	12:00:00	12:01:00	342
/pricing	12:01:00	12:02:00	401

Rule of thumb. Aggregate streams on event-time with a watermark, not processing-time, so late/out-of-order records still land in the correct window. Pick tumbling for non-overlapping buckets, sliding for smoothed rolling metrics, and session for activity-gap grouping — the watermark tolerance is the single knob that trades completeness (longer wait, fewer dropped late records) against latency (shorter wait, faster emit).

Worked example — splitting a hot shard to add capacity

Detailed explanation. With a sound partition key, a shard can still be hot simply because traffic grew. The remedy is SplitShard: pick the shard's hash-range midpoint and split it into two children, each carrying half the range and getting its own capacity. Walk through the split and the lineage drain.

• Before. shardId-0003 owns hash range [R_start, R_end], carrying 1.8 MB/s (near the 1 MB/s... actually over, throttling).
• Split point. The midpoint of [R_start, R_end].
• After. Two children, each owning half the range, ~0.9 MB/s each.

Question. Split the hot shard at its midpoint and describe how consumers preserve ordering across the reshard.

Input.

Element	Value
Hot shard	shardId-0003
Range	[R_start, R_end]
Load	~1.8 MB/s (throttling)
New starting hash key	(R_start + R_end) / 2

Code.

import boto3
kinesis = boto3.client("kinesis")

# 1. Look up the hot shard's hash-key range
shards = kinesis.list_shards(StreamName="events")["Shards"]
hot = next(s for s in shards if s["ShardId"] == "shardId-000000000003")
start = int(hot["HashKeyRange"]["StartingHashKey"])
end   = int(hot["HashKeyRange"]["EndingHashKey"])
midpoint = (start + end) // 2

# 2. Split it — creates two child shards, each owning half the range
kinesis.split_shard(
    StreamName="events",
    ShardToSplit="shardId-000000000003",
    NewStartingHashKey=str(midpoint),      # child B starts here; child A = [start, midpoint-1]
)

# 3. Consumers: DRAIN the parent to SHARD_END, THEN read children.
#    KCL does this automatically. A hand-rolled loop must:
def read_with_lineage(shard_id):
    it = get_iterator(shard_id, "TRIM_HORIZON")
    while it:
        resp = kinesis.get_records(ShardIterator=it)
        for r in resp["Records"]:
            process(r)
        it = resp.get("NextShardIterator")
        if it is None:                       # SHARD_END reached
            for child in child_shards_of(shard_id):
                read_with_lineage(child)      # only NOW start the children
            return

Step-by-step explanation.

1. list_shards exposes each shard's HashKeyRange. The hot shard owns [start, end]; splitting requires a NewStartingHashKey inside that range — the midpoint gives an even split so each child carries ~half the traffic.
2. split_shard marks shardId-0003 as a parent that stops receiving new records and creates two child shards: child A owns [start, midpoint-1], child B owns [midpoint, end]. New records now route to whichever child owns their key's hash.
3. Total ingress capacity for that range doubles (two shards, each 1 MB/s / 1000 rec/s), relieving the throttling — provided the traffic actually spreads across both halves (which it does when the partition key is high-cardinality; a constant key would still hit one child, the section 2 lesson).
4. The lineage rule: the parent still holds un-read records written before the split. Consumers must read the parent to SHARD_END (iterator returns NextShardIterator = null) before starting the children, so no record after the split is processed before a record before it — preserving per-key order across the boundary.
5. KCL handles all of this automatically via the DynamoDB lease table (it won't lease a child until the parent lease is checkpointed at SHARD_END). A hand-rolled consumer must implement the parent-then-children recursion shown, or it risks reordering.

Output.

Shard	Range	Status	Capacity
shardId-0003 (parent)	[start, end]	closed, drain to SHARD_END	frozen
child A	[start, midpoint-1]	open	1 MB/s
child B	[midpoint, end]	open	1 MB/s
Range total	[start, end]	2 shards	doubled

Rule of thumb. Split a hot shard at its hash-range midpoint to double capacity for that range, but only after confirming the partition key is high-cardinality (a constant key re-concentrates on one child). Always drain the parent to SHARD_END before reading children — let KCL enforce the lineage, and if you hand-roll a consumer, implement parent-then-children explicitly or you will silently reorder data across the reshard.

Worked example — uniform scaling with UpdateShardCount

Detailed explanation. For a general capacity change (not a single hot shard), UpdateShardCount scales the whole stream to a target shard count with uniform shards, doing the splits/merges for you. Walk through doubling a stream from 10 to 20 shards ahead of a known traffic ramp.

• Before. 10 uniform shards, 10 MB/s capacity.
• Trigger. A marketing event will 1.8× traffic.
• Target. 20 shards, 20 MB/s, evenly sized.

Question. Scale the stream to 20 shards uniformly and note the constraints on scaling rate.

Input.

Element	Value
Current shards	10
Target shards	20
Scaling type	UNIFORM_SCALING
Rate limit	≤ 2× per call; limited calls/day

Code.

import boto3
kinesis = boto3.client("kinesis")

# Scale 10 → 20 shards uniformly; Kinesis performs the splits internally
kinesis.update_shard_count(
    StreamName="events",
    TargetShardCount=20,                 # even shards; must be within 2x of current
    ScalingType="UNIFORM_SCALING",
)

# Poll until the reshard completes (stream status returns to ACTIVE)
import time
while True:
    desc = kinesis.describe_stream_summary(StreamName="events")["StreamDescriptionSummary"]
    if desc["StreamStatus"] == "ACTIVE" and desc["OpenShardCount"] == 20:
        break
    time.sleep(5)
print("resharded to 20 shards")

Step-by-step explanation.

1. update_shard_count(TargetShardCount=20, ScalingType="UNIFORM_SCALING") tells Kinesis to reach 20 evenly-sized shards. Internally it performs a series of splits so every shard owns an equal slice of the hash space — no manual midpoint math.
2. The target must be within 2× (or ½×) of the current count per call: 10→20 is allowed, 10→50 is not (you'd call it in steps). There are also per-day limits on scaling operations, so scaling is planned, not reflexive.
3. During the reshard the stream stays ACTIVE for producers/consumers, but the shard set changes: the 10 parents close and 20 children open. Consumers see new shards appear and must honour parent→child lineage (KCL does).
4. The code polls describe_stream_summary until StreamStatus == ACTIVE and OpenShardCount == 20, confirming the reshard finished before relying on the new capacity.
5. UPDATE_SHARD_COUNT with UNIFORM_SCALING is the right tool for a broad capacity change; a single hot shard is better handled by a targeted SplitShard (previous example), and unpredictable traffic is better handled by switching to on-demand mode, which does this continuously and automatically.

Output.

Metric	Before	After
Open shards	10	20
Ingress capacity	10 MB/s / 10k rec/s	20 MB/s / 20k rec/s
Shard sizing	uniform	uniform
Scaling type	—	UNIFORM_SCALING
Stream status during	ACTIVE	ACTIVE

Rule of thumb. Use UpdateShardCount with UNIFORM_SCALING for planned, broad capacity changes (respecting the ≤2×-per-call and per-day limits); use targeted SplitShard for a single hot shard; and switch to on-demand mode when traffic is spiky enough that you'd otherwise be resharding constantly. Always confirm the stream returns to ACTIVE at the new shard count before trusting the added capacity.

Data engineering interview question on scaling and windowed analytics

A senior interviewer might ask: "A 12-shard provisioned stream feeds a Managed Flink app computing 1-minute per-region click counts. A product launch will triple traffic in one region, and you're already seeing IteratorAge climb on two shards. Walk me through scaling the stream, keeping the windowed aggregation correct across the reshard, and deciding between manual resharding, UpdateShardCount, and on-demand mode."

Solution Using a targeted split for hot shards plus event-time windows that survive resharding

# 1. Confirm which shards are hot (shard-level IteratorAgeMilliseconds)
#    shardId-0004 and shardId-0009 (the launch region's keys) are climbing.

# 2. Because ordering here is per-region (partition key = region_id) and one
#    region is 3x, salt THAT region's key to spread it, then split its shards.
import boto3
kinesis = boto3.client("kinesis")

LAUNCH = "region-eu"
K = 4
def partition_key(ev):
    if ev["region_id"] == LAUNCH:
        return f'{ev["region_id"]}#{ev["click_id"] % K}'   # spread the hot region
    return ev["region_id"]

# 3. Split the two hot shards at their midpoints to add physical capacity
for sid in ["shardId-000000000004", "shardId-000000000009"]:
    s = next(x for x in kinesis.list_shards(StreamName="clicks")["Shards"] if x["ShardId"] == sid)
    lo = int(s["HashKeyRange"]["StartingHashKey"]); hi = int(s["HashKeyRange"]["EndingHashKey"])
    kinesis.split_shard(StreamName="clicks", ShardToSplit=sid,
                        NewStartingHashKey=str((lo + hi) // 2))

-- 4. The Flink window is UNAFFECTED by resharding — it aggregates on event-time,
--    not on shard identity. Re-sum the salted region's sub-keys at query time.
SELECT
    CASE WHEN region_id LIKE 'region-eu#%' THEN 'region-eu' ELSE region_id END AS region,
    window_start,
    SUM(clicks) AS clicks                          -- re-merge salted sub-keys
FROM (
    SELECT region_id, window_start, COUNT(*) AS clicks
    FROM TABLE(TUMBLE(TABLE clicks, DESCRIPTOR(event_time), INTERVAL '1' MINUTE))
    GROUP BY region_id, window_start, window_end
)
GROUP BY CASE WHEN region_id LIKE 'region-eu#%' THEN 'region-eu' ELSE region_id END, window_start;

Step-by-step trace.

Step	Action	Result
Diagnose	shard-level IteratorAge	shardId-0004, 0009 hot (launch region)
Key	salt launch region region-eu#0..3	hot region spreads across 4 sub-keys
Split	SplitShard on the two hot shards	+2 shards; range capacity doubled there
Lineage	drain parents to SHARD_END	Flink/KCL reads children after parents → order kept
Window	event-time TUMBLE unchanged	aggregation correct across reshard
Re-merge	SUM salted sub-keys in query	per-region count restored

After the change, the launch region's tripled traffic fans across four salted sub-keys and two freshly-split shards, IteratorAge flattens, and the Flink 1-minute window keeps computing correct per-region counts because it aggregates on event-time — the reshard changed the physical shard set, not the logical windows — with a final query-time SUM re-merging the salted sub-keys back into a per-region number.

Output:

Metric	Before	After
Hot shards	2 (IteratorAge climbing)	0 (flat)
Launch-region key	single (region-eu)	4 salted sub-keys
Shards	12	14 (2 split)
Window correctness	at risk under lag	correct on event-time
Per-region count	native	re-summed from sub-keys

Why this works — concept by concept:

• Shard-level IteratorAge — per-shard lag, not stream-average, is what reveals the two hot shards. Diagnose which shards are behind before touching capacity.
• Salt the hot region's key — spreading only region-eu across four sub-keys relieves the concentration while leaving every other region's clean per-region ordering untouched; surgical, not global.
• Targeted SplitShard + lineage — splitting exactly the two hot shards doubles their range capacity, and draining parents to SHARD_END before children keeps per-key order across the reshard (KCL/Flink enforce it).
• Event-time windows survive resharding — the Flink TUMBLE aggregates by event_time, which is a record attribute, not a shard property, so changing the shard set does not disturb window membership or counts; a query-time SUM re-merges the salted sub-keys.
• Cost — two extra shards plus a trivial query-time re-sum; no app redeploy, no window rewrite. O(1) per record on ingest; the re-merge is a cheap group-by. Compared to on-demand's per-GB premium (better for unpredictable spikes) or a full UpdateShardCount (better for broad growth), a targeted split is the cheapest fix for a known, localized hotspot.

Real-time
Topic — real-time-analytics
Windowed aggregation and streaming-analytics problems

Practice →

Streaming
Topic — streaming
Resharding and stream-scaling problems

Practice →

---

Cheat sheet — Amazon Kinesis recipes

• Service picker. Kinesis Data Streams = durable, ordered (per shard), replayable log you consume yourself — pick it for multiple independent consumers, replay, or sub-second latency. Firehose = fully-managed delivery to S3/Redshift/OpenSearch with buffering, no shards, no replay — pick it for load-to-lake/warehouse. Managed Flink (Kinesis Analytics) = windowed stream compute — pick it when the deliverable is an aggregate. Common topology: Data Streams backbone → Flink + Firehose + Lambda consumers.
• Shard capacity math. Each shard: 1 MB/s OR 1,000 records/s ingress, 2 MB/s egress (shared). Size shards = max(ceil(bytes/s ÷ 1 MB), ceil(records/s ÷ 1000)) then add 20–30% headroom. PutRecords batches ≤500 records / ≤5 MB; GetRecords ≤5 calls/s/shard, ≤10 MB / 10k records per call. Small-record streams are record-bound, not byte-bound.
• Partition-key rules. MD5(partition_key) → 128-bit hash → shard's hash-key range. Same key → same shard → ordered by SequenceNumber. No cross-shard ordering. Use a high-cardinality field (user_id, device_id) for even spread; put the entity that needs ordering in the key. A constant/skewed key = hot shard that more shards won't fix.
• Hot-shard playbook. Turn on enhanced (shard-level) monitoring; find the skewed shard via per-shard IncomingBytes/IncomingRecords/IteratorAge. Fix the key first (repartition or salt key#bucket), then SplitShard. Splitting a shard whose skew is a constant key changes nothing. For recurring spikes, switch to on-demand.
• Shared vs enhanced fan-out. Shared: all consumers split one 2 MB/s/shard, 5 GetRecords/s, ~200 ms, free. Enhanced fan-out (EFO): each registered consumer gets a dedicated 2 MB/s/shard via SubscribeToShard HTTP/2 push, ~70 ms, up to 20 consumers, billed per consumer-shard-hour + per-GB. Switch to EFO at ≥3 consumers or a sub-100 ms SLA.
• KCL checkpointing. One worker per shard; ownership via a DynamoDB lease table; progress = last SequenceNumber checkpointed. Checkpoint once per batch (per-record throttles DynamoDB; never-checkpoint replays the world). Kinesis is at-least-once → make consumers idempotent (dedupe by a record id) for effectively-once effects. Checkpoint SHARD_END when a parent drains.
• Iterator types. TRIM_HORIZON (oldest retained), LATEST (only new), AT_/AFTER_SEQUENCE_NUMBER (checkpoint resume), AT_TIMESTAMP (replay from a wall-clock time). Replay = extend retention + start a separate consumer at AT_TIMESTAMP; register it as EFO so it doesn't steal live throughput.
• Firehose buffer. Flush on size (1–128 MB) OR interval (60–900 s), whichever first. Big buffer → query-friendly ~128 MB files but staler; small buffer → fresh but tiny-file problem. Set size near 128 MB and interval = freshness SLA (safety net). Need sub-minute + big files? Let Firehose emit small and compact downstream.
• Firehose format + partitioning. Enable JSON→Parquet/ORC conversion via a Glue schema; enable dynamic partitioning with a JQ MetadataExtraction query to write key=value/ prefixes queries filter on (date + a dimension). Always set ErrorOutputPrefix so malformed records are quarantined, not lost.
• Firehose→Redshift. Not a direct insert — Firehose stages GZIP objects in S3 then runs COPY. Size the S3 buffer for efficient COPY batches (tens of MB), set a generous RetryOptions.DurationInSeconds (e.g. 3600) to survive maintenance, and monitor the error prefix; the staging bucket is a replayable landing zone.
• Resharding. SplitShard (one → two children, split the hash range, adds capacity) for hot shards; MergeShards (two adjacent → one, removes capacity) to cut cost; UpdateShardCount + UNIFORM_SCALING for broad even changes (≤2× per call, per-day limits). Consumers must drain the parent to SHARD_END before children — KCL enforces lineage; hand-rolled consumers must too.
• On-demand + windows. On-demand mode auto-splits/merges to 2× the 30-day peak — no capacity planning, higher per-GB. Ordering/keys still per shard. Managed Flink windows: tumbling (fixed buckets), sliding/hop (overlapping), session (gap-based); aggregate on event-time + watermark so late records land correctly. Windows aggregate by event-time, so they stay correct across a reshard.

Frequently asked questions

What is Amazon Kinesis in one sentence?

Amazon Kinesis is a family of AWS streaming services for moving and processing real-time data at scale: Kinesis Data Streams is a durable, ordered (per shard), replayable log you consume yourself; Amazon Data Firehose is a fully-managed delivery pipe that buffers records and lands them in S3, Redshift, or OpenSearch with no shards and no replay; and Managed Service for Apache Flink (formerly Kinesis Data Analytics) runs windowed SQL/Flink compute over a stream. The interview signal is knowing which service owns ordering, which owns replay, and which one you never scale by hand — and that a common production topology is Data Streams as the backbone with Firehose and Flink attached as independent consumers.

Kinesis Data Streams vs Firehose — when do I pick each?

Pick Data Streams when you need per-shard ordering, replay (24-hour to 365-day retention lets a consumer rewind), sub-second latency, or multiple independent consumers reading the same events — you own the consumer (KCL, Lambda, Flink, or Firehose-as-consumer) and the shard count (or use on-demand). Pick Firehose when the requirement is "land these records in a store as files/rows" — it buffers by size/time, optionally transforms and converts to Parquet, and delivers to S3/Redshift/OpenSearch with zero consumer code, but has no replay and no ordering guarantee. They compose: many teams run Data Streams as the durable log and attach a Firehose delivery stream as one of its consumers for the lake. If the question mentions "reprocess," "replay," or "ordered per user," the answer is Data Streams.

What is a shard and how many do I need?

A shard is the unit of both throughput and ordering in a Kinesis Data Stream: each shard accepts 1 MB/second or 1,000 records/second of ingress and serves 2 MB/second of egress, and records within a shard are strictly ordered by SequenceNumber. Size the shard count as max(ceil(bytes-per-second ÷ 1 MB), ceil(records-per-second ÷ 1,000)), then add 20–30% headroom — small-record streams are usually record-bound (the 1,000 rec/s cap binds before the 1 MB/s cap), which is the most common sizing mistake. To go faster you add shards (reshard); there is no "bigger shard." If traffic is spiky or unpredictable, use on-demand mode and skip the arithmetic — it auto-scales to twice the trailing 30-day peak at a higher per-GB price.

What does a partition key do in Kinesis?

The partition key is a string you attach to every record; Kinesis computes MD5(partition_key) into a 128-bit integer and routes the record to whichever shard owns that value in its hash-key range. Its two consequences are the whole ballgame: records with the same partition key always land on the same shard and are therefore delivered in order, and the key's value distribution determines load balance across shards. Put the entity that needs ordering — user_id, device_id, account_id — in the key so its events stay ordered, and make sure that field is high-cardinality and evenly distributed so traffic spreads. A low-cardinality or skewed key (like country or a constant) concentrates traffic onto one hot shard that no amount of extra shards will relieve until you fix the key.

Shared throughput vs enhanced fan-out — what's the difference?

In the shared throughput model, every classic consumer of a shard splits a single 2 MB/second egress budget and the 5 GetRecords/second poll limit, so a third or fourth consumer starves the others and reads start throttling. Enhanced fan-out (EFO) registers each consumer separately and, via a SubscribeToShard HTTP/2 push, gives each its own dedicated 2 MB/second per shard at roughly 70 ms latency, with up to 20 registered consumers — at the cost of a per-consumer-shard-hour plus per-GB-retrieved charge. Stay on shared throughput for one or two latency-tolerant consumers; switch to EFO at three or more consumers or whenever a consumer has a sub-100 ms latency SLA. Either way, checkpoint progress (the last SequenceNumber) with the KCL so a restart resumes instead of replaying, and make consumers idempotent because Kinesis delivers at-least-once.

How do I handle a hot shard, and what is resharding?

A hot shard is one shard saturating while the stream sits below its paid capacity — the signature is write throttling (ProvisionedThroughputExceededException) or rising per-shard IteratorAge on one shard. It is almost always a partition-key problem, so the playbook is: enable shard-level monitoring to find the skewed shard, fix the key first (repartition to a high-cardinality field, or salt it as key#bucket), and then reshard. Resharding changes shard count on a live stream: SplitShard divides one shard's hash range into two children to add capacity, MergeShards combines two adjacent shards to cut cost, and UpdateShardCount with UNIFORM_SCALING does a broad even resize (≤2× per call). Consumers must drain the parent shard to SHARD_END before reading its children so ordering survives the reshard — KCL enforces this automatically. Note that splitting a shard whose skew comes from a constant key changes nothing, and for recurring unpredictable spikes, on-demand mode auto-reshards for you.

Practice on PipeCode

• Drill the streaming practice library → for the shard-sizing, partition-key, enhanced-fan-out, and resharding problems senior interviewers love.
• Rehearse on the real-time analytics practice library → for windowed aggregation, Firehose lake-loading, and streaming-compute scenarios.
• Sharpen the event side with the event processing practice library → for consumer checkpointing, delivery-semantics, and event-ordering patterns.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the Kinesis service picker against real graded inputs.

Lock in Amazon Kinesis muscle memory

Docs explain the APIs. PipeCode drills explain the decision — when a stream is record-bound not byte-bound, when a partition key hides a hot shard, when enhanced fan-out earns its cost, when Firehose's buffer trades freshness for file size, when a reshard must drain the parent first. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice streaming problems →
Practice real-time analytics →