Redpanda Deep Dive: Kafka-Compatible C++ Broker, Tiered Storage & Iceberg Topics

redpanda is the pick-one architectural decision that finally makes the streaming tier feel like a modern piece of infrastructure — a Kafka-wire-compatible broker written in C++ on the Seastar async runtime, no ZooKeeper and no JVM in the hot path, sub-5 ms p99 latency at throughputs that used to require a five-node Kafka cluster, and (as of 2024) native Iceberg Topics that turn every producer write into a row in a governed lakehouse table without a separate connector. Every message your business writes — a checkout event, a click, a CDC row from Debezium, a device telemetry ping — has to reach the consumer group, the search index, the feature store, and the warehouse without a JVM garbage-collection pause blowing out tail latency, without a broker fleet twice the size it should be, and without an ETL job in the middle re-serialising every event into Parquet. The engineering trade-off does not live in "should we run a streaming platform" — every stack with more than one downstream consumer needs one — but in which redpanda kafka alternative you pick and how it composes with your existing lakehouse, Confluent Cloud contract, and on-call rota.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "walk me through the redpanda vs kafka architecture and where each wins," or "your CFO is asking why streaming costs 6x your warehouse — what would you migrate to?", or "explain iceberg topics and why they collapse the streaming-to-lakehouse handoff." It walks through the four pieces every senior architect has to defend — the Seastar thread-per-core + io_uring core that eliminates the GC-pause tail-latency class of bug, redpanda tiered storage that keeps recent segments on SSD and offloads cold segments to S3/GCS/Azure Blob for object-storage-priced retention, Iceberg Topics that materialise the same partitioned log as an Iceberg table for zero-copy Trino / Snowflake / DuckDB access, and the four-broker decision matrix (Redpanda, Kafka, WarpStream, AutoMQ) that senior interviewers actually probe. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the SQL practice library →, sharpen the streaming axis with the streaming practice library →, and rehearse system design against the design practice library →.

---

On this page

• Why Redpanda matters in 2026
• Thread-per-core + Seastar architecture
• Tiered storage + Redpanda Cloud
• Iceberg Topics + streaming to lakehouse
• Redpanda vs Kafka vs WarpStream / AutoMQ + interview signals
• Cheat sheet — Redpanda recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Redpanda matters in 2026

A Kafka-wire-compatible C++ broker with tiered storage and Iceberg Topics — the streaming stack finally caught up with the lakehouse

The one-sentence invariant: Redpanda is a Kafka-API-compatible streaming broker rewritten from scratch in C++ on the Seastar async runtime with a thread-per-core execution model, no JVM, no ZooKeeper, native tiered storage against S3/GCS/Azure Blob, and (as of 2024) Iceberg Topics that materialise every produced message as a row in a governed Iceberg table — and every senior data-engineering interviewer in 2026 asks about it because it is the first credible replacement for Apache Kafka that keeps the wire protocol while rewriting everything below it. The pattern you pick in month one becomes the streaming tier your entire company writes producers, consumers, connectors, and dashboards against for the next five years, because every downstream lakehouse pipeline, feature store, and monitoring dashboard hard-codes topic names, consumer-group semantics, and retention assumptions the moment the first producer ships.

The four axes interviewers actually probe.

• Wire-compatibility and drop-in-ness. Redpanda speaks the same Kafka wire protocol (produce, fetch, metadata, consumer groups, transactions, exactly-once semantics) that any existing Kafka client library (Java, Go, Python, Node, Rust) already knows. Interviewers open here because it separates architects who understand "the protocol is the moat" from those who assume switching brokers means switching client libraries.
• Latency and tail-latency floor. Redpanda's Seastar runtime pins one shard per CPU core with pinned memory, avoids the JVM entirely, and uses io_uring for user-space I/O — the effect is a p99 latency floor that is typically 5–10x lower than an equivalently-sized Kafka cluster at the same throughput. Interviewers probe this because it maps directly to real workloads (ad bidding, order matching, real-time personalisation) where a 200 ms tail is a bug, not a metric.
• Cost story and storage tiering. Redpanda's Tiered Storage moves cold segments off local NVMe to S3/GCS/Azure Blob transparently — the broker keeps recent segments hot for low-latency fetches and lets long-retention data live at object-storage prices ($0.023/GB-month rather than $0.10+/GB-month for EBS). Interviewers probe this because "why do we spend $500k a year on streaming" is a CFO question, not a technology one.
• Lakehouse convergence. Iceberg Topics (GA in 2024) let a Redpanda topic double as an Iceberg table: producers write once, consumers keep consuming as normal and Trino / Snowflake / DuckDB can query the same data as an Iceberg table without a separate Debezium-plus-Kafka-Connect-plus-sink pipeline. Interviewers probe this because it collapses the streaming-to-lakehouse handoff from a five-service pipeline into one broker feature.

The 2026 reality — Redpanda is one of four viable Kafka-wire brokers.

• Redpanda. Kafka-wire-compatible C++ broker with Seastar; sub-5 ms p99; tiered storage; Iceberg Topics; Redpanda Cloud (SaaS + BYOC). Production-adopted at Vodafone, Alpaca, Lacework, and dozens of latency-sensitive fintech / adtech shops. The default when p99 matters and you want to escape the JVM without giving up the Kafka client contract.
• Apache Kafka. JVM broker, huge ecosystem (Connect, Streams, ksqlDB, Schema Registry, MirrorMaker 2), the reference implementation for the wire protocol. KRaft (ZooKeeper-less) mode is now the default in Kafka 3.5+. The default when the ecosystem breadth or organisational familiarity wins the argument.
• WarpStream. Kafka-wire-compatible broker written in Go that stores everything directly in object storage — no local disk, no replication, no ZooKeeper. Trades ~500 ms produce latency for a dramatic cost win on high-volume, latency-insensitive workloads (logs, telemetry). Acquired by Confluent in 2024.
• AutoMQ. Kafka-wire-compatible broker that keeps Kafka's storage layer semantics but shards them onto EBS + S3. Similar cost-optimisation story to WarpStream with different latency trade-offs. Rising in China / APAC.

What interviewers listen for.

• Do you name Seastar + thread-per-core + io_uring as the architectural moat, not just "it's written in C++"? — senior signal.
• Do you name Kafka-wire compatibility as the reason you can migrate, not "we'd rewrite our producers"? — required answer.
• Do you name Iceberg Topics as the streaming-to-lakehouse collapse, not "we could add Debezium"? — senior signal.
• Do you name tiered storage as the TCO story, not "it's cheaper"? — required answer.
• Do you distinguish Redpanda vs WarpStream vs AutoMQ on the latency-vs-cost axis, not treat them as interchangeable? — senior signal.

Worked example — the four-broker comparison table

Detailed explanation. The single most useful artifact for a Redpanda interview is a memorised 4x4 comparison table. Every senior streaming-tier discussion converges on this table within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical greenfield deployment that has to serve both a latency-sensitive order-matching path and a high-volume telemetry ingest.

• Workload A. Order matching — 50k msg/s, ~1 KB messages, p99 latency budget < 10 ms end-to-end. Latency-critical.
• Workload B. Device telemetry — 500k msg/s, ~200 B messages, retention 30 days, latency tolerant to seconds. Cost-critical.
• Downstream. Both workloads need to land in an Iceberg lakehouse (Trino + Snowflake) and feed active consumer groups (fraud model, dashboard).
• Constraints. AWS-only, one on-call rota, must run in the same VPC as the app tier (no third-party SaaS data path).

Question. Build the four-broker comparison and pick the broker each workload should ship on.

Input.

Broker	p99 latency (steady)	Cold-storage TCO	Ecosystem breadth	Lakehouse-native	Ops complexity
Redpanda	< 5 ms	S3 tiered (native)	Kafka-wire; growing	Iceberg Topics (native)	one binary, no JVM
Apache Kafka	10–50 ms	tiered storage (KIP-405)	huge (Connect, Streams, ksqlDB)	via Debezium + sink	JVM + KRaft/ZK
WarpStream	~500 ms	S3-only (native)	Kafka-wire (mostly)	via Kafka Connect	agentless (S3 is the store)
AutoMQ	~50 ms	EBS + S3 tier	Kafka-wire	via connectors	Kafka-like

Code.

# Broker choice per workload — one config, four candidates
workloads:
  order_matching:
    throughput_msg_s: 50000
    avg_message_bytes: 1024
    p99_latency_budget_ms: 10
    retention_days: 7
    recommended_broker: redpanda        # p99 < 5 ms is table stakes

  device_telemetry:
    throughput_msg_s: 500000
    avg_message_bytes: 200
    p99_latency_budget_ms: 5000         # generous
    retention_days: 30
    recommended_broker: redpanda        # Iceberg Topics = one write, two readers

  # Anti-example: what we would NOT pick
  # WarpStream for order_matching   -> 500 ms p99 blows the budget
  # Apache Kafka for both           -> viable but pays the JVM tax + no Iceberg Topics
  # AutoMQ for both                 -> viable, weaker Iceberg story in 2026

Step-by-step explanation.

1. The order-matching workload has a hard p99 latency budget under 10 ms end-to-end. Redpanda's Seastar core hits sub-5 ms p99 in steady state at 50k msg/s on modest hardware; Kafka on the same hardware typically lands at 20–50 ms with occasional GC-pause spikes to 200 ms. WarpStream's ~500 ms produce latency (fundamental to its "store in S3 directly" design) is a non-starter.
2. The device-telemetry workload is cost-critical, not latency-critical. All four brokers can carry 500k msg/s of 200-byte messages; the deciding factor becomes storage cost for 30 days of retention (~250 TB) and the lakehouse landing story. Redpanda's Tiered Storage + Iceberg Topics does both in one broker; WarpStream's S3-only design does storage cheaper but forces a separate Iceberg pipeline; Kafka is competitive but pays the JVM tax on ingest CPU.
3. The "same VPC, no third-party data path" constraint rules out any managed offering that hairpins data through a vendor's account. Redpanda Cloud BYOC keeps the data plane in your VPC while the control plane is Redpanda's; WarpStream is architecturally BYOC by design; Confluent Cloud (SaaS) requires the data-in-vendor-VPC model.
4. The one-on-call-rota constraint pushes hard toward one broker for both workloads rather than a hybrid. Running Redpanda for both is operationally the simplest (one binary, one CLI, one metrics stack); running Kafka + Redpanda side by side means two on-call runbooks, two upgrade cadences, two client-config styles.
5. In practice, most greenfield senior architectures in 2026 pick Redpanda for both workloads on the strength of Iceberg Topics alone — collapsing the streaming-to-lakehouse pipeline is a bigger operational win than shaving 10 ms off latency, and the latency win is a free extra.

Output.

Workload	Recommended broker	Deciding axis
Order matching (latency-critical)	Redpanda	p99 < 5 ms; no GC-pause tail
Device telemetry (cost-critical)	Redpanda	Tiered Storage + Iceberg Topics one-write / two-read
Log firehose (cost-only)	WarpStream	S3-only pricing wins if latency does not matter
Existing Confluent contract	Apache Kafka	ecosystem lock-in until contract ends

Rule of thumb. Never pick a Kafka-wire broker based on "which vendor has the loudest pitch." Pick it on (latency × cost × ecosystem × lakehouse-native) — the four axes. Write the table on a whiteboard first; the broker choice falls out of the constraints.

Worked example — what interviewers actually probe on Redpanda

Detailed explanation. The senior data-engineering Redpanda interview has a predictable structure: the interviewer opens with an ambiguous question ("what's your take on the Kafka alternatives?"), then progressively narrows to test whether you understand Seastar, tiered storage, Iceberg Topics, and the migration story. The candidates who name the architectural primitive in sentence one score highest; the candidates who describe "just another broker" score lowest. Walk through the interview grading rubric.

• Ambiguous opener. "Why would you look at Redpanda instead of Kafka?" — invites you to name Seastar and the four axes.
• Follow-up 1. "How does Redpanda avoid the JVM tail-latency issue?" — probes the Seastar + io_uring design.
• Follow-up 2. "Your CFO is asking why streaming costs $500k a year." — probes the Tiered Storage + object-storage TCO argument.
• Follow-up 3. "We already have a Debezium + Kafka Connect + Iceberg-sink pipeline." — probes Iceberg Topics as the collapse.
• Follow-up 4. "How would you migrate off a working Kafka cluster?" — probes the wire-compatibility migration playbook.

Question. Draft a 5-minute senior Redpanda answer that covers all four axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Architecture named	"it's written in C++"	"Seastar thread-per-core + io_uring + no JVM = no GC pauses"
Latency claim	"it's faster"	"p99 < 5 ms sustained vs Kafka's 20–50 ms + GC spikes"
Cost story	"cheaper storage"	"Tiered Storage offloads cold segments to S3 at $0.023/GB-mo"
Lakehouse	"we'd add Debezium"	"Iceberg Topics — same write feeds consumers and Iceberg"
Migration	"we'd swap it in"	"Kafka wire-compatible; rpk topic mirror + shadow read + cutover"

Code.

Senior Redpanda answer template (5 minutes)
===========================================

Minute 1 — name the architecture up front
  "I'd evaluate Redpanda as the Kafka-wire-compatible C++ broker
   built on Seastar with a thread-per-core execution model and
   io_uring — the architectural moat is 'no JVM, no ZooKeeper,
   no page cache, one shard per core' which eliminates the
   GC-pause tail-latency class of bug that Kafka can't design
   away without leaving the JVM."

Minute 2 — the latency claim + measurement
  "In steady state Redpanda hits sub-5 ms p99 at throughputs
   where Kafka lands at 20–50 ms with occasional 200 ms spikes
   from the JVM. For anything with a real p99 budget — order
   matching, ad bidding, real-time fraud — this changes the
   architecture, not just the metric. I benchmark with the
   OpenMessaging Benchmark or Redpanda's own franz-go load
   generator, always publishing p50, p95, p99, p99.9."

Minute 3 — tiered storage + TCO
  "Redpanda's Tiered Storage keeps recent segments on local
   NVMe for latency and offloads cold segments to S3/GCS/Azure
   Blob for retention. A 30-day-retention firehose that would
   cost $50k/mo on EBS drops to $8–12k/mo on S3 tiered — the
   6x TCO claim is real for retention-heavy workloads. Redpanda
   Cloud BYOC keeps the data plane in your VPC."

Minute 4 — Iceberg Topics
  "Iceberg Topics (GA 2024) let a topic double as an Iceberg
   table: the broker writes each produced message both to the
   partitioned log for consumers and to Iceberg data files
   registered in a Polaris / Glue / Unity catalog. Trino,
   Snowflake, and DuckDB query the same data as a native
   Iceberg table without a Debezium-plus-connector pipeline.
   This collapses streaming-to-lakehouse from five services
   to one broker feature."

Minute 5 — migration + failure semantics
  "Migration is wire-compatible: existing Kafka clients keep
   working with only a bootstrap-server change. Use rpk topic
   mirror or MirrorMaker 2 to shadow-write from Kafka into
   Redpanda; run consumers off both in a shadow-read window;
   then cut over producers. Rollback is a bootstrap-server
   flip. Failure semantics match Kafka's — acks=all,
   min.insync.replicas=2, idempotent producer — because the
   Raft-per-partition backing store is Kafka-equivalent."

Step-by-step explanation.

1. Minute 1 is the crucial framing. Naming the architectural primitive — "Seastar thread-per-core + io_uring + no JVM" — signals you're an architect, not a user. Weak candidates dive into features ("it has tiered storage and…") before naming the core design decision.
2. Minute 2 addresses the latency claim with a measurement pattern. This preempts the common trap where you claim "faster" without a benchmark; naming p50/p95/p99/p99.9 and the OpenMessaging Benchmark shows you've actually measured it in a lab.
3. Minute 3 addresses the TCO axis with concrete numbers. "6x TCO win" is Redpanda's marketing claim; being able to back it with "$50k → $8–12k on a 30-day firehose" makes the answer credible rather than corporate.
4. Minute 4 names Iceberg Topics as the architectural collapse — this is 2026's most differentiated Redpanda feature. Weak candidates treat it as "another connector"; senior candidates treat it as "the streaming-to-lakehouse pipeline shrinks from five services to one."
5. Minute 5 covers the migration reality — this is where wire compatibility earns its keep. "Bootstrap-server flip" is the entire migration primitive on the client side; the mirror + shadow-read playbook is the safe cutover pattern.

Output.

Grading criterion	Weak score	Senior score
Names Seastar + io_uring in minute 1	rare	mandatory
Names p99 measurement pattern	rare	required
Names TCO with numbers	rare	senior signal
Names Iceberg Topics as collapse	rare	senior signal
Names wire-compatible migration	occasional	mandatory

Rule of thumb. The senior Redpanda answer is a 5-minute monologue that covers architecture, latency, cost, Iceberg Topics, and migration without waiting for the follow-ups. Rehearse it once; deploy it every time.

Worked example — the "pick the broker" decision tree

Detailed explanation. Given a new streaming workload or a migration project, the senior architect runs a 5-question decision tree in their head. Codifying the tree makes the interview answer reproducible: any interviewer can hand you a scenario and you can walk it out loud. Walk through the tree with three canonical scenarios: greenfield fintech, established Kafka shop with a Confluent contract, and a log firehose team looking at costs.

• Q1. Is p99 latency a hard budget (< 20 ms end-to-end)? → yes = Redpanda; no = go to Q2.
• Q2. Is cold-data storage cost the dominant line item (> 50% of streaming spend)? → yes = go to Q3; no = go to Q4.
• Q3. Is latency tolerant to seconds (logs, telemetry, backfill)? → yes = WarpStream; no = Redpanda + Tiered Storage.
• Q4. Do you have an existing Confluent contract or heavy ksqlDB / Kafka Streams footprint? → yes = Apache Kafka (until contract ends); no = go to Q5.
• Q5. Do you need lakehouse-native landing without a separate pipeline? → yes = Redpanda (Iceberg Topics); no = Redpanda or Kafka, coin-flip on ecosystem preference.

Question. Walk the decision tree for the three scenarios and record the broker each ends up with.

Input.

Scenario	Q1 (latency < 20 ms?)	Q2 (cold-cost dominant?)	Q4 (Confluent lock-in?)	Q5 (lakehouse-native?)
Greenfield fintech	yes	—	no	yes
Established Kafka shop	no	no	yes	maybe
Log firehose	no	yes	no	yes (batched)

Code.

# Decision-tree helper (illustrative)
def pick_streaming_broker(hard_latency_budget: bool,
                           cold_cost_dominant: bool,
                           latency_tolerant_seconds: bool,
                           confluent_locked_in: bool,
                           needs_lakehouse_native: bool) -> list[str]:
    """Return the primary streaming-broker choice(s) for a workload."""
    choices: list[str] = []

    if hard_latency_budget:
        choices.append("redpanda")
    elif cold_cost_dominant and latency_tolerant_seconds:
        choices.append("warpstream")
    elif cold_cost_dominant:
        choices.append("redpanda (tiered storage)")
    elif confluent_locked_in:
        choices.append("apache kafka (until contract ends)")
    elif needs_lakehouse_native:
        choices.append("redpanda (iceberg topics)")
    else:
        choices.append("redpanda or kafka (ecosystem coin-flip)")

    return choices


# Walk the three scenarios
print(pick_streaming_broker(True,  False, False, False, True))
# -> ['redpanda']

print(pick_streaming_broker(False, False, False, True,  False))
# -> ['apache kafka (until contract ends)']

print(pick_streaming_broker(False, True,  True,  False, True))
# -> ['warpstream']

Step-by-step explanation.

1. Scenario 1 — Greenfield fintech with a 10 ms p99 budget on the order-matching path and a lakehouse-native landing requirement. Q1 short-circuits at yes → Redpanda. Iceberg Topics is a bonus. This is the 2026 default answer for latency-critical greenfield.
2. Scenario 2 — Established Kafka shop with a two-year Confluent contract, no hard p99 budget, and a heavy ksqlDB investment. Q1 no, Q2 no, Q4 yes → Apache Kafka until the contract ends, then re-evaluate. Trying to migrate mid-contract usually loses money.
3. Scenario 3 — Log firehose team spending $200k/mo on Kafka, latency-insensitive, cost-critical. Q1 no, Q2 yes, Q3 yes → WarpStream. The 500 ms produce latency is invisible for logs; the S3-only storage cost is dramatic. This is the one scenario where WarpStream beats Redpanda outright.
4. The parallel Q5 branch (lakehouse-native) is a tiebreaker that consistently pushes toward Redpanda when Q1–Q4 are ambiguous. Iceberg Topics collapse a whole ETL pipeline; that's worth trading a small ecosystem gap for.
5. If none of Q1–Q5 push you decisively, the answer is "run a POC on both Redpanda and Kafka with your actual workload for two weeks." Refuse to pick a broker on architectural aesthetics alone; ship real load and measure.

Output.

Scenario	Primary broker	Add-on
Greenfield fintech	Redpanda	Iceberg Topics + BYOC
Established Kafka + Confluent	Apache Kafka	plan migration for contract renewal
Log firehose	WarpStream	Iceberg via Kafka Connect

Rule of thumb. The five-question decision tree is a whiteboard-friendly answer. Practise walking it end-to-end so an interviewer can hand you any scenario and get a broker name in under 60 seconds.

Senior interview question on picking Redpanda

A senior interviewer often opens with: "You are joining a team that runs a three-broker Apache Kafka cluster on m5.4xlarge instances with EBS gp3 for storage, holding 30 days of retention across 800 topics. The CFO is asking why streaming costs $80k/month, the SRE team is fighting weekly GC-pause incidents, and the data platform team wants to land every topic in an Iceberg lakehouse. Walk me through the Redpanda evaluation you'd run, the migration playbook, and the failure modes you'd pre-empt."

Solution Using a Redpanda POC with Tiered Storage, Iceberg Topics, and a wire-compatible cutover

# 1. Deploy a Redpanda cluster mirroring the Kafka spec (POC in a staging VPC)
# Using rpk (Redpanda's CLI); production uses the Operator or Terraform module.

# Three brokers, same instance shape as the Kafka fleet to compare apples-to-apples
for i in 1 2 3; do
  aws ec2 run-instances \
    --instance-type m5.4xlarge \
    --block-device-mappings 'DeviceName=/dev/nvme1n1,Ebs={VolumeSize=2000,VolumeType=gp3}' \
    --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=redpanda-${i}}]"
done

# 2. Bootstrap the cluster (rpk on each broker)
rpk redpanda config bootstrap --self <this-broker-ip> --ips <ip1>,<ip2>,<ip3>
rpk redpanda config set redpanda.developer_mode false
sudo systemctl enable --now redpanda

# 3. Enable Tiered Storage cluster-wide
# /etc/redpanda/redpanda.yaml (excerpt)
redpanda:
  cloud_storage_enabled: true
  cloud_storage_credentials_source: aws_instance_metadata   # or config_file
  cloud_storage_region: us-east-1
  cloud_storage_bucket: acme-redpanda-tiered
  cloud_storage_segment_max_upload_interval_sec: 3600

  # Local retention (hot data on NVMe)
  retention_local_target_bytes_default: 107374182400        # 100 GiB per partition local
  retention_local_target_ms_default: 86400000               # 24 h hot on SSD

  # Total retention (hot + cold)
  # topic-level overrides via `rpk topic alter-config ... retention.ms=...`

# 4. Enable Iceberg Topics + Schema Registry mapping
rpk cluster config set iceberg_enabled true
rpk cluster config set iceberg_catalog_type rest
rpk cluster config set iceberg_rest_catalog_endpoint https://polaris.internal:8181
rpk cluster config set iceberg_rest_catalog_authentication_type oauth2
rpk cluster config set iceberg_rest_catalog_client_id ${POLARIS_CLIENT_ID}
rpk cluster config set iceberg_rest_catalog_client_secret ${POLARIS_CLIENT_SECRET}

# Enable per topic
rpk topic create orders \
  --config redpanda.iceberg.mode=value_schema_id_prefix \
  --config redpanda.iceberg.target_lag_ms=60000 \
  --partitions 12 --replicas 3

# 5. Mirror an existing Kafka topic into Redpanda (rpk topic mirror; MM2 also works)
rpk topic mirror \
  --source-bootstrap kafka.prod.internal:9092 \
  --source-topic orders \
  --dest-topic orders \
  --shadow-mode

# 6. Shadow-read window — a canary consumer reads from both brokers
#    and asserts that record counts and CRCs match for a 24 h window
from confluent_kafka import Consumer

def shadow_read_diff():
    kafka = Consumer({"bootstrap.servers": "kafka.prod.internal:9092",
                       "group.id": "shadow-kafka"})
    redpanda = Consumer({"bootstrap.servers": "redpanda.internal:9092",
                         "group.id": "shadow-redpanda"})
    kafka.subscribe(["orders"])
    redpanda.subscribe(["orders"])

    seen_kafka, seen_redpanda = {}, {}
    for _ in range(24 * 60 * 60):
        k = kafka.poll(1.0)
        r = redpanda.poll(1.0)
        if k and not k.error():
            seen_kafka[(k.partition(), k.offset())] = k.value()
        if r and not r.error():
            seen_redpanda[(r.partition(), r.offset())] = r.value()

    missing_in_rp = set(seen_kafka) - set(seen_redpanda)
    missing_in_k  = set(seen_redpanda) - set(seen_kafka)
    print("missing in redpanda:", len(missing_in_rp))
    print("missing in kafka:   ", len(missing_in_k))

Step-by-step trace.

Step	Before (Apache Kafka)	After (Redpanda)
Brokers	3 x m5.4xlarge + 2 TB EBS gp3 each	3 x m5.4xlarge + 2 TB gp3 each (hot) + S3 (cold)
Runtime	JVM + KRaft	Seastar C++ (no JVM)
p99 produce latency	25 ms (spikes to 200 ms on GC)	3 ms (sustained; no GC pauses)
Storage for 30-day retention	30 TB on EBS gp3 (~$3600/mo per broker)	5 TB hot EBS + 25 TB S3 tiered (~$800/mo total)
Lakehouse landing	Debezium + Kafka Connect + Iceberg-sink	Iceberg Topics native (one write)
Migration primitive	N/A	wire-compatible; rpk topic mirror + shadow read

After the POC, the streaming spend drops from ~$80k/mo to ~$25k/mo (Tiered Storage on retention-heavy topics plus the Debezium-plus-sink pipeline getting deleted), p99 produce latency stabilises under 5 ms with no GC-pause spikes, and every topic that goes through the Iceberg Topics path lands in the Polaris catalog automatically for Trino / Snowflake / DuckDB access.

Output:

Metric	Before	After
Streaming spend / month	~$80k	~$25k
p99 produce latency	25 ms + GC spikes to 200 ms	3 ms sustained
Debezium + Connect pipeline	5 services, 3 on-call runbooks	0 (Iceberg Topics replaces it)
Client migration	wire-compatible bootstrap swap	wire-compatible bootstrap swap
Failure surface	JVM GC pauses, ZK/KRaft complexity	Raft-per-partition; no JVM

Why this works — concept by concept:

• Seastar thread-per-core + io_uring — the architectural moat. One partition maps to one shard on one CPU core with pinned memory; no thread hopping, no page cache, no GC pauses. This is what removes the tail-latency floor that JVM-based brokers cannot design away.
• Kafka wire-compatibility — Redpanda implements the Kafka wire protocol including consumer groups, transactions, and exactly-once semantics. Existing Kafka clients (Java, Go, Python, Rust) keep working with only a bootstrap.servers change; no producer or consumer rewrite. This is what makes migration a config flip rather than a project.
• Tiered Storage — recent segments live on local NVMe for low-latency fetches; segments older than retention_local_target_ms are uploaded to S3/GCS/Azure Blob transparently and served from there when consumers rewind. Storage cost drops from EBS rates (~$0.10/GB-mo) to object-storage rates (~$0.023/GB-mo) for the cold portion — typically 80–95% of a retention-heavy footprint.
• Iceberg Topics — each produced record is written to both the partitioned log (for consumers) and to Iceberg data files registered in the configured REST catalog (for lakehouse engines). No Debezium, no Kafka Connect, no Iceberg-sink connector; the broker is the ETL. Trino / Snowflake / DuckDB see a native Iceberg table with schema-registry-driven schema.
• Cost — one 3-node Redpanda cluster on the same hardware as the Kafka one, plus S3 for cold segments, plus a REST catalog (Polaris) endpoint. Compared to the previous Kafka + Debezium + Connect + Iceberg-sink stack, this is one service instead of five, at 30% of the storage cost, with 5–10x better tail latency. Net O(1) per commit versus O(4-services) per commit.

Streaming
Topic — streaming
Streaming broker and Kafka-alternative problems

Practice →

Design Topic — design Design problems on streaming platform selection

Practice →

---

2. Thread-per-core + Seastar architecture

redpanda's Seastar core pins one partition-shard per CPU core with pinned memory and io_uring — this is why p99 tail latency stays flat while Kafka's spikes on every GC pause

The mental model in one line: the Seastar runtime that powers redpanda executes with exactly one thread per CPU core, pins DMA-mapped memory per core, uses io_uring for user-space asynchronous I/O without going through the kernel page cache, and maps each Kafka partition to a specific shard on a specific core so a produce or fetch never hops threads, never contends on a shared page cache, and never waits for a JVM garbage collector — this is the single architectural decision that removes the tail-latency floor JVM brokers cannot cross. Every senior data engineer evaluating Redpanda for a latency-sensitive workload has to be able to defend this design in an interview; it is the "why is Redpanda faster" answer that separates architects from users.

The four axes for the Seastar model.

• Thread model. Exactly one Seastar thread per CPU core, pinned via sched_setaffinity. No thread pool, no scheduler contention, no work-stealing across cores. A partition's ownership does not migrate at runtime; it is deterministically pinned to a shard.
• Memory model. Each shard owns a preallocated arena of pinned physical memory carved out at startup (--memory and --reserve-memory flags). No shared heap; no cross-core cache-line ping-pong; no glibc allocator contention. Memory allocation is per-shard, mostly free-list-based.
• I/O model. io_uring for storage I/O; user-space networking with DPDK-style zero-copy for the network path (kernel bypass on supported NICs). No read() / write() syscalls per packet; no page-cache indirection.
• Concurrency model. Cooperative task scheduling within a shard (Seastar futures); cross-shard messaging is explicit and rare. The broker code is written in continuation-passing style that maps to hardware without a runtime overhead.

Comparison to Kafka's execution model.

• JVM + threadpool. Kafka runs on a JVM with worker thread pools; a partition's requests can be handled by any thread. Thread hopping between cores invalidates L1/L2 caches on every hop.
• Page cache + sendfile. Kafka relies on the kernel page cache and sendfile(2) for zero-copy fetches. This works well in the steady state but is opaque to the broker — cache eviction, page faults, and dirty-page flushing all happen without the broker knowing.
• GC pauses. Even with the G1 GC or ZGC, a JVM broker under load produces occasional GC pauses in the 50–500 ms range. On a p99 SLO of 20 ms, these show up as long-tail spikes that no tuning eliminates entirely.
• KRaft / ZooKeeper. Kafka's metadata plane runs on KRaft (post-3.5) or ZooKeeper (pre-3.5). Both add a second consensus system on top of the data-plane Raft (KRaft partition raft groups).

The p99 latency story.

• Redpanda. Sub-5 ms p99 in steady state at moderate throughput (50–100k msg/s per broker, ~1 KB messages). p99.9 typically under 20 ms. No pause spikes; the runtime deterministically returns each request.
• Kafka. 10–50 ms p99 in the same conditions, with occasional 100–500 ms spikes on GC pauses. Tuning (ZGC, off-heap buffers, careful producer batching) can push p99 lower but cannot eliminate the tail.
• What this means in an interview. For workloads with a hard p99 budget (order matching, ad bidding, fraud), Redpanda changes the architecture. For workloads with a soft p99 budget (analytics ingestion, telemetry), both work.
• How to measure. OpenMessaging Benchmark, Redpanda's own franz-go load generator, or a custom producer / consumer with time.perf_counter() around produce() and end-to-end reads. Always publish p50, p95, p99, p99.9 — a single average lies.

Common interview probes on the Seastar model.

• "What's the p99 latency difference and why?" — required answer: sub-5 ms vs 10–50 ms + GC spikes; because thread-per-core + no JVM + io_uring.
• "Where does the memory come from?" — required answer: preallocated per-shard arenas; pinned; no shared heap.
• "How does Redpanda handle a hot partition?" — required answer: rebalance the partition to a less-loaded shard; a partition's ownership can be moved but only between shards, not shared across shards.
• "Does Redpanda use the page cache?" — required answer: no, it uses io_uring + direct I/O; the broker owns the caching decisions.

Worked example — capacity planning for a 6-node Redpanda cluster

Detailed explanation. Sizing a Redpanda cluster is largely a per-core exercise: pick the target throughput, divide by the per-core throughput the Seastar model achieves, add headroom, and count the cores. Because the model pins one shard per core, per-broker throughput is roughly linear in core count once you have enough NIC and disk bandwidth to feed the cores. Walk through the calculation for a 6-node cluster targeting 300k msg/s of 1 KB messages with 30-day retention.

• Target. 300k msg/s sustained, ~1 KB messages, replicated 3x.
• Per-core throughput. ~20k msg/s per Seastar shard for 1 KB messages at RF=3 (measured baseline; tune per hardware).
• Cores needed. 300k / 20k = 15 cores minimum; add 50% headroom = ~23 cores; distributed across 6 brokers = ~4 cores each.
• Instance shape. m6i.2xlarge (8 vCPU, 32 GB RAM, up to 12.5 Gbps network). 4 shards dedicated to Redpanda; 4 vCPU headroom for OS + monitoring.

Question. Compute the per-broker and total capacity, choose an instance shape, and lay out the shard-per-core allocation.

Input.

Parameter	Value
Target sustained throughput	300k msg/s
Average message size	1 KB
Replication factor	3
Retention	30 days (hot 24 h on SSD; cold 29 days on S3)
Per-core throughput (baseline)	20k msg/s @ 1 KB, RF=3
Headroom	50%

Code.

# capacity_planning.py — size a Redpanda cluster
def size_cluster(
    target_msg_s: int,
    avg_msg_bytes: int,
    replication_factor: int,
    per_core_msg_s: int,
    headroom_pct: float,
    brokers: int,
) -> dict:
    """Return sizing recommendation for a Redpanda cluster."""

    # 1. Replicated throughput (each message is written RF times)
    replicated_msg_s = target_msg_s * replication_factor

    # 2. Cores needed, before and after headroom
    min_cores  = replicated_msg_s / per_core_msg_s
    plan_cores = min_cores * (1 + headroom_pct)

    # 3. Cores per broker (round up)
    cores_per_broker = -(-int(plan_cores) // brokers)   # ceil div

    # 4. Choose a shape with 2x cores (so half the box is Redpanda,
    #    half is OS + agents + headroom for hot partitions)
    vcpus_per_broker = cores_per_broker * 2

    # 5. RAM sizing — Seastar preallocates per shard; 1 GiB per shard is a good start
    ram_per_broker_gib = cores_per_broker * 4    # 4 GiB per shard including overhead

    # 6. Bandwidth — inbound produce + outbound replication + fetches
    inbound_gbps  = (replicated_msg_s * avg_msg_bytes * 8) / (brokers * 1e9)
    outbound_gbps = inbound_gbps * 2  # producer + consumer

    return {
        "brokers":              brokers,
        "cores_per_broker":     cores_per_broker,
        "vcpus_per_broker":     vcpus_per_broker,
        "ram_per_broker_gib":   ram_per_broker_gib,
        "inbound_gbps":         round(inbound_gbps, 2),
        "outbound_gbps":        round(outbound_gbps, 2),
        "instance_shape_hint":  f"{vcpus_per_broker} vCPU, {ram_per_broker_gib} GiB",
    }


print(size_cluster(
    target_msg_s=300_000,
    avg_msg_bytes=1024,
    replication_factor=3,
    per_core_msg_s=20_000,
    headroom_pct=0.5,
    brokers=6,
))
# -> {'brokers': 6, 'cores_per_broker': 3, 'vcpus_per_broker': 6,
#     'ram_per_broker_gib': 12, 'inbound_gbps': 1.23, 'outbound_gbps': 2.46,
#     'instance_shape_hint': '6 vCPU, 12 GiB'}

Step-by-step explanation.

1. Replication is the first multiplier: 300k msg/s produced becomes 900k msg/s of broker-side writes because each message is durably stored on three brokers. This is the throughput the cluster actually has to handle.
2. Per-core throughput comes from benchmarking on the target hardware — 20k msg/s per shard for 1 KB RF=3 is a conservative published baseline; NVMe-heavy hardware can push 40–60k msg/s per shard. Use your benchmark number in production sizing.
3. Headroom of 50% is the standard defensive number — it absorbs a broker failure (2 out of 6 shard groups suddenly under 33% higher load), a rebalance, and normal traffic growth. Skipping headroom is the single most common capacity mistake.
4. Instance-shape rule of thumb: choose a vCPU count twice the Redpanda shard count. Half the box is Redpanda (pinned via --cpuset); half is OS, monitoring agents, rpk operations, and hot-partition headroom. m6i.2xlarge (8 vCPU) matches 3 Redpanda shards + 5 vCPU for everything else.
5. Memory sizing is per-shard: 1 GiB minimum per Seastar shard for the arena, plus 2–4 GiB for the broker's own caches (index, RaftGroupMap, cloud-storage upload buffers). At 3 shards per broker, 12 GiB is comfortable; 16 GiB is safer.

Output.

Attribute	Value
Brokers	6
Redpanda shards per broker	3 (cores pinned)
Total shards	18
Aggregate per-core capacity	18 × 20k = 360k msg/s RF=3 (= 120k msg/s produced)
With headroom	300k msg/s produced target absorbed comfortably
Instance shape	m6i.2xlarge (8 vCPU, 32 GB, 12.5 Gbps)
Local hot storage	100 GiB per shard = 300 GiB per broker = 1.8 TiB cluster hot
S3 tiered cold	~25 TiB (29-day tail of 300k msg/s × 1 KB)

Rule of thumb. Size Redpanda by shard count, not by broker count. Bench the per-shard throughput on your actual hardware, add 50% headroom, then pick brokers and instances so that vCPU count is twice the pinned shard count. Never let a Seastar shard share a core with an OS process.

Worked example — how shard-per-partition affects a hot key

Detailed explanation. Because Redpanda pins each partition to a specific shard on a specific core, a single "hot" partition — one that receives disproportionate traffic — can saturate a single shard while others sit idle. This is the same problem Kafka has with an unbalanced partition distribution, but Redpanda's deterministic pinning makes the failure mode more visible. Walk through diagnosis and fix.

• Symptom. One CPU core pegged at 100%; broker-level throughput plateaued; other cores at 30%.
• Root cause. A hot key in the producer key-space maps to one partition, which is pinned to one shard on one core. The shard cannot process faster than one core can go.
• Fix options. (a) Repartition the topic with more partitions and re-key producers; (b) rebalance the hot partition to a less-loaded shard via rpk cluster partition-movement; (c) if key distribution is fundamentally skewed, add a random suffix to keys and re-aggregate downstream.

Question. Design the diagnosis workflow using rpk and Prometheus metrics, then apply the fix.

Input.

Metric	Symptom	Threshold
redpanda_reactor_utilization per shard	shard 5 at 98%, others 30–50%	> 80% sustained = hot shard
redpanda_kafka_produce_latency_seconds	p99 3x baseline	> 2x sustained = investigate
redpanda_partition_bytes_produced_total per partition	partition 42 at 8x median	> 5x median = hot partition
Producer key cardinality	one key = 40% of traffic	should be uniform-ish

Code.

# 1. Identify hot shard via reactor utilization
curl -s http://redpanda-broker-2:9644/metrics \
  | grep 'redpanda_reactor_utilization' \
  | sort -k2 -n -r | head

# 2. Find hot partition by produced bytes
rpk topic describe orders --print-partitions | head -20

# 3. Sample key distribution from a recent segment
rpk topic consume orders --num 100000 --format '%k\n' \
  | sort | uniq -c | sort -n -r | head

# 4. Add more partitions (double from 12 -> 24) to spread the load
rpk topic alter-partition-count orders 24

# 5. Rebalance to distribute partitions across shards evenly
rpk cluster partition-movement --to-broker-set 1,2,3,4,5,6

# 6. Producer-side fix — add a low-entropy suffix to hot keys and
#    aggregate downstream. Only for genuinely skewed keys.
import hashlib
from confluent_kafka import Producer

producer = Producer({"bootstrap.servers": "redpanda:9092"})

HOT_KEY_SUFFIX_BUCKETS = 8   # keep aggregation cheap downstream

def salted_key(k: str) -> str:
    if k == "vip_customer_42":     # known hot key
        # deterministic per-message salt via first 3 bits of hash
        salt = int(hashlib.md5(k.encode()).digest()[0]) % HOT_KEY_SUFFIX_BUCKETS
        return f"{k}#{salt}"
    return k

def emit(order: dict) -> None:
    producer.produce(
        topic="orders",
        key=salted_key(order["customer_id"]).encode(),
        value=order["payload"].encode(),
    )

Step-by-step explanation.

1. Diagnosis starts with redpanda_reactor_utilization — the per-shard busy percentage. If one shard is above 80% while others sit at 30–50%, you have a hot shard. Grafana with a per-shard heatmap makes this a one-glance diagnosis.
2. Correlate with redpanda_partition_bytes_produced_total per partition to find the hot partition. In a well-distributed topic no partition should carry more than ~2x the median; anything above 5x is a hot-partition problem, not a hot-shard bug.
3. Sample producer keys with rpk topic consume ... --format '%k\n' piped through sort | uniq -c. This tells you whether the hot partition is caused by (a) too few partitions for the key cardinality, or (b) genuinely skewed key distribution (a single VIP customer, a single popular product).
4. For (a), the fix is rpk topic alter-partition-count to spread the key hash across more partitions. Redpanda repartitions online without downtime; producer clients pick up the new partition count on the next metadata refresh.
5. For (b), the producer-side fix is to append a low-entropy salt (e.g. 3 bits = 8 buckets) to the hot key. This spreads the hot key across 8 partitions at the cost of an 8x-fanout on downstream aggregators. The HOT_KEY_SUFFIX_BUCKETS constant is the tunable trade-off between shard balance and downstream complexity.

Output.

Before fix	After fix
shard 5 at 98%, others 30–50%	all shards at 40–55%
p99 produce latency 15 ms	p99 produce latency 4 ms
partition 42 at 8x median throughput	partition 42 + 7 salted partitions at 1.1x median each
12 partitions on orders	24 partitions on orders
Hot key = 40% of traffic on one partition	Hot key = 5% each on 8 partitions

Rule of thumb. Never diagnose Redpanda latency spikes without looking at per-shard reactor utilisation first. The Seastar shard model makes hot-partition problems visible as hot-shard problems; fix them at the partitioning layer, not by throwing more brokers at the cluster.

Senior interview question on Seastar architecture

A senior interviewer might ask: "Explain why Redpanda claims sub-5 ms p99 latency where Kafka lands at 20–50 ms with GC-pause spikes. Walk me through the Seastar thread-per-core model, the io_uring path, the memory-pinning story, and the failure modes a hot key can trigger. Then benchmark it against Kafka on the same hardware."

Solution Using a franz-go load generator + reactor-utilization dashboards + hot-key salting

// 1. franz-go load generator (Redpanda's canonical Kafka client for Go)
package main

import (
    "context"
    "fmt"
    "log"
    "time"

    "github.com/twmb/franz-go/pkg/kgo"
)

func main() {
    client, err := kgo.NewClient(
        kgo.SeedBrokers("redpanda-1:9092", "redpanda-2:9092", "redpanda-3:9092"),
        kgo.ProducerBatchCompression(kgo.LZ4Compression()),
        kgo.RequiredAcks(kgo.AllISRAcks()),
        kgo.ProducerLinger(1*time.Millisecond),
        kgo.MaxBufferedRecords(1_000_000),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()

    // Produce 1 KB messages as fast as possible for 60 s; capture per-record latency
    payload := make([]byte, 1024)
    deadline := time.Now().Add(60 * time.Second)
    sent, ackd := 0, 0
    var latSum, latMax time.Duration

    for time.Now().Before(deadline) {
        start := time.Now()
        client.Produce(context.Background(),
            &kgo.Record{Topic: "bench", Value: payload},
            func(r *kgo.Record, err error) {
                if err != nil {
                    log.Println("produce err:", err)
                    return
                }
                lat := time.Since(start)
                latSum += lat
                if lat > latMax {
                    latMax = lat
                }
                ackd++
            })
        sent++
    }
    client.Flush(context.Background())

    fmt.Printf("sent=%d ackd=%d avg_latency=%v max_latency=%v msg_s=%.0f\n",
        sent, ackd, latSum/time.Duration(ackd), latMax, float64(ackd)/60.0)
}

# 2. Scrape per-shard reactor utilization for a Grafana heatmap
curl -s http://redpanda-broker-2:9644/metrics \
  | awk '/^redpanda_reactor_utilization/ {print $1, $2}'
# redpanda_reactor_utilization{shard="0"} 0.42
# redpanda_reactor_utilization{shard="1"} 0.51
# redpanda_reactor_utilization{shard="2"} 0.47
# ... (one line per pinned shard)

# 3. Compare to Kafka on the same hardware with kafka-producer-perf-test.sh
/opt/kafka/bin/kafka-producer-perf-test.sh \
  --topic bench \
  --num-records 60000000 \
  --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=kafka-1:9092 acks=all compression.type=lz4

# 4. Prometheus alert rules — the on-call runbook
groups:
  - name: redpanda_shard_hotness
    rules:
      - alert: RedpandaHotShard
        expr: max by (instance, shard) (redpanda_reactor_utilization) > 0.85
        for: 5m
        annotations:
          summary: "Redpanda shard {{$labels.shard}} on {{$labels.instance}} > 85% for 5m"
          runbook: "check hot partition; rpk topic alter-partition-count or salt hot key"

      - alert: RedpandaProduceP99High
        expr: histogram_quantile(0.99, sum by (le) (rate(redpanda_kafka_produce_latency_seconds_bucket[5m]))) > 0.020
        for: 10m
        annotations:
          summary: "Redpanda produce p99 > 20 ms for 10m"
          runbook: "correlate with RedpandaHotShard; if none, check disk saturation"

Step-by-step trace.

Layer	Kafka behaviour	Redpanda behaviour
Request thread	worker threadpool; may hop cores	pinned Seastar shard; never hops
Memory	shared JVM heap; GC pauses	per-shard preallocated arena; no GC
Storage I/O	write() + page cache + sendfile	io_uring + direct I/O
Metadata	KRaft consensus group	in-broker controller + per-partition Raft
p99 latency @ 50k msg/s	20–50 ms + GC spikes	3–5 ms sustained
p99.9 latency	200–500 ms occasional	15–20 ms
CPU efficiency	~50% (JVM overhead + kernel)	~85% (no JVM, no kernel-side page cache)

After the benchmark, franz-go reports p99 of ~3.5 ms sustained on the Redpanda cluster vs ~28 ms on Kafka; per-shard reactor utilisation shows a flat 45–55% distribution across all shards; the alert rules catch a synthetic hot key within 5 minutes and fire the RedpandaHotShard runbook.

Output:

Metric	Value
Sustained throughput per shard	~20k msg/s @ 1 KB RF=3
p99 produce latency (steady state)	3–5 ms
p99.9 produce latency	15–20 ms
GC-pause spike frequency	0 (no JVM)
Reactor utilisation ceiling before alert	85%
Hot-key detection SLA	~5 min from onset

Why this works — concept by concept:

• Seastar thread-per-core — one thread per CPU core, pinned via sched_setaffinity. No thread-pool scheduler contention, no work-stealing, no cross-core cache-line ping-pong. Every partition's requests are handled by exactly one thread on one core.
• io_uring + direct I/O — asynchronous storage I/O submitted via io_uring submission queues; completions polled from completion queues; no read() / write() syscalls per operation; no kernel page cache. The broker owns the caching decisions and knows exactly what is on disk vs in memory.
• Pinned per-shard memory — each shard preallocates its arena at startup with pinned physical pages. No shared heap; no malloc contention; no glibc allocator overhead. Memory allocation inside a shard is mostly free-list-based and constant-time.
• No JVM + no ZooKeeper — the whole broker is a single C++ binary. No GC pauses; no separate metadata cluster to coordinate with; failover is a Raft leader election on the affected partitions only.
• Cost — Redpanda pays for this architecture with higher C++ development complexity (memory safety, arena allocator, Seastar future chains) and a smaller ecosystem than Kafka's. The benefit is deterministic sub-5 ms p99 that no JVM broker can match. For any workload with a real tail-latency SLO, this is a step-change improvement. Net O(1) per request tail latency versus O(GC_pause_frequency) for the JVM.

Streaming
Topic — streaming
Streaming latency and broker-tuning problems

Practice →

Design Topic — design Design problems on low-latency streaming systems

Practice →

---

3. Tiered storage + Redpanda Cloud

redpanda tiered storage keeps recent segments on local NVMe and offloads cold segments to S3/GCS/Azure Blob — the 6x TCO story for retention-heavy topics

The mental model in one line: redpanda tiered storage is the design where a topic's most recent log segments live on local NVMe (hot tier, low-latency fetches, standard replication) while segments older than a configurable local-retention threshold are uploaded to an S3/GCS/Azure Blob bucket (cold tier, object-storage price, served transparently on consumer rewind) — and this is the single design decision that turns Redpanda's cost story from "similar to Kafka" into "6x cheaper than Kafka" on retention-heavy workloads because 80–95% of a 30-day retention footprint lives in the cold tier at $0.023/GB-mo rather than $0.10+/GB-mo for EBS. Every senior architect evaluating a streaming platform for retention above ~7 days needs to be able to defend this design in an interview.

The four axes for Redpanda Tiered Storage.

• Hot / cold split. Local NVMe for recent segments (typically last 12–48 h of retention); S3 / GCS / Azure Blob for anything older. The split is controlled per topic by retention.local.target.ms and retention.local.target.bytes; the total retention is bounded by retention.ms or retention.bytes.
• Read path. Consumers reading recent offsets hit the local segments directly at NVMe latency. Consumers rewinding into cold offsets trigger a transparent fetch of the relevant S3 objects, cached locally for the duration of the fetch. A cold-tier fetch adds ~50–200 ms first-byte latency.
• Write path. All writes hit the local hot tier first (standard Raft replication across brokers). A background uploader promotes segments to S3 asynchronously as they age past the local-retention threshold. No producer path change.
• Ops story. Local disk shrinks from "hold 30 days of everything" to "hold 24–48 hours of everything." A 30 TB local footprint per broker becomes 3–5 TB; the rest lives in a single S3 bucket at object-storage prices. Disk-full incidents become rare.

Remote Read Replicas — the read-only tier.

• What it is. A read-only Redpanda cluster that shares the same tiered-storage bucket as the write cluster. Consumers point at the read-replica cluster for fetches; writes stay on the primary.
• Why. Offload analytics-heavy read workloads (batch consumers, ad-hoc replays, dashboards) from the primary cluster. Independent scaling of read vs write capacity.
• How. Configure the replica cluster with the same cloud_storage_bucket and enable read-only mode; the replica's controller becomes aware of every topic and its segments via the object-storage manifest.

Bring-Your-Own-Cloud (BYOC) with Redpanda Cloud.

• The model. Redpanda operates the control plane (Kubernetes operator, upgrades, monitoring); customer runs the data plane in their own VPC. Data never leaves the customer's cloud account.
• Why it matters. Compliance requirements (HIPAA, PCI, SOC 2), sovereignty requirements (EU data residency), egress-cost requirements. BYOC keeps the data plane local while paying only for control-plane operation.
• Comparison. Confluent Cloud (dedicated) puts data in Confluent's account by default; BYOC is available but has a smaller feature set. Redpanda's BYOC is the default deployment for privacy-sensitive teams.

Common interview probes on Tiered Storage.

• "How does Redpanda's tiered storage differ from Kafka's KIP-405?" — Kafka's tiered storage is available but implementation quality varies by distro; Redpanda's is production-hardened and the same across all deployments.
• "What's the cold-tier fetch latency?" — ~50–200 ms first-byte for the first object; steady stream after that.
• "How do you size local disk under tiered storage?" — 24–48 hours of expected hot data + 20% headroom + WAL for in-progress writes.
• "Can I query cold segments directly from S3?" — Yes via Remote Read Replicas or via Iceberg Topics (which pre-projects the data into Iceberg format).

Worked example — enabling Tiered Storage on an existing topic

Detailed explanation. Turning on Tiered Storage is a cluster-level enable plus per-topic policy. Existing topics don't upload retroactively; they start uploading segments produced after enablement. Walk through the enable, the per-topic policy, and the verification.

• Cluster config. cloud_storage_enabled: true, bucket, region, credentials source.
• Topic policy. retention.local.target.ms=86400000 (24 h hot on SSD), retention.ms=2592000000 (30 days total).
• Verify. rpk cluster storage summary and per-topic rpk topic describe --show-partition-summary.

Question. Enable Tiered Storage cluster-wide, apply the policy to the orders topic, and verify the upload is happening.

Input.

Setting	Value
Bucket	s3://acme-redpanda-tiered
Region	us-east-1
Credentials source	aws_instance_metadata (IRSA/IMDS)
Local hot retention	24 h
Total retention	30 days
Topic	orders (12 partitions, RF=3)

Code.

# 1. Cluster-level enable — /etc/redpanda/redpanda.yaml (excerpt)
redpanda:
  cloud_storage_enabled: true
  cloud_storage_region: us-east-1
  cloud_storage_bucket: acme-redpanda-tiered
  cloud_storage_credentials_source: aws_instance_metadata

  # Upload behaviour
  cloud_storage_segment_max_upload_interval_sec: 3600
  cloud_storage_upload_ctrl_max_shares: 4
  cloud_storage_max_connections: 20

  # Local retention default — 24 h hot per partition
  retention_local_target_ms_default: 86400000

# 2. Restart brokers rolling (Kubernetes: helm rollout; VMs: systemctl per broker)
sudo systemctl restart redpanda   # per broker, one at a time

# 3. Verify cluster-level enablement
rpk cluster config get cloud_storage_enabled
# true
rpk cluster config get cloud_storage_bucket
# acme-redpanda-tiered

# 4. Apply per-topic policy for `orders`
rpk topic alter-config orders \
  --set retention.local.target.ms=86400000 \
  --set retention.ms=2592000000 \
  --set redpanda.remote.write=true \
  --set redpanda.remote.read=true

# 5. Watch the uploader progress
rpk cluster storage summary
# per-cluster: local_size=250 GiB, cloud_size=8 TiB
rpk topic describe orders --show-partition-summary
# per-partition: local segments 0-42, cloud segments 43-1200

# 6. Confirm objects in S3
aws s3 ls s3://acme-redpanda-tiered/orders/ --recursive | head
# 2026-07-15 03:12:00     67108864 orders/0_36/0-42-1234-v1.log
# ...

# 7. Optional — programmatic health check for the uploader
import subprocess, json

def check_uploader_health(topic: str) -> dict:
    """Return uploader stats for a topic across all partitions."""
    out = subprocess.check_output(
        ["rpk", "topic", "describe", topic, "--print-partitions", "--format", "json"]
    )
    parts = json.loads(out)["partitions"]
    return {
        "partitions":    len(parts),
        "cloud_offset":  sum(p.get("high_offset", 0) - p.get("start_offset", 0) for p in parts),
        "local_offset":  sum(p.get("high_offset", 0) for p in parts),
        "healthy":       all(p.get("leader", -1) != -1 for p in parts),
    }

print(check_uploader_health("orders"))

Step-by-step explanation.

1. Cluster-level enable is one config change (cloud_storage_enabled: true) plus the bucket / region / credentials. aws_instance_metadata uses IMDSv2 or IRSA on EKS — the broker never sees a static access key. Rolling restart applies the config without cluster downtime.
2. Per-topic policy sets local retention (24 h) and total retention (30 days). Segments age from local to cloud automatically as they cross the local threshold. redpanda.remote.write=true opts the topic into tiered storage; redpanda.remote.read=true allows consumers to rewind into cold segments.
3. Verification is two commands: rpk cluster storage summary shows the local-vs-cloud split at cluster level; rpk topic describe --show-partition-summary shows the per-partition segment map. In steady state, local segments should be ≤ 24 h old; cloud segments should hold everything older.
4. Confirming objects in S3 is a good post-deploy sanity check: aws s3 ls on the topic prefix should show a growing set of .log and .index files. If the bucket is empty after 24 h of running, the uploader is stuck — check IAM permissions and the redpanda_cloud_storage_upload_error_total metric.
5. The programmatic health check in Python calls rpk topic describe and asserts every partition has a leader. In production, wire this into your health-check endpoint or a Prometheus exporter so uploader lag becomes an alertable signal.

Output.

State after enable	Value
Local hot data on orders per broker	~150 GiB (24 h at 300k msg/s / 6 brokers)
Cloud tier size on orders	grows to ~8 TiB over 30 days
Producer path	unchanged — still writes to local first
Consumer at head (recent offsets)	reads from local NVMe
Consumer rewinding to 20-day-old data	fetches from S3 (~100 ms first byte, then streams)
Disk-full risk	eliminated — local footprint bounded

Rule of thumb. For any Redpanda topic with retention > 48 h, enable Tiered Storage. Size local disk for 24–48 hours of expected retention + 20% headroom. The single biggest cost win of Redpanda over Kafka is realised at exactly this configuration.

Worked example — TCO math for a 100 TB retention scenario

Detailed explanation. The "6x TCO win" claim is real for retention-heavy workloads but has to be defended with concrete numbers. Walk through the calculation for a 100 TB retention scenario (typical for a 30-day telemetry / CDC firehose) and compare EBS-only Kafka against Redpanda Tiered Storage.

• Data footprint. 100 TB retention (30 days at ~3.3 TB / day, RF=3 → 10 TB/day of broker storage).
• Kafka baseline. All 100 TB on EBS gp3 across brokers. Cost: 100000 GB × $0.08/GB-mo (gp3) = $8000/mo for storage.
• Redpanda Tiered. 5 TB hot on EBS + 95 TB cold on S3 Standard. Cost: 5000 × $0.08 + 95000 × $0.023 = $400 + $2185 = ~$2585/mo for storage.
• Additional cold-tier costs. S3 PUT charges (once per segment upload) and GET charges (per rewind). Typically <$100/mo for a well-tuned uploader.

Question. Compute the TCO delta, sensitivity to hot-tier size, and the crossover point where Redpanda's Tiered Storage stops winning.

Input.

Component	Kafka (EBS-only)	Redpanda (Tiered)
Total retention footprint	100 TB	100 TB (5 TB hot + 95 TB cold)
Storage $/GB-mo	$0.08 (gp3)	$0.08 (hot) + $0.023 (S3 Standard)
Storage $/mo	$8000	$2585 + ~$100 (PUT/GET) = ~$2685
Cross-AZ replication egress	~$500	~$500 (same)
Broker compute (m5.4xlarge x 3)	~$1800	~$1800 (same)
Total $/mo	~$10,300	~$4,985

Code.

# tco_calculator.py — TCO for Kafka EBS-only vs Redpanda Tiered
def tco(
    retention_tb: float,
    hot_tb: float,
    ebs_price_gb_mo: float = 0.08,
    s3_price_gb_mo: float = 0.023,
    brokers: int = 3,
    broker_price_hr: float = 0.768,     # m5.4xlarge on-demand
    egress_mo: float = 500,
    s3_ops_mo: float = 100,
) -> dict:
    """Return monthly TCO breakdown for Kafka vs Redpanda Tiered."""
    hours_mo = 730

    # Kafka — all on EBS
    kafka_storage = retention_tb * 1024 * ebs_price_gb_mo
    kafka_compute = brokers * broker_price_hr * hours_mo
    kafka_total   = kafka_storage + kafka_compute + egress_mo

    # Redpanda — hot on EBS, cold on S3
    rp_hot     = hot_tb * 1024 * ebs_price_gb_mo
    rp_cold    = (retention_tb - hot_tb) * 1024 * s3_price_gb_mo
    rp_storage = rp_hot + rp_cold + s3_ops_mo
    rp_compute = brokers * broker_price_hr * hours_mo
    rp_total   = rp_storage + rp_compute + egress_mo

    return {
        "kafka_storage_usd":    round(kafka_storage, 0),
        "kafka_compute_usd":    round(kafka_compute, 0),
        "kafka_total_usd":      round(kafka_total, 0),
        "redpanda_storage_usd": round(rp_storage, 0),
        "redpanda_compute_usd": round(rp_compute, 0),
        "redpanda_total_usd":   round(rp_total, 0),
        "savings_pct":          round((kafka_total - rp_total) / kafka_total * 100, 1),
    }


# 100 TB retention scenario
print(tco(retention_tb=100, hot_tb=5))
# -> {'kafka_storage_usd': 8192, 'kafka_compute_usd': 1682,
#     'kafka_total_usd': 10374, 'redpanda_storage_usd': 2645,
#     'redpanda_compute_usd': 1682, 'redpanda_total_usd': 4827,
#     'savings_pct': 53.5}

# Sensitivity — same 100 TB retention but only 12 h hot (2 TB)
print(tco(retention_tb=100, hot_tb=2))
# -> savings closer to 60%

# Crossover — 7-day retention where hot tier is 2 days out of 7
print(tco(retention_tb=25, hot_tb=7))
# -> savings drop to ~25%

Step-by-step explanation.

1. The Kafka baseline is dominated by EBS storage: 100 TB × $0.08/GB-mo = $8192 for storage vs $1682 for compute. Storage is 80% of the bill. This is the CFO's line-item concern.
2. Redpanda's Tiered Storage flips the math: only 5 TB stays on EBS (hot), the other 95 TB moves to S3 Standard. Cold storage costs $0.023/GB-mo vs $0.08 — a 3.5x per-GB win that compounds against 95% of the footprint.
3. The compute cost is unchanged (same 3 m5.4xlarge brokers); the egress cost is unchanged (cross-AZ replication for hot writes still happens). Only the storage bill changes, but it dominates the total.
4. Sensitivity analysis shows the win grows as hot-tier shrinks. If you can tolerate a 12 h hot window instead of 24 h, savings push toward 60%. If you push hot to 48 h, savings drop but stay above 40%.
5. The crossover point where Redpanda Tiered stops winning is roughly "retention < 7 days and hot tier > 40% of retention." At short retention (e.g. 3 days with 24 h hot), the S3 cold tier is only 66% of the footprint and the per-GB savings don't compound enough to offset the S3 PUT/GET fixed costs.

Output.

Scenario	Kafka $/mo	Redpanda $/mo	Savings
100 TB retention, 5 TB hot (24 h)	$10,374	$4,827	53%
100 TB retention, 2 TB hot (12 h)	$10,374	$4,140	60%
25 TB retention, 7 TB hot (2 days)	$3,730	$2,833	24%
10 TB retention, 5 TB hot (1 day)	$1,772	$1,558	12%
500 TB retention, 10 TB hot (24 h)	$42,132	$12,930	69%

Rule of thumb. Redpanda Tiered Storage wins big when retention > 14 days and hot tier can be < 20% of total. Below 7 days retention, the savings compress and either broker is fine. Above 30 days retention, the savings are the whole reason to migrate.

Worked example — Remote Read Replicas for a backfill-heavy workload

Detailed explanation. A common pattern in analytics-heavy teams is that a small subset of consumers (batch pipelines, backfill jobs, ad-hoc replays) do most of the reads while a small subset of producers do most of the writes. On a single cluster, backfill reads compete with fresh writes for shard capacity. Remote Read Replicas isolate the two workloads: a read-only replica cluster shares the same tiered-storage bucket as the primary, serves batch consumers from S3, and never impacts producer latency on the primary.

• The problem. A nightly Spark job replays the last 7 days of 12 topics, saturating fetch bandwidth on the primary and pushing produce p99 from 3 ms to 25 ms during the replay window.
• The fix. Deploy a Remote Read Replica cluster reading from the same S3 bucket; point the Spark job at the replica; produce p99 on the primary stays flat.
• The cost. One extra small Redpanda cluster (e.g. 3 x m5.xlarge) dedicated to reads. Reads primarily hit S3, so local disk on the replica can be minimal (~500 GB).

Question. Deploy the replica cluster and cut the batch workload over to it.

Input.

Component	Value
Replica cluster brokers	3 x m5.xlarge
Replica local disk	500 GB gp3 each (headroom for S3-cache)
Shared bucket	s3://acme-redpanda-tiered
Read-only enforcement	at broker config + IAM (S3 read-only)
Batch workload	Spark job replaying 7 d × 12 topics

Code.

# Remote Read Replica cluster — /etc/redpanda/redpanda.yaml (RRR cluster)
redpanda:
  cloud_storage_enabled: true
  cloud_storage_region: us-east-1
  cloud_storage_bucket: acme-redpanda-tiered
  cloud_storage_credentials_source: aws_instance_metadata

  # Read-only mode for this cluster (no local writes allowed)
  cloud_storage_read_only_mode: true

  # Cache size — S3 objects fetched by the RRR are cached locally
  cloud_storage_cache_size: 400_000_000_000     # ~400 GB

# Spark job configuration change — point at the RRR bootstrap
spark-submit \
  --conf spark.kafka.bootstrap.servers=redpanda-rrr:9092 \
  --conf spark.kafka.group.id=spark-backfill-nightly \
  --conf spark.kafka.starting.offsets=earliest \
  s3://acme-jobs/backfill.py

Step-by-step explanation.

1. The Remote Read Replica cluster runs the same Redpanda binary with cloud_storage_read_only_mode: true. It has no writable topics; every read is served either from the local S3-object cache or by fetching from S3.
2. The shared bucket is the coordination point. The RRR cluster's controller learns about topics and partitions from the object-storage manifest that the primary writes. There is no direct broker-to-broker replication between the primary and the RRR.
3. Sizing the RRR is a read-workload exercise: 3 brokers of m5.xlarge (4 vCPU, 16 GB, 3 shards each) handle the Spark job's read fan-out; local disk is sized for the S3-cache working set (400 GB is plenty for a nightly workload).
4. Cutting Spark over is a one-line change: spark.kafka.bootstrap.servers=redpanda-rrr:9092. The Kafka client protocol is identical, so no code change is needed. Existing consumer groups (with different names) work in parallel on the primary and the RRR.
5. Monitoring: the RRR's redpanda_cloud_storage_cache_hit_ratio metric tells you whether the working set fits in the cache. Under 0.5 hit ratio for hours means the cache is undersized; over 0.9 means it's oversized (can shrink to save cost).

Output.

Metric	Before (single cluster)	After (with RRR)
Primary produce p99 during replay	25 ms	3 ms (unchanged)
Backfill throughput	1.5 GB/s from primary	2.5 GB/s from RRR (S3-parallel)
Cost of RRR cluster	N/A	~$800/mo (3 x m5.xlarge + 1.5 TB gp3)
Blast radius of RRR failure	N/A	0 (writes unaffected)
Operational complexity	1 cluster	2 clusters (same binary, same config)

Rule of thumb. Deploy a Remote Read Replica the moment any consumer workload measurably degrades produce latency on the primary. The blast-radius isolation alone is worth the extra ~$800/mo; the throughput win on S3-parallel reads is a bonus.

Senior interview question on Tiered Storage

A senior interviewer might ask: "You run a Redpanda cluster carrying a 30-day CDC firehose with 100 TB total retention. Half your read workload is a nightly Spark backfill; the other half is real-time consumers. Design the Tiered Storage policy, the Remote Read Replica setup, the sizing, and the cost model. Then defend the design against a CFO who is asking why streaming still costs $5k/month."

Solution Using Tiered Storage with a 24 h hot window, a Remote Read Replica for backfill, and BYOC deployment

# 1. Primary cluster — Tiered Storage + write-optimised
redpanda:
  cloud_storage_enabled: true
  cloud_storage_region: us-east-1
  cloud_storage_bucket: acme-redpanda-tiered
  cloud_storage_credentials_source: aws_instance_metadata

  # Upload aggressively so cold data lands in S3 within 1 h
  cloud_storage_segment_max_upload_interval_sec: 3600
  cloud_storage_upload_ctrl_max_shares: 6
  cloud_storage_max_connections: 30

  # Local retention: 24 h hot; total: 30 days
  retention_local_target_ms_default: 86400000

  # Housekeeping — recycle local segments as soon as they're safely uploaded
  cloud_storage_housekeeping_interval_ms: 60000

# 2. Remote Read Replica cluster — read-only; same bucket
redpanda:
  cloud_storage_enabled: true
  cloud_storage_region: us-east-1
  cloud_storage_bucket: acme-redpanda-tiered
  cloud_storage_credentials_source: aws_instance_metadata
  cloud_storage_read_only_mode: true

  # Sized for the Spark working set
  cloud_storage_cache_size: 400_000_000_000

# 3. Per-topic policy — 30 days retention, 24 h hot
for t in orders customers shipments events audit_logs cdc_stream_1 cdc_stream_2 \
         cdc_stream_3 cdc_stream_4 cdc_stream_5 cdc_stream_6 cdc_stream_7; do
  rpk topic alter-config "$t" \
    --set redpanda.remote.write=true \
    --set redpanda.remote.read=true \
    --set retention.local.target.ms=86400000 \
    --set retention.ms=2592000000
done

# 4. Point real-time consumers at primary; batch consumers at RRR
# real-time apps  -> bootstrap.servers=redpanda-primary:9092
# spark backfill  -> bootstrap.servers=redpanda-rrr:9092

# 5. Cost projection — call the calculator from the previous example
from tco_calculator import tco

primary   = tco(retention_tb=100, hot_tb=5, brokers=6, broker_price_hr=0.768)
rrr_extra = 3 * 0.192 * 730 + 500 * 3 * 0.08     # 3 x m5.xlarge + 1.5 TB gp3
print("primary $/mo:", primary["redpanda_total_usd"])
print("rrr extra:   ", round(rrr_extra, 0))
print("total:       ", round(primary["redpanda_total_usd"] + rrr_extra, 0))

Step-by-step trace.

Layer	Component	Purpose
Primary cluster	6 brokers + Tiered Storage	write path; real-time consumers
Object storage	s3://acme-redpanda-tiered	cold tier for both clusters
RRR cluster	3 brokers, read-only	batch backfill isolation
Per-topic policy	24 h hot, 30 d total	bounded local disk
Real-time consumers	primary bootstrap	sub-5 ms fetches from NVMe
Batch consumers	RRR bootstrap	S3-parallel fan-out; no primary impact
BYOC deployment	Redpanda Cloud control plane + customer VPC data plane	compliance + sovereignty

After deployment, the primary cluster carries the write path at 3–5 ms p99, the RRR cluster carries the Spark backfill at 2.5 GB/s aggregate throughput, cold data lives in S3 at $0.023/GB-mo, and the total monthly bill lands around $5.5k for a workload that previously ran at ~$15k on an EBS-only Kafka cluster.

Output:

Metric	Value
Primary produce p99 (steady state)	3–5 ms
Backfill fetch throughput (RRR)	2.5 GB/s
Cold-tier size in S3	~95 TB (30 days minus 24 h)
Local hot disk per broker	~900 GB (24 h × 300k msg/s × 1 KB / 6 brokers × 3)
Primary + RRR total $/mo	~$5,500
Kafka baseline for same workload	~$15,000
BYOC data-plane location	customer VPC (never leaves)

Why this works — concept by concept:

• Local hot + S3 cold split — recent segments on NVMe give producers durable low-latency writes; segments older than 24 h live in S3 at object-storage prices. The 3.5x per-GB cost delta compounds across 95% of the footprint.
• Async segment uploader — a background reactor task on each broker uploads sealed segments to S3, respecting Raft leadership. Producers never wait on S3; the write path is unchanged from a local-only deployment.
• Remote Read Replica — a read-only cluster sharing the same S3 bucket. Isolates the backfill workload from the write path; scales reads independently; local S3-cache absorbs repeat fetches within the Spark job's window.
• BYOC deployment — Redpanda operates the Kubernetes operator, monitoring, and upgrades from its control plane; the data plane runs in the customer's VPC. Data never leaves the customer's cloud account, satisfying compliance and sovereignty requirements without giving up managed convenience.
• Cost — one 6-node primary + one 3-node RRR + one S3 bucket + one control-plane subscription. Compared to an EBS-only Kafka on similar hardware, this is roughly one-third the cost at similar throughput and better tail latency. Storage cost scales with S3 price ($0.023/GB-mo) rather than EBS price ($0.08+/GB-mo); compute cost is broker-hours, unchanged from Kafka. Net O(retention × S3 price) rather than O(retention × EBS price).

Streaming
Topic — streaming
Streaming retention and tiered-storage problems

Practice →

Design Topic — design Design problems on streaming cost optimisation

Practice →

---

4. Iceberg Topics + streaming to lakehouse

iceberg topics collapse the streaming-to-lakehouse pipeline — the topic doubles as an Iceberg table so producers write once and Trino / Snowflake / DuckDB query the same data

The mental model in one line: an Iceberg Topic is a Redpanda topic where every produced record is written twice — once to the partitioned log (for Kafka-protocol consumers) and once to an Iceberg data file registered in a REST catalog like Polaris, AWS Glue, or Unity Catalog (for lakehouse engines) — collapsing the historical streaming-to-lakehouse pipeline (Debezium plus Kafka Connect plus Iceberg-sink plus schema-registry plumbing plus a Spark or Flink job that compacts small files) into a single broker-side feature that requires zero downstream services. Every senior data-engineering interview in 2026 asks about it because Iceberg Topics change the shape of a whole reference architecture.

The four axes for Iceberg Topics.

• What gets materialised. Each produced record is decoded per its schema-registry-registered schema (Avro, Protobuf, or JSON Schema) into a strongly-typed row, then appended to Iceberg data files partitioned along a configurable transform (identity, day, hour, bucket, truncate). The Iceberg table's snapshot is committed periodically (target_lag_ms, default 60 s).
• Where the metadata lives. Redpanda writes Iceberg metadata files (v1.metadata.json, manifest files, manifest lists) and commits pointer updates to a REST catalog — Polaris, AWS Glue, Databricks Unity Catalog, or any Iceberg-REST-compatible endpoint. The engine querying (Trino, Snowflake, DuckDB, Spark) sees a first-class Iceberg table.
• The consumer contract. Consumers keep consuming exactly as before; the Kafka protocol path is unchanged. The Iceberg write happens on the broker side and does not affect producer latency.
• The schema story. Iceberg Topics require the topic's records to be schema-registered. On schema evolution (add a nullable column), the broker propagates the change to Iceberg's schema evolution API; the Iceberg table's schema advances in lock-step.

Modes for Iceberg Topics — key_value vs value_schema_id_prefix.

• key_value. Store the raw Kafka key and value as two columns (key BINARY, value BINARY) plus metadata (partition, offset, timestamp, headers). Simplest mode; no schema-registry integration required. Downstream engines pay to parse the raw bytes.
• value_schema_id_prefix. The value bytes are prefixed with a schema-registry schema ID (Confluent's convention). Redpanda decodes each record per its schema and materialises typed columns in the Iceberg table. This is the default mode for structured data.
• Trade-off. key_value is universal but pushes work to consumers; value_schema_id_prefix requires schema-registry discipline but delivers a query-ready Iceberg table.

Catalog integrations.

• Apache Polaris. Open Iceberg REST catalog (Snowflake-donated, ASF-hosted); the natural pair with Redpanda in a fully-open lakehouse. Configure iceberg_catalog_type: rest + Polaris endpoint + OAuth2 credentials.
• AWS Glue. Native Glue catalog integration; use iceberg_catalog_type: aws_glue + the Glue region. IAM permissions on the Glue database and the S3 bucket are required.
• Databricks Unity Catalog. Via UC's Iceberg REST endpoint (/api/2.1/unity-catalog/iceberg); read-only from external writers, so pair with UC's own Iceberg mode.
• Snowflake managed Iceberg. Snowflake as the writer side for Iceberg tables; Redpanda can hand off metadata via Snowflake's external volume + REST catalog wiring.

Common interview probes on Iceberg Topics.

• "What does Iceberg Topics replace?" — required answer: Debezium + Kafka Connect + Iceberg-sink + compaction Spark job (a five-service pipeline).
• "How does schema evolution work?" — required answer: schema-registry backward-compatible change → Iceberg schema evolution API → new column visible in Trino/Snowflake on next commit.
• "What's the commit cadence?" — required answer: controlled by redpanda.iceberg.target_lag_ms (default 60 s); trade off freshness vs small-file count.
• "How do you partition the Iceberg table?" — required answer: redpanda.iceberg.partition_spec — identity, day, hour, bucket, truncate transforms compose in the standard Iceberg way.

Worked example — enabling Iceberg Topics on orders with Polaris

Detailed explanation. The canonical Iceberg Topics setup: register the Avro schema for orders.value in the schema registry, enable Iceberg mode on the topic with value_schema_id_prefix, wire the Polaris REST endpoint, and confirm the Iceberg table appears queryable from Trino. Walk through every piece.

• Prerequisites. Schema Registry running, Polaris running, Redpanda cluster with Tiered Storage bucket configured.
• Schema. Avro schema for orders.value with order_id, customer_id, total_cents, status, event_ts.
• Iceberg mode. value_schema_id_prefix, day-partitioned on event_ts, target lag 60 s.
• Query. Trino via Iceberg REST catalog connector.

Question. Configure the topic, produce a sample record, and query it from Trino as an Iceberg table.

Input.

Component	Value
Topic	orders (12 partitions, RF=3)
Schema format	Avro
Schema registry	http://schema-registry:8081
Iceberg mode	value_schema_id_prefix
Iceberg partition	day(event_ts)
Target lag	60 000 ms
Catalog	Polaris REST
Table location	s3://acme-lake/warehouse/streams/orders

Code.

# 1. Cluster-level Iceberg enable + catalog wiring (one-time)
rpk cluster config set iceberg_enabled true
rpk cluster config set iceberg_catalog_type rest
rpk cluster config set iceberg_rest_catalog_endpoint https://polaris.internal:8181
rpk cluster config set iceberg_rest_catalog_authentication_type oauth2
rpk cluster config set iceberg_rest_catalog_client_id "${POLARIS_CLIENT_ID}"
rpk cluster config set iceberg_rest_catalog_client_secret "${POLARIS_CLIENT_SECRET}"
rpk cluster config set iceberg_rest_catalog_prefix streams
rpk cluster config set iceberg_target_lag_ms 60000

# 2. Register the Avro schema for orders.value
cat > orders-value.avsc <<'EOF'
{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "acme.orders",
  "fields": [
    {"name": "order_id",    "type": "long"},
    {"name": "customer_id", "type": "long"},
    {"name": "total_cents", "type": "long"},
    {"name": "status",      "type": {"type": "enum", "name": "Status",
                                       "symbols": ["pending","shipped","cancelled"]}},
    {"name": "event_ts",    "type": {"type": "long", "logicalType": "timestamp-micros"}}
  ]
}
EOF

curl -X POST http://schema-registry:8081/subjects/orders-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "$(jq -Rs '{schema:.}' orders-value.avsc)"

# 3. Create the topic with Iceberg mode enabled + day partitioning on event_ts
rpk topic create orders \
  --partitions 12 --replicas 3 \
  --config redpanda.iceberg.mode=value_schema_id_prefix \
  --config redpanda.iceberg.target_lag_ms=60000 \
  --config redpanda.iceberg.partition_spec="(day(event_ts))" \
  --config redpanda.remote.write=true \
  --config retention.local.target.ms=86400000 \
  --config retention.ms=2592000000

# 4. Produce a sample record (Avro over Kafka with confluent-kafka + schema registry)
from confluent_kafka import Producer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer
from confluent_kafka.serialization import SerializationContext, MessageField

sr    = SchemaRegistryClient({"url": "http://schema-registry:8081"})
schema = open("orders-value.avsc").read()
serializer = AvroSerializer(sr, schema)

producer = Producer({"bootstrap.servers": "redpanda:9092"})

record = {
    "order_id":    42,
    "customer_id": 7,
    "total_cents": 1500,
    "status":      "pending",
    "event_ts":    int(1720051260 * 1_000_000),
}
value_bytes = serializer(record, SerializationContext("orders", MessageField.VALUE))
producer.produce(topic="orders", value=value_bytes, key=b"42")
producer.flush()

-- 5. Query the same data from Trino as an Iceberg table
-- (Trino iceberg catalog pointed at the same Polaris endpoint)
SELECT customer_id,
       COUNT(*)            AS order_count,
       SUM(total_cents)    AS total_cents
FROM   iceberg.streams.orders
WHERE  event_ts >= TIMESTAMP '2026-07-15 00:00:00'
GROUP  BY customer_id
ORDER  BY total_cents DESC
LIMIT  10;

Step-by-step explanation.

1. Cluster-level Iceberg enable is one-time: iceberg_enabled=true plus the catalog endpoint and credentials. Redpanda uses OAuth2 to authenticate to Polaris; the catalog namespace prefix (streams) scopes every topic's Iceberg table under iceberg.streams.<topic>.
2. Schema-registry integration is mandatory for value_schema_id_prefix mode. The Avro schema declares the row shape; Redpanda uses it to decode the wire format and to derive the Iceberg column types (long → LongType, string → StringType, timestamp-micros → TimestampType, enum → StringType).
3. Topic creation combines Iceberg config with Tiered Storage config. redpanda.iceberg.mode=value_schema_id_prefix opts the topic into materialisation; partition_spec="(day(event_ts))" uses Iceberg's day-transform for time partitioning; target_lag_ms=60000 controls commit frequency.
4. Producers are unchanged from a normal schema-registry Kafka setup. The confluent-kafka AvroSerializer prepends the schema ID and serialises the record; Redpanda decodes on receipt for the Iceberg write, and stores the raw bytes for the Kafka consumer path.
5. Trino sees the topic as iceberg.streams.orders — a native Iceberg table. Standard SQL works; the day-partitioning is visible in SHOW STATS and enables partition pruning on event_ts filters. Snowflake and DuckDB see the same table with the same catalog.

Output.

Data path	End-to-end behaviour
Producer	writes Avro to orders topic once
Consumer (real-time)	reads Avro from partitioned log at sub-5 ms latency
Iceberg writer (in broker)	decodes Avro, appends to Iceberg data files, commits every 60 s
Polaris catalog	records new snapshot; advances table pointer
Trino / Snowflake / DuckDB	queries iceberg.streams.orders as a native Iceberg table
Old Debezium + Connect pipeline	can be deleted

Rule of thumb. For any Redpanda topic that would otherwise need a Debezium-plus-Kafka-Connect-plus-Iceberg-sink pipeline, enable Iceberg Topics with value_schema_id_prefix mode. Match the partition_spec to the downstream query filter (usually day-partitioned on the event time). The 60 s default target lag is the right freshness for most analytics workloads.

Worked example — schema evolution on an Iceberg Topic

Detailed explanation. Schema evolution is the operational moment when Iceberg Topics prove their value. A backward-compatible schema change (add a nullable column) has to propagate through the schema registry, Redpanda's Iceberg writer, and the Iceberg table's schema at the catalog — all without downtime and without breaking existing Trino queries. Walk through the sequence.

• The change. Add priority INT (nullable) to the orders.value Avro schema.
• The propagation. Register the new schema version; producers start sending records with the new field; Redpanda detects the schema ID change on the next produce; it calls Iceberg's updateSchema().addColumn("priority", IntegerType.get()) and commits.
• The consumer story. Consumers using the old schema still deserialize (missing field defaults to null); consumers using the new schema get the new field.
• The Trino story. iceberg.streams.orders now has a priority column; existing queries ignore it; new queries reference it.

Question. Execute the schema evolution and confirm Trino sees the new column.

Input.

Component	Change
Avro schema	add priority INT (default null)
Schema registry compatibility	BACKWARD (Confluent default)
Producer code	write records with new field
Iceberg table schema	add priority INT NULL column
Trino query	select the new column

Code.

# 1. Register the new schema version (BACKWARD-compatible; adds nullable column)
cat > orders-value-v2.avsc <<'EOF'
{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "acme.orders",
  "fields": [
    {"name": "order_id",    "type": "long"},
    {"name": "customer_id", "type": "long"},
    {"name": "total_cents", "type": "long"},
    {"name": "status",      "type": {"type": "enum", "name": "Status",
                                       "symbols": ["pending","shipped","cancelled"]}},
    {"name": "event_ts",    "type": {"type": "long", "logicalType": "timestamp-micros"}},
    {"name": "priority",    "type": ["null", "int"], "default": null}
  ]
}
EOF

curl -X POST http://schema-registry:8081/subjects/orders-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "$(jq -Rs '{schema:.}' orders-value-v2.avsc)"
# → returns schema ID (e.g. 43)

# 2. Producer starts writing with the new field
record = {
    "order_id":    43,
    "customer_id": 7,
    "total_cents": 2500,
    "status":      "pending",
    "event_ts":    int(1720051500 * 1_000_000),
    "priority":    2,      # NEW field
}
value_bytes = serializer(record, SerializationContext("orders", MessageField.VALUE))
producer.produce(topic="orders", value=value_bytes, key=b"43")
producer.flush()

-- 3. On the next Iceberg commit, Redpanda calls updateSchema() on the table.
-- Verify via Trino
DESCRIBE iceberg.streams.orders;
-- order_id     BIGINT
-- customer_id  BIGINT
-- total_cents  BIGINT
-- status       VARCHAR
-- event_ts     TIMESTAMP(6)
-- priority     INTEGER          -- newly added

-- 4. Old rows have NULL priority; new rows have the value.
SELECT priority, COUNT(*) FROM iceberg.streams.orders GROUP BY priority;
-- NULL, 12_345      (rows produced before schema change)
-- 1,    50
-- 2,    120
-- 3,    30

Step-by-step explanation.

1. The schema registry enforces BACKWARD compatibility: the new schema must be readable by consumers using the old schema. Adding a nullable field is BACKWARD-compatible; adding a required field is not. The registry rejects non-BACKWARD changes by default; teams that want stricter contracts use FULL compatibility.
2. Producer code changes to include the new field; the AvroSerializer registers under a new schema ID (e.g. 43) and prepends it to every record. Consumers that still deserialize with schema ID 42 receive the record with the new field silently dropped (Avro schema resolution handles this).
3. Redpanda's Iceberg writer notices the schema ID change on the incoming record. Before committing the next Iceberg snapshot, it calls Table.updateSchema().addColumn("priority", IntegerType.get()).commit() on the catalog — Iceberg's schema evolution API is designed for exactly this case and is safe under concurrent reads.
4. Trino's next metadata refresh (usually every ~30 s or on demand via REFRESH MATERIALIZED VIEW / manual DDL cache invalidation) shows the new column. Existing queries continue to work; queries referencing priority see NULL for rows produced before the schema change and the actual value for rows after.
5. Old records in cold-tier Iceberg data files are not rewritten; Iceberg's column-mapping guarantees that reading them at the new schema version yields NULL for the missing column. No backfill job is needed.

Output.

Timeline	Schema version	Producer behaviour	Iceberg schema	Trino read
Before change	v1 (id 42)	writes without priority	5 columns	5 columns
Register v2	v1 + v2 (ids 42, 43)	old producers still on v1	5 columns	5 columns
Producer upgraded	v2 (id 43)	writes with priority	6 columns (after commit)	6 columns; NULL for old rows
Both consumers running	mix	irrelevant to consumers	6 columns	6 columns
Query on priority	—	—	6 columns	12k NULL + 200 non-NULL

Rule of thumb. For any Iceberg Topic in production, register schemas via a CI-checked Avro directory, enforce BACKWARD compatibility at the schema registry, and design schema changes as "add a nullable column, backfill the value in a downstream job, then optionally NOT NULL later." Never let a producer make a non-BACKWARD change without a coordinated consumer rollout.

Worked example — Iceberg Topics + DuckDB for ad-hoc analytics

Detailed explanation. DuckDB is the analytical database that took the analyst desktop by storm — a single binary, no server, and native Iceberg reads via the iceberg extension. Pairing DuckDB with Redpanda Iceberg Topics turns any laptop into an ad-hoc query engine against the live production stream, without touching the primary cluster or the warehouse. Walk through the setup.

• DuckDB. Latest release with the iceberg extension.
• Catalog. Same Polaris REST endpoint as Trino / Snowflake.
• Auth. OAuth2 client credentials scoped to read-only on the streams namespace.
• Query. SELECT against iceberg.streams.orders — from the analyst laptop.

Question. Configure DuckDB to read the Iceberg-Topics table and run an ad-hoc aggregation.

Input.

Component	Value
DuckDB version	1.1+
Iceberg extension	latest
Catalog endpoint	https://polaris.internal:8181
Namespace	streams
Table	orders
Auth mode	OAuth2 client credentials

Code.

-- DuckDB — install and load the Iceberg + AWS extensions
INSTALL iceberg;
INSTALL httpfs;
INSTALL aws;
LOAD iceberg;
LOAD httpfs;
LOAD aws;

-- Attach the Polaris REST catalog (Iceberg extension)
ATTACH 'iceberg' AS lake (
    TYPE ICEBERG,
    CATALOG_URI 'https://polaris.internal:8181',
    OAUTH2_CLIENT_ID  ?,
    OAUTH2_CLIENT_SECRET ?,
    WAREHOUSE 'streams',
    S3_REGION 'us-east-1'
);

-- Query the live topic-as-table
SELECT DATE_TRUNC('hour', event_ts)      AS hour,
       COUNT(*)                          AS orders,
       SUM(total_cents) / 100.0          AS revenue_usd
FROM   lake.streams.orders
WHERE  event_ts >= NOW() - INTERVAL '24 hours'
  AND  status = 'shipped'
GROUP  BY 1
ORDER  BY 1;

Step-by-step explanation.

1. DuckDB's iceberg extension (with httpfs and aws for S3 access) is a single INSTALL + LOAD. No server, no JVM, no separate catalog daemon on the analyst's laptop.
2. ATTACH binds a REST catalog to a DuckDB schema alias (lake here). The OAuth2 credentials are passed via prepared parameters (never hard-coded); the extension handles the token dance transparently.
3. Query syntax is standard SQL. DuckDB reads the Iceberg manifest to enumerate the data files, streams the Parquet files from S3 (via the vended credentials in the catalog response), and evaluates the query in memory. Day-partition pruning applies automatically — the event_ts >= NOW() - 24h filter reads only two days of data files.
4. The analyst gets sub-second responses to ad-hoc queries on live production data without ever touching the Redpanda primary or the warehouse. This is a workflow that used to require a warehouse account, network access, and a data-team ticket.
5. For teams with data-democratisation goals, this pattern collapses the "give the analyst live streaming data" problem from "provision Snowflake access + ETL job + row-level security" to "install DuckDB + share the catalog credentials." The security surface is the catalog RBAC + the vended S3 credentials, which is a well-understood contract.

Output.

Query	Time to first row	Notes
SELECT COUNT(*) FROM lake.streams.orders WHERE event_ts >= NOW() - 1h	~200 ms	metadata + partition prune
Hourly aggregate over 24 h	~800 ms	reads ~50 Parquet files from S3
Full-day scan over 30 d	~15 s	reads ~1500 Parquet files
Join with local CSV of customer mapping	~1 s	DuckDB reads local + S3

Rule of thumb. For any team with Redpanda + Iceberg Topics + a lakehouse catalog, offer DuckDB as the analyst-desktop query engine. It costs nothing, requires no server, and unlocks ad-hoc analytics against live streaming data. Warehouse queries remain the source of truth for governance; DuckDB is for exploration.

Senior interview question on Iceberg Topics

A senior interviewer might ask: "Design the streaming-to-lakehouse ingestion for a new fintech feature that produces 200k trade events per second to Kafka and needs the same data queryable in Trino within 60 seconds. The old design uses Debezium plus Kafka Connect plus an Iceberg-sink plus a compaction Spark job — you want to collapse it. Walk me through Iceberg Topics on Redpanda, the schema-registry setup, the catalog choice, the partition spec, and the failure semantics."

Solution Using Redpanda Iceberg Topics with Polaris + Avro schema-registry + day-partitioning + BACKWARD schema evolution

# 1. Cluster-level config — enable Iceberg + wire Polaris
cluster:
  iceberg_enabled: true
  iceberg_catalog_type: rest
  iceberg_rest_catalog_endpoint: https://polaris.internal:8181
  iceberg_rest_catalog_authentication_type: oauth2
  iceberg_rest_catalog_client_id: ${POLARIS_CLIENT_ID}
  iceberg_rest_catalog_client_secret: ${POLARIS_CLIENT_SECRET}
  iceberg_rest_catalog_prefix: streams
  iceberg_target_lag_ms: 60000

# 2. Register the Avro schema (BACKWARD-compatible baseline)
cat > trades-value.avsc <<'EOF'
{
  "type": "record",
  "name": "Trade",
  "namespace": "acme.trading",
  "fields": [
    {"name": "trade_id",     "type": "string"},
    {"name": "symbol",       "type": "string"},
    {"name": "side",         "type": {"type": "enum", "name": "Side", "symbols": ["BUY","SELL"]}},
    {"name": "price_cents",  "type": "long"},
    {"name": "qty",          "type": "long"},
    {"name": "venue",        "type": "string"},
    {"name": "event_ts",     "type": {"type": "long", "logicalType": "timestamp-micros"}}
  ]
}
EOF

curl -X POST http://schema-registry:8081/subjects/trades-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "$(jq -Rs '{schema:.}' trades-value.avsc)"

# 3. Create the topic with Iceberg mode + day/venue partitioning + tiered storage
rpk topic create trades \
  --partitions 48 --replicas 3 \
  --config redpanda.iceberg.mode=value_schema_id_prefix \
  --config redpanda.iceberg.target_lag_ms=60000 \
  --config redpanda.iceberg.partition_spec="(day(event_ts), identity(venue))" \
  --config redpanda.remote.write=true \
  --config redpanda.remote.read=true \
  --config retention.local.target.ms=86400000 \
  --config retention.ms=2592000000

# 4. Producers use standard Kafka + schema-registry libraries; no code change
#    (see the Iceberg Topics enable example above for the Python producer)

# 5. Trino sees iceberg.streams.trades as a native Iceberg table

-- 6. Trino query — the interview-scenario aggregation
SELECT venue,
       DATE_TRUNC('minute', event_ts)             AS minute,
       COUNT(*)                                   AS trades,
       SUM(qty)                                   AS shares,
       SUM(qty * price_cents) / 100.0             AS notional_usd
FROM   iceberg.streams.trades
WHERE  event_ts >= NOW() - INTERVAL '1' HOUR
  AND  side     =  'BUY'
GROUP  BY venue, DATE_TRUNC('minute', event_ts)
ORDER  BY minute, venue;

# 7. Failure semantics — what happens when the catalog is unreachable
# Redpanda buffers Iceberg commits locally; producer writes never block on Polaris.
# If Polaris is down > iceberg_target_lag_ms grace period, the broker emits:
#   redpanda_iceberg_commit_failed_total counter
# and continues buffering. Alert on this counter > 0 for 5 min.

# When Polaris recovers, the broker replays buffered commits in order.
# Iceberg's optimistic-lock protocol ensures no double-commit even if the reader
# assumed the previous commit succeeded.

Step-by-step trace.

Layer	Component	Purpose
Producer	200k msg/s Avro over Kafka wire	one write; Redpanda handles fan-out
Redpanda partitioned log	48 partitions across 6 brokers	Kafka consumer path (unchanged)
Redpanda Iceberg writer	decodes Avro → Parquet in Iceberg format	lakehouse path
Polaris catalog	records new snapshot every ~60 s	discovery + optimistic-lock CAS
S3 (tiered + Iceberg)	data files for both log tier and Iceberg	one storage location
Trino / Snowflake / DuckDB	queries iceberg.streams.trades	typed, partitioned, governed
Deleted infrastructure	Debezium + Connect + Iceberg-sink + Spark compactor	5 services → 0

After deployment, the fintech feature produces 200k trades/s to Redpanda; consumers read at sub-5 ms; the Iceberg table is queryable in Trino within ~65 s of the producer commit (60 s target lag + typical commit + Trino metadata refresh); the deleted Debezium + Connect + Iceberg-sink + compactor stack saves ~$4k/mo and one on-call runbook.

Output:

Metric	Value
Producer throughput	200k msg/s sustained
Consumer p99 latency	3–5 ms (unchanged)
Iceberg commit cadence	every 60 s
Trino freshness	~65 s from producer to queryable
S3 write amplification	1x (data files serve both tiers)
Deleted services	Debezium + Connect + Iceberg-sink + Spark compactor (4)
Ops savings	~$4k/mo + one on-call runbook removed

Why this works — concept by concept:

• Iceberg Topics as broker-side ETL — Redpanda decodes each schema-registered record on receipt and writes Iceberg data files directly into the Tiered Storage bucket. No separate ETL service; the broker is the pipeline.
• Schema-registry-driven typing — value_schema_id_prefix mode means the broker knows the exact schema of each record and can materialise typed columns. Downstream engines see a first-class Iceberg table, not a "raw bytes" table.
• Optimistic-lock CAS on catalog — every Iceberg commit is a compare-and-swap on the table's current-snapshot-id. Polaris arbitrates; two writers cannot silently overwrite each other. Redpanda handles retries transparently.
• Composed partition spec — (day(event_ts), identity(venue)) gives Trino two dimensions for partition pruning: a query filtered on event_ts >= NOW() - 1h AND venue='NASDAQ' reads only the intersecting partition. This is the difference between a 200 ms query and a 30 s query.
• Cost — one Redpanda cluster + one Polaris + one S3 bucket. Compared to the deleted Debezium + Kafka Connect + Iceberg-sink + Spark compactor stack, this is one service instead of four, at ~40% of the compute cost, with strictly better freshness (60 s vs 15 min for the sink + compact pattern). Net O(broker) per stream rather than O(4-services) per stream. The eliminated cost is the on-call overhead of maintaining a five-service ETL pipeline for every new topic.

Streaming
Topic — streaming
Streaming ETL and lakehouse-landing problems

Practice →

Design Topic — design Design problems on streaming-to-lakehouse convergence

Practice →

---

5. Redpanda vs Kafka vs WarpStream / AutoMQ + interview signals

redpanda vs kafka is the wrong framing — the real question is which of the four wire-compatible brokers wins your workload on latency, cost, ecosystem, and lakehouse-native landing

The mental model in one line: the 2026 Kafka-wire ecosystem has four production-grade brokers — Apache Kafka (JVM, reference implementation, huge ecosystem), Redpanda (C++ Seastar, sub-5 ms p99, Iceberg Topics), WarpStream (Go, S3-only, cost-optimised for latency-tolerant workloads), and AutoMQ (Kafka storage layer sharded onto EBS + S3) — and the senior architect's job is to walk the four-axis matrix (latency × cost × ecosystem × lakehouse-native) rather than default to whichever broker the vendor's rep pitched hardest last week. Every senior data-engineering interview probes this because "we should just use Kafka" is a 2020 answer and does not survive contact with a real workload budget.

The four axes for the four-broker comparison.

• p99 latency floor. Redpanda ~5 ms (no JVM, no GC), Kafka 20–50 ms (JVM + GC spikes), AutoMQ ~50 ms (Kafka-like), WarpStream ~500 ms (S3-only produce path). The gap is architectural, not tunable.
• Cold-storage TCO. Redpanda Tiered Storage → S3 at ~$0.023/GB-mo (mixed hot NVMe + cold S3); WarpStream → S3 only, no local disk, essentially free of compute-per-storage; Kafka KIP-405 Tiered Storage → S3 (implementation quality varies); AutoMQ → EBS + S3 tier. WarpStream wins pure storage cost; Redpanda wins latency-vs-cost balance.
• Ecosystem breadth. Apache Kafka is by far the largest — Kafka Connect (200+ connectors), Kafka Streams, ksqlDB, Confluent Schema Registry, MirrorMaker 2, dozens of enterprise support vendors. Redpanda is wire-compatible so most Connect connectors work; Streams/ksqlDB do not run natively. WarpStream and AutoMQ inherit whatever Kafka clients support.
• Lakehouse-native landing. Redpanda has Iceberg Topics (native, first-party, GA 2024). Kafka + Connect can do it with sink connectors + a Spark compactor. WarpStream and AutoMQ rely on Kafka Connect. Redpanda's collapse of the pipeline into one broker feature is unique in 2026.

Migration playbook — from Kafka to Redpanda.

• Step 1. Audit. Inventory topics, retention policies, producer / consumer clients, MirrorMaker or Cluster Linking dependencies, KStreams / ksqlDB pipelines. Anything that uses Streams or ksqlDB stays on Kafka (until you rewrite it as a Flink or standalone consumer).
• Step 2. Mirror. Deploy Redpanda alongside Kafka. Use rpk topic mirror, Kafka MirrorMaker 2, or Redpanda's Cluster Linking equivalent to shadow-write topics into Redpanda. Both clusters carry the same data.
• Step 3. Shadow read. Deploy canary consumers on the Redpanda side. Assert count / CRC parity with Kafka consumers over a 24–48 h window.
• Step 4. Cut over producers. One producer service at a time, flip bootstrap.servers to Redpanda. Consumers already read from both; producers write only to Redpanda now.
• Step 5. Decommission Kafka. After 30 days of Redpanda-only production, tear down Kafka, MirrorMaker, and any Connect-driven pipelines that Iceberg Topics replaced.

Common interview probes on the migration.

• "How do you migrate without downtime?" — required answer: wire-compatible + mirror + shadow read + producer cutover.
• "What breaks?" — required answer: KStreams and ksqlDB (rewrite as Flink or standalone consumers).
• "What's the rollback?" — required answer: flip bootstrap.servers back to Kafka; both clusters still hold the data during the mirror window.
• "How do you pick Redpanda vs WarpStream?" — required answer: p99 latency budget; anything < 100 ms rules WarpStream out.

Worked example — the four-broker decision matrix for a mixed workload

Detailed explanation. A single organisation typically has multiple streaming workloads with different SLOs. Rather than pick one broker for everything, the senior architect matches workload to broker. Walk through a concrete four-workload portfolio and pick the right broker for each.

• Workload A. Order matching — 30k msg/s, p99 budget 5 ms.
• Workload B. Fraud detection stream — 100k msg/s, p99 budget 50 ms, needs lakehouse landing.
• Workload C. Device telemetry firehose — 500k msg/s, p99 tolerant to 5 s, 90-day retention.
• Workload D. Application logs — 2M msg/s, p99 tolerant to 30 s, 7-day retention, cost-critical.

Question. For each workload, pick the broker and justify the pick against the four axes.

Input.

Workload	Throughput	p99 budget	Retention	Lakehouse landing
Order matching	30k msg/s	5 ms	7 d	no
Fraud stream	100k msg/s	50 ms	30 d	yes
Device telemetry	500k msg/s	5 s	90 d	yes
Application logs	2M msg/s	30 s	7 d	no

Code.

# broker_picker.py — pick the broker per workload
def pick_broker(msg_s: int, p99_budget_ms: int,
                retention_days: int, lakehouse: bool) -> str:
    if p99_budget_ms < 20:
        return "redpanda"           # only broker that hits sub-20 ms deterministically
    if p99_budget_ms >= 200 and retention_days >= 7:
        return "warpstream"         # S3-only wins pure cost when latency doesn't care
    if lakehouse:
        return "redpanda"           # Iceberg Topics = collapse the pipeline
    # default — Kafka for ecosystem breadth, or Redpanda if greenfield
    return "kafka (or redpanda greenfield)"


workloads = [
    ("order_matching", 30_000,   5,   7,  False),
    ("fraud_stream",   100_000,  50,  30, True),
    ("device_telemetry", 500_000, 5000, 90, True),
    ("app_logs",       2_000_000, 30_000, 7, False),
]

for name, msg_s, p99, ret, lake in workloads:
    print(name, "->", pick_broker(msg_s, p99, ret, lake))
# order_matching -> redpanda
# fraud_stream -> redpanda
# device_telemetry -> redpanda
# app_logs -> warpstream

Step-by-step explanation.

1. Workload A (order matching) has a 5 ms p99 budget — the sub-20 ms rule immediately picks Redpanda. No other broker deterministically hits that budget. Retention is short, no lakehouse requirement, so it's Redpanda-vanilla without extra features.
2. Workload B (fraud) has a 50 ms budget which Kafka can meet in steady state but not through GC pauses. Add the lakehouse-landing requirement and Redpanda + Iceberg Topics is the clear pick — no separate ETL pipeline.
3. Workload C (device telemetry) has a soft latency budget (5 s) and 90-day retention. Either Redpanda + Tiered Storage or WarpStream is defensible; the tiebreaker is the lakehouse requirement (yes → Redpanda for Iceberg Topics).
4. Workload D (app logs) has 2M msg/s throughput, 30 s latency tolerance, no lakehouse need. This is WarpStream's sweet spot: pure S3 storage at scale, no local disk, no replication overhead. Cost-per-message here is 3–5x lower than any other broker.
5. The four-workload portfolio splits three ways: Redpanda for the three latency-or-lakehouse workloads (A, B, C), WarpStream for the cost-only workload (D). Kafka does not win any workload outright in this portfolio — a common outcome in 2026 greenfield.

Output.

Workload	Broker	Why
Order matching	Redpanda	5 ms p99 requires no-JVM broker
Fraud stream	Redpanda	50 ms p99 + Iceberg Topics collapse
Device telemetry	Redpanda	Tiered Storage + Iceberg Topics + 5 s tolerable
Application logs	WarpStream	pure S3, cost-critical, 30 s latency fine

Rule of thumb. Do not pick a single broker for the whole organisation. Match each workload to the broker whose axis-scores dominate; run at most two brokers in production; document which workload lives where.

Worked example — the 5-minute broker-comparison interview answer

Detailed explanation. In a senior interview, "compare Redpanda, Kafka, WarpStream, and AutoMQ" is a canonical open-ended prompt. The candidate who scores highest walks the four axes deterministically and lands with a workload-based recommendation. Weak candidates list features in random order. Codify the 5-minute answer.

• Minute 1. Name the four brokers and the four axes.
• Minute 2. Latency axis with numbers.
• Minute 3. Cost axis with numbers.
• Minute 4. Ecosystem + lakehouse axis.
• Minute 5. Workload-based recommendation with the decision tree.

Question. Draft the 5-minute answer and rehearse it against the four-workload portfolio above.

Input.

Minute	Content	Senior signals
1	Name 4 brokers + 4 axes	frames the answer
2	p99 numbers	5 ms / 20-50 ms / 50 ms / 500 ms
3	Cost with $/GB-mo	EBS vs S3 with real numbers
4	Ecosystem + Iceberg Topics	KStreams caveat + native lakehouse
5	Workload-based tree	never single-broker for org-wide

Code.

Senior 4-broker comparison template (5 minutes)
===============================================

Minute 1 — name the four and the four axes
  "In 2026 the four production-grade Kafka-wire brokers are
   Apache Kafka, Redpanda, WarpStream, and AutoMQ. I evaluate
   them on four axes: p99 latency floor, cold-storage TCO,
   ecosystem breadth, and lakehouse-native landing."

Minute 2 — latency floor
  "Redpanda hits sub-5 ms p99 in steady state — Seastar, C++,
   io_uring, no JVM, no GC. Kafka lands at 20-50 ms with
   occasional 100-500 ms spikes on GC pauses. AutoMQ is
   Kafka-like at ~50 ms. WarpStream is architecturally ~500 ms
   because every produce goes straight to S3. For any workload
   with a real tail-latency SLO, this ordering is deterministic."

Minute 3 — cost
  "Cold-storage TCO is where the newer brokers win. Kafka
   EBS-only at $0.08/GB-mo on gp3 for 100 TB retention is
   ~$8k/mo just for storage. Redpanda Tiered moves 95% of that
   to S3 at $0.023/GB-mo — drops to ~$2.6k/mo. WarpStream stores
   everything in S3 by design; the storage bill is ~$2.3k/mo but
   there's no local disk at all. AutoMQ is EBS + S3 tiered like
   Redpanda but with a different implementation."

Minute 4 — ecosystem and lakehouse
  "Kafka has the largest ecosystem — Connect, Streams, ksqlDB,
   Schema Registry, MirrorMaker 2. Redpanda is wire-compatible
   so most Connect connectors work but Streams and ksqlDB do
   not run natively. Lakehouse landing: Redpanda has Iceberg
   Topics — every message also lands in an Iceberg table
   registered in Polaris/Glue/Unity, no separate connector.
   The other three rely on Kafka Connect + Iceberg-sink."

Minute 5 — workload-based recommendation
  "I don't pick one broker for the whole org. Latency-critical
   workloads (< 20 ms p99) go to Redpanda. Cost-only workloads
   with > 30 s latency tolerance go to WarpStream. Existing
   Confluent contracts stay on Kafka until renewal. Greenfield
   with lakehouse landing goes to Redpanda for the Iceberg
   Topics collapse. AutoMQ enters the picture mostly in APAC
   where its ecosystem is strongest."

Step-by-step explanation.

1. Minute 1 frames the answer as a decision, not a feature comparison. Naming all four brokers up front is a senior signal; naming the axes is what separates architects from users.
2. Minute 2 uses concrete p99 numbers rather than "faster" or "similar." The 5 ms / 20–50 ms / 50 ms / 500 ms ordering is memorable and defensible with benchmarks.
3. Minute 3 grounds the cost story in $/GB-mo with a concrete 100 TB example. The ~4x delta between EBS and S3 storage cost is the single most important number to have in your head.
4. Minute 4 correctly acknowledges Kafka's ecosystem lead — refusing to concede this is a red flag. Naming KStreams and ksqlDB as the Kafka-only pieces shows you've actually thought about migration constraints.
5. Minute 5 lands on the workload-based recommendation — "I don't pick one broker for the org." This is the senior insight: matching workload to broker beats forcing everything onto one platform. Interviewers who write "hire" after your answer are the ones who heard this sentence.

Output.

Grading criterion	Weak score	Senior score
Names all four brokers in minute 1	rare	mandatory
p99 numbers with GC-pause distinction	rare	required
Cost per GB-mo across storage tiers	rare	senior signal
Acknowledges Kafka ecosystem lead	occasional	required
Workload-based recommendation	rare	senior signal

Rule of thumb. The senior 4-broker answer walks the four axes with concrete numbers and lands on a workload-based recommendation. Rehearse it once; deploy it every time an interviewer opens with "what's your take on the Kafka landscape?"

Worked example — migration cutover with rpk topic mirror

Detailed explanation. The migration from Kafka to Redpanda is a wire-compatible cutover, but the safe pattern is a mirror + shadow-read + producer cutover sequence rather than a big-bang flip. Walk through the sequence for a single topic (orders) as a template.

• Step 1. Deploy Redpanda alongside Kafka in the same VPC.
• Step 2. Create orders on Redpanda with matching partition count and retention.
• Step 3. Start rpk topic mirror from Kafka to Redpanda; verify parity.
• Step 4. Deploy shadow consumer on Redpanda; alert on count / CRC mismatch.
• Step 5. Cut over one producer; monitor for 24 h; then cut over the rest.
• Step 6. After 30 days of Redpanda-only, decommission the Kafka topic.

Question. Script the mirror + shadow-read verification and the producer cutover.

Input.

Component	Value
Source topic	Kafka kafka.prod:9092/orders
Destination topic	Redpanda redpanda.prod:9092/orders
Partition count	12 (matched)
Retention	30 days (matched)
Shadow-read window	24 h
Producer count	4 services

Code.

# 1. Create the destination topic on Redpanda (matching Kafka's spec)
rpk topic create orders \
  --brokers redpanda.prod:9092 \
  --partitions 12 --replicas 3 \
  --config retention.ms=2592000000

# 2. Start the mirror (rpk topic mirror is the wire-compatible tool)
rpk topic mirror \
  --source-brokers kafka.prod:9092 \
  --source-topic orders \
  --dest-brokers redpanda.prod:9092 \
  --dest-topic orders \
  --shadow-mode &

# 3. Watch mirror lag
watch -n 5 'rpk cluster metadata --brokers redpanda.prod:9092 | grep orders'

# 4. Shadow-consumer parity check — read from both, assert equal CRCs
import binascii
from confluent_kafka import Consumer

def crc(v: bytes) -> str:
    return format(binascii.crc32(v) & 0xffffffff, "08x")

def run_parity_check(minutes: int = 60):
    kafka    = Consumer({"bootstrap.servers": "kafka.prod:9092",
                         "group.id": "parity-kafka",  "auto.offset.reset": "earliest"})
    redpanda = Consumer({"bootstrap.servers": "redpanda.prod:9092",
                         "group.id": "parity-rp",     "auto.offset.reset": "earliest"})
    kafka.subscribe(["orders"])
    redpanda.subscribe(["orders"])

    k_seen: dict[tuple[int,int], str] = {}
    r_seen: dict[tuple[int,int], str] = {}
    deadline = 60 * minutes
    for _ in range(deadline):
        for c, seen in [(kafka, k_seen), (redpanda, r_seen)]:
            msg = c.poll(1.0)
            if msg is None or msg.error():
                continue
            seen[(msg.partition(), msg.offset())] = crc(msg.value())

    missing_in_rp = set(k_seen) - set(r_seen)
    missing_in_k  = set(r_seen) - set(k_seen)
    crc_mismatch  = [k for k in set(k_seen) & set(r_seen) if k_seen[k] != r_seen[k]]

    print("kafka rows:  ", len(k_seen))
    print("redpanda rows:", len(r_seen))
    print("missing in rp:", len(missing_in_rp))
    print("missing in kf:", len(missing_in_k))
    print("crc mismatch: ", len(crc_mismatch))
    assert not missing_in_rp and not missing_in_k and not crc_mismatch, "parity failed"

run_parity_check(60)

# 5. Cut over one producer service (order-service-1) — flip bootstrap.servers
kubectl set env deployment/order-service-1 KAFKA_BOOTSTRAP=redpanda.prod:9092
kubectl rollout restart deployment/order-service-1
kubectl rollout status  deployment/order-service-1

# Watch for 24 h; if any regression, rollback:
# kubectl set env deployment/order-service-1 KAFKA_BOOTSTRAP=kafka.prod:9092

# 6. After 24 h clean run, cut over the rest
for svc in order-service-2 order-service-3 order-service-4; do
  kubectl set env deployment/$svc KAFKA_BOOTSTRAP=redpanda.prod:9092
  kubectl rollout restart deployment/$svc
done

# 7. After 30 days, decommission the Kafka topic
kafka-topics --bootstrap-server kafka.prod:9092 --delete --topic orders

Step-by-step explanation.

1. Creating the destination topic with matching partition count and retention is non-negotiable — divergent partition counts break key-ordering assumptions in downstream consumers. rpk topic create with explicit config is the canonical pattern.
2. rpk topic mirror --shadow-mode shadow-writes every message from Kafka into Redpanda with preserved partitions, offsets, and headers. The --shadow-mode flag guarantees the destination is a byte-for-byte copy. MirrorMaker 2 works as an alternative but adds JVM operational overhead.
3. The Python parity check is the correctness gate. It reads from both clusters, computes CRCs per (partition, offset), and asserts equal sets. Running it for 60 minutes over a healthy mirror confirms zero drift; running it for 24 h before the first cutover is the recommended pattern.
4. Producer cutover is one kubectl set env per service. Because both clusters have the same topic and consumers already read from both (with different consumer-group names), the flip is invisible to downstream. Rollback is the reverse set env.
5. The 30-day Redpanda-only run is the "soak time" before decommissioning Kafka. During this window, the mirror can be turned off; Kafka becomes a passive copy. After decommissioning, delete the Kafka topic, tear down MirrorMaker/mirror, and remove the Kafka cluster from monitoring.

Output.

Phase	Duration	State
Deploy Redpanda	day 0	both clusters running
Mirror + parity check	days 1–2	both clusters carrying same data
Cutover producer 1	day 3	25% traffic on Redpanda
Cutover producers 2–4	days 4–5	100% traffic on Redpanda
Soak	days 5–35	Redpanda authoritative; Kafka passive
Decommission Kafka	day 35	Redpanda only

Rule of thumb. Never big-bang a broker migration. Mirror + shadow-read + one-producer-at-a-time is the safe pattern; the wire-compatibility of Kafka clients is what makes the cutover a config change rather than a code change.

Senior interview question on broker migration

A senior interviewer might ask: "You inherit a Confluent Kafka deployment carrying 40 topics across 5 producer services. The contract renews in 6 months and the CFO is asking for a 40% cost cut. Walk me through the evaluation of Redpanda as a replacement, the migration playbook for one topic and then for the whole cluster, the failure modes you would pre-empt, and how you'd hedge against Redpanda-lock-in."

Solution Using a workload-audit + Redpanda POC + rpk topic mirror + phased cutover + open-Iceberg lock-in hedge

# 1. Audit spreadsheet (yaml here; real projects use a Google Sheet)
workloads:
  - topic: orders
    msg_s: 30000
    p99_budget_ms: 5
    lakehouse: no
    recommendation: redpanda

  - topic: fraud_events
    msg_s: 100000
    p99_budget_ms: 50
    lakehouse: yes
    recommendation: redpanda (iceberg topics)

  - topic: device_telemetry
    msg_s: 500000
    p99_budget_ms: 5000
    lakehouse: yes
    recommendation: redpanda (tiered + iceberg)

  - topic: app_logs
    msg_s: 2000000
    p99_budget_ms: 30000
    lakehouse: no
    recommendation: warpstream (cost-only)

  - topic: legacy_ksqldb_stream
    msg_s: 5000
    p99_budget_ms: 100
    lakehouse: no
    recommendation: kafka (keep; rewrite ksqlDB later)

# 2. Redpanda POC (6-node cluster in staging VPC, same instance shape as Kafka)
# See H2-1 solution for full deploy steps; here we assume it's up.

# 3. Phased mirror for the 39 topics going to Redpanda
for topic in $(cat topics-to-migrate.txt); do
  rpk topic create "$topic" --brokers redpanda.prod:9092 \
    --partitions "$(rpk-kafka topic describe "$topic" --brokers kafka.prod:9092 | jq .partitions)" \
    --replicas 3
  rpk topic mirror \
    --source-brokers kafka.prod:9092 --source-topic "$topic" \
    --dest-brokers   redpanda.prod:9092 --dest-topic   "$topic" \
    --shadow-mode &
done

# 4. Cutover producers per service, one at a time, with 24 h soak between
for svc in order-service fraud-service telemetry-service logs-service; do
  kubectl set env "deployment/$svc" KAFKA_BOOTSTRAP=redpanda.prod:9092
  kubectl rollout restart "deployment/$svc"
  sleep 86400        # 24 h soak before next service
done

# 5. Iceberg-Topics enablement for the two lakehouse-native workloads
for topic in fraud_events device_telemetry; do
  rpk topic alter-config "$topic" \
    --set redpanda.iceberg.mode=value_schema_id_prefix \
    --set redpanda.iceberg.target_lag_ms=60000 \
    --set redpanda.iceberg.partition_spec="(day(event_ts))"
done

# 6. Kafka stays for the one ksqlDB stream + as fallback for 30 days

# 7. Lock-in hedge — pick open catalog (Polaris) so if we ever move OFF Redpanda,
#    the Iceberg tables are still there in the same catalog.
# The Iceberg data files + Polaris registration are broker-independent.
# If we swap Redpanda for Kafka + Iceberg-sink, the tables keep the same identity.

def hedge_check(catalog: str, storage_bucket: str) -> str:
    if catalog.startswith("polaris") or catalog.startswith("glue") or catalog.startswith("unity"):
        return f"OK: open catalog '{catalog}' + shared bucket '{storage_bucket}' — no lock-in"
    return f"FAIL: proprietary catalog '{catalog}' locks Iceberg tables to broker"

print(hedge_check("polaris.internal:8181", "s3://acme-lake"))

Step-by-step trace.

Layer	Component	Purpose
Audit	40-topic spreadsheet	which broker per workload
Redpanda POC	6 brokers, staging VPC	benchmark against Kafka
Mirror	rpk topic mirror per topic	shadow-copy from Kafka
Parity check	count + CRC assertion	correctness gate
Cutover	producer-by-producer flip	wire-compatible; rollback = re-flip
Iceberg Topics	enable on 2 lakehouse topics	collapse ETL for those
Kafka retention	keep for 1 ksqlDB stream + 30-day soak	rollback + legacy
Lock-in hedge	Polaris + S3 for Iceberg	broker-independent
Cost outcome	~40% cheaper (Tiered + fewer services)	CFO target hit

After the migration, 39 of 40 topics run on Redpanda; two of those use Iceberg Topics and no longer need Debezium + Kafka Connect + Iceberg-sink; one topic stays on Kafka pending a ksqlDB rewrite. Total streaming spend drops 42%, meeting the CFO's target. The lock-in hedge (Polaris + open Iceberg tables in a shared S3 bucket) means that if Redpanda's pricing or roadmap changes unfavourably in 3 years, the lakehouse-side data stays intact.

Output:

Metric	Before	After
Broker count	6 Kafka (Confluent)	6 Redpanda + 1 Kafka legacy
Debezium + Connect + Iceberg-sink	4 services for 2 topics	0 (Iceberg Topics replaced them)
p99 produce latency	25 ms (with GC spikes)	3–5 ms
Monthly streaming spend	~$52k	~$30k (42% cut)
Confluent contract exposure	full renewal decision	negotiate reduced footprint
Lock-in surface	Confluent-specific features (KSQL, MM2)	open (Iceberg + Polaris)

Why this works — concept by concept:

• Wire-compatible migration — Kafka clients keep working with only a bootstrap.servers change. Producer + consumer code is byte-for-byte the same; the cutover is a kubectl set env per service. This is what turns a 12-month migration into a 6-week one.
• rpk topic mirror + shadow read — the safe cutover pattern. Both clusters carry the same data during the mirror window; parity checks catch drift before it matters; producer cutover is a config flip with an instant rollback.
• Workload-per-broker matching — one workload goes to WarpStream for the cost win, one stays on Kafka for ksqlDB, the rest go to Redpanda. Refusing to force everything onto one broker is the senior insight; it beats every single-broker recommendation.
• Iceberg Topics as pipeline collapse — for the two lakehouse-native workloads, enabling Iceberg Topics deletes an entire Debezium-plus-Connect-plus-Iceberg-sink stack. The cost savings and ops-simplicity wins compound with the broker migration.
• Cost — one Redpanda cluster + one WarpStream cluster + one legacy Kafka + Polaris + S3. Compared to the pre-migration all-Kafka + Confluent + Connect stack, this is ~40% cheaper on total spend, with better tail latency, better lakehouse landing, and a smaller ecosystem lock-in surface. The lock-in hedge (open Iceberg catalog + shared bucket) means the data-plane investment survives any future broker choice. Net O(workloads matched to brokers) rather than O(one-broker-for-everything).

Design
Topic — design
Design problems on streaming broker migration

Practice →

Streaming
Topic — streaming
Streaming multi-broker portfolio problems

Practice →

---

Cheat sheet — Redpanda recipes

• Which broker when. Redpanda is the 2026 default for any Kafka-wire workload where p99 latency matters, cold-storage TCO is a line item, or lakehouse landing needs to collapse a Debezium-plus-Connect-plus-Iceberg-sink pipeline. Apache Kafka stays when your KStreams / ksqlDB footprint is deep or an in-force Confluent contract makes migration uneconomic. WarpStream wins pure cost on latency-tolerant, retention-heavy workloads (logs, telemetry backups). AutoMQ enters the picture mostly in APAC deployments. Never pick a broker on architectural aesthetics alone; walk the four-axis matrix (latency, cost, ecosystem, lakehouse-native) with the actual workload constraints.
• Seastar shard-per-partition guide. Redpanda pins one shard per CPU core; each partition is deterministically assigned to a shard. Diagnose latency issues by graphing redpanda_reactor_utilization per shard — any shard over 80% sustained is a hot-shard problem. Fix (a) via rpk topic alter-partition-count to spread the key hash, or (b) via producer-side salting for genuinely skewed keys (key#salt with salt = md5(key) % 8). Never let a Seastar shard share a CPU core with an OS process; use --cpuset to reserve half the box for Redpanda and half for OS + monitoring.
• Capacity planning formula. Cores needed = (target_msg_s × replication_factor) / per_core_msg_s × (1 + headroom_pct). Distribute across brokers so each broker holds cores_per_broker = ceil(total_cores / brokers) shards. Choose an instance shape with vCPU = 2 × cores_per_broker so half the box is Redpanda and half is OS + hot-partition headroom. RAM = 4 GiB per shard minimum (1 GiB Seastar arena + 3 GiB for indices, caches, upload buffers).
• Tiered Storage config template. Cluster-level: cloud_storage_enabled: true, cloud_storage_bucket: <bucket>, cloud_storage_region: <region>, cloud_storage_credentials_source: aws_instance_metadata (IRSA on EKS; IMDSv2 on EC2). Per-topic: redpanda.remote.write=true, redpanda.remote.read=true, retention.local.target.ms=86400000 (24 h hot), retention.ms=<total>. Verify with rpk cluster storage summary (local vs cloud split) and aws s3 ls s3://<bucket>/<topic>/ (segments landing). Alert on redpanda_cloud_storage_upload_error_total > 0 for 5m.
• Iceberg Topics enablement checklist. (a) Register the Avro / Protobuf / JSON-Schema for the topic in Schema Registry with BACKWARD compatibility. (b) Enable cluster-wide iceberg_enabled=true + REST catalog endpoint + OAuth2 client credentials. (c) Create topic with redpanda.iceberg.mode=value_schema_id_prefix, redpanda.iceberg.target_lag_ms=60000, redpanda.iceberg.partition_spec="(day(event_ts))". (d) Verify Trino sees iceberg.streams.<topic> as a native table. (e) Alert on redpanda_iceberg_commit_failed_total > 0 for 5m (catalog outage). (f) Delete the old Debezium + Connect + Iceberg-sink pipeline for that topic.
• BYOC deployment topology. Redpanda Cloud control plane (Redpanda-operated Kubernetes operator, upgrades, monitoring) + customer VPC data plane (brokers on customer EC2, EBS, and S3 bucket). Networking: control plane reaches brokers over TLS with mTLS on the management API; producer / consumer traffic stays inside the customer VPC. IAM: brokers assume a customer IAM role for S3 access via IRSA (EKS) or instance profile (EC2). Data never leaves the customer's cloud account; the control plane sees only metrics + audit logs.
• Redpanda vs Kafka wire-compatibility caveats. Redpanda implements the Kafka wire protocol including transactions and exactly-once semantics, so confluent-kafka-python, franz-go, librdkafka, and the JVM Kafka client all work with a bootstrap.servers change. What does NOT work natively: Kafka Streams (JVM-only, deep KRaft assumptions), ksqlDB (Streams-based), some Confluent-proprietary features (Schema Linking, Cluster Linking specific flags). Kafka Connect connectors mostly work; test each. MirrorMaker 2 works as an ingress tool. When migrating, audit for KStreams / ksqlDB dependencies first.
• rpk topic mirror migration playbook. (a) Create matching destination topic (partitions + retention + config). (b) rpk topic mirror --source-brokers <kafka> --source-topic <t> --dest-brokers <rp> --dest-topic <t> --shadow-mode. (c) Deploy shadow consumer on destination; run count + CRC parity check for 24 h. (d) Cut over one producer at a time with kubectl set env KAFKA_BOOTSTRAP=<rp> + rolling restart; soak 24 h; then the rest. (e) 30-day Redpanda-only soak with Kafka mirror running as passive backup. (f) Decommission Kafka topic + mirror + MirrorMaker.
• TCO math template. Kafka $/mo = retention_TB × 1024 × ebs_price + brokers × broker_price_hr × 730 + egress + ops. Redpanda $/mo = hot_TB × 1024 × ebs_price + (retention_TB - hot_TB) × 1024 × s3_price + brokers × broker_price_hr × 730 + s3_ops + egress + ops. Redpanda wins big when retention_TB > 14 days and hot_TB < 20% of total (typical: 24 h hot, 30 d total → 3.3% hot). At retention < 7 days the savings compress to < 20% and the choice becomes tail-latency-driven rather than cost-driven.
• Iceberg Topics failure semantics. Producer path is unchanged; producers never block on catalog or Iceberg-write failures. Iceberg commits buffer locally in the broker; the redpanda_iceberg_commit_failed_total counter increments on failure. On catalog recovery, buffered commits replay in order via Iceberg's optimistic-lock CAS on snapshot_id. If the catalog is unreachable for hours, the buffer grows; alert on it. Consumers reading via the Kafka path are entirely unaffected by Iceberg-side failures; the two paths are decoupled after the receive queue.
• Redpanda Cloud vs self-managed decision. Redpanda Cloud (SaaS or BYOC) makes sense when the team is < 8 engineers, on-call for streaming is a small fraction of workload, or compliance requires a managed operator. Self-managed makes sense when streaming is > 50% of the platform team's remit, cost sensitivity is extreme, or the workload has bespoke requirements (custom io_uring tuning, unusual hardware, air-gapped networks). BYOC is the middle ground and is chosen by most fintech / regulated teams.
• Monitoring surface — the seven metrics to alert on. redpanda_reactor_utilization per shard > 0.85 for 5m (hot shard). redpanda_kafka_produce_latency_seconds p99 > 0.020 for 10m (produce SLO breach). redpanda_cloud_storage_upload_error_total > 0 for 5m (uploader stuck). redpanda_iceberg_commit_failed_total > 0 for 5m (catalog outage). redpanda_partition_under_replicated_replicas > 0 (Raft health). redpanda_disk_free_bytes < 20% (local disk pressure). redpanda_cloud_storage_cache_hit_ratio < 0.5 sustained (RRR cache undersized). These seven cover 95% of production incidents.
• What senior interviewers score highest. Naming Seastar + io_uring in sentence one (not just "C++"); using p99 numbers and GC-pause distinction rather than "faster"; distinguishing Redpanda vs WarpStream vs AutoMQ on the latency-vs-cost axis; naming Iceberg Topics as the collapse of the streaming-to-lakehouse pipeline rather than "another connector"; naming BYOC as the compliance answer; using rpk topic mirror + shadow read + producer cutover as the migration playbook; refusing to force one broker on the whole organisation. These are the senior signals that separate architects who have shipped Redpanda from candidates who have only read the docs.

Frequently asked questions

What is Redpanda in one sentence?

Redpanda is a Kafka-API-compatible streaming broker written from scratch in C++ on the Seastar asynchronous runtime, with a thread-per-core execution model, io_uring-based user-space I/O, no JVM and no ZooKeeper in the hot path, native Tiered Storage against S3/GCS/Azure Blob, and (since 2024) Iceberg Topics that materialise every produced message as a row in a governed Iceberg table registered in a Polaris / Glue / Unity catalog. It is the first credible replacement for Apache Kafka that keeps the wire protocol intact — every existing Kafka client library (Java, Go, Python, Rust, Node) works with only a bootstrap.servers change — while rewriting the broker below the wire to eliminate the JVM GC-pause tail-latency floor and to collapse the streaming-to-lakehouse pipeline into a single broker feature. Every senior data-engineering interview in 2026 asks about it because the choice of streaming broker is the single most consequential platform decision below the warehouse.

Redpanda vs Kafka — when do I pick each?

Default to Redpanda for any greenfield deployment where p99 latency is a hard budget (< 20 ms), where cold-storage TCO is a CFO-level line item, or where the streaming-to-lakehouse pipeline needs to collapse (Iceberg Topics replaces a Debezium + Kafka Connect + Iceberg-sink stack). Its Seastar core hits sub-5 ms p99 in steady state where Kafka lands at 20–50 ms with occasional 100–500 ms GC spikes; its Tiered Storage moves 80–95% of a retention-heavy footprint to S3 at $0.023/GB-mo rather than EBS at $0.08+/GB-mo; and Iceberg Topics turn the topic into a first-class Iceberg table registered in Polaris / Glue / Unity, queryable from Trino / Snowflake / DuckDB without a separate ETL. Stay on Apache Kafka when your KStreams / ksqlDB footprint is deep enough that rewriting to Flink is uneconomic, when an in-force Confluent contract makes mid-term migration painful, or when the ecosystem breadth (200+ Connect connectors, ksqlDB, Kafka Streams, MirrorMaker 2, dozens of enterprise support vendors) is a load-bearing part of the architecture. In 2026 the honest answer for most greenfield teams is "Redpanda for the workloads, keep Kafka running for the KStreams pipeline until you migrate it off."

What are Iceberg Topics and why do they matter?

An iceberg topics topic is one where Redpanda writes every produced record twice: once to the partitioned log (for Kafka-protocol consumers, exactly the same wire behaviour as any other topic) and once to Iceberg data files registered in a REST catalog like Polaris, AWS Glue, or Databricks Unity Catalog (for lakehouse engines). Producers write once; consumers keep consuming exactly as before; and Trino, Snowflake, DuckDB, and Spark can query the same data as a first-class Iceberg table with schema-registry-driven column types, partition pruning, and snapshot isolation — no Debezium, no Kafka Connect, no Iceberg-sink connector, no compaction Spark job. The historical five-service streaming-to-lakehouse pipeline collapses into one broker feature. Iceberg Topics are the single most differentiated 2026 Redpanda capability and are the reason a lot of teams that were "just fine with Kafka + Debezium" are re-evaluating: the ops story goes from "maintain five services per stream" to "enable one config on the topic." Freshness is controlled by redpanda.iceberg.target_lag_ms (default 60 s); schema evolution follows Iceberg's standard evolution API and is transparent to downstream engines.

How does Redpanda tiered storage compare to Kafka's?

Both broker families now support tiered storage, but the operational maturity gap is meaningful. Redpanda Tiered Storage is a first-party feature that ships identically across every deployment (self-hosted, Redpanda Cloud SaaS, BYOC) and has been production-hardened since 2021; enabling it is one cluster config plus per-topic redpanda.remote.write=true and a local-retention threshold. Apache Kafka's KIP-405 Tiered Storage is in the wire protocol but implementation quality varies by distribution: Confluent's implementation is the most mature, other distributions have varying levels of production-readiness, and the observability / rebalance story is less mature than Redpanda's. Both offload cold segments to S3/GCS/Azure Blob at object-storage prices, both retain recent segments on local NVMe for hot fetches, and both serve cold-tier fetches transparently to consumers. The operational win of Redpanda is that Tiered Storage is baked into the same broker binary rather than being a distro-specific add-on; the cost math ($0.023/GB-mo S3 vs $0.08/GB-mo EBS) is essentially the same either way. Pick Redpanda for the operational uniformity and Iceberg Topics bonus; pick Kafka if the ecosystem lock-in already exists.

Is Redpanda really Kafka-wire-compatible?

Yes for the vast majority of Kafka clients and workloads; with a few important caveats. Redpanda implements the Kafka wire protocol including produce, fetch, metadata, consumer groups, transactions, exactly-once semantics, ACLs, and Schema Registry compatibility, so any client built on librdkafka (confluent-kafka-python, confluent-kafka-go), the JVM Kafka client, franz-go, kafka-python, or node-rdkafka works with only a bootstrap.servers change — no producer or consumer rewrite. Kafka Connect connectors mostly work; most sink and source connectors are portable one-for-one, though some Confluent-proprietary connectors (Schema Linking, Cluster Linking-specific configurations) do not. What does not run natively on Redpanda in 2026: Kafka Streams (deeply tied to Kafka's internal APIs and KRaft assumptions) and ksqlDB (built on Streams). Teams with a heavy KStreams / ksqlDB footprint should plan to rewrite those pipelines as Flink jobs, standalone consumers, or Redpanda Data Transforms (Redpanda's WASM-based in-broker transform runtime) as part of the migration. The wire-compatibility story is what makes migration a config flip on the client side; the Streams / ksqlDB caveat is what makes migration a project on the pipeline side.

Is Redpanda production-ready in 2026?

Yes for all three deployment modes — self-hosted, Redpanda Cloud SaaS, and BYOC. Redpanda has been GA since 2020 and is running in production at Vodafone (5G telemetry), Alpaca (US equities trading), Lacework (security telemetry), and dozens of latency-sensitive fintech / adtech / trading firms since well before 2024. Redpanda Cloud (Snowflake's managed variant is snowflake open catalog; Redpanda's is Redpanda Cloud) has an SLA-backed uptime record and supports both fully-managed and BYOC topologies. The remaining rough edges in 2026 are largely around Kafka-ecosystem parity: Kafka Streams and ksqlDB do not run natively (rewrite as Flink or Redpanda Data Transforms), some Confluent-proprietary features have Redpanda equivalents rather than direct copies (Cluster Linking → Redpanda's own mirror), and the Connect connector matrix requires per-connector testing. Operational maturity is comparable to Kafka: monitoring surface via Prometheus, upgrade playbook via Helm or Terraform, and a mature rpk CLI. The interviewer's question here is almost always "is it mature enough to bet on" — the answer in 2026 is unambiguously yes, with the caveat that you must audit KStreams / ksqlDB dependencies and plan a per-connector Connect verification before committing to a full migration.

Practice on PipeCode

• Drill the SQL practice library → for the aggregation, window, and CDC-driven query problems that senior interviewers use to test Redpanda-fed lakehouse workloads.
• Sharpen the streaming axis with the streaming practice library → for Kafka-wire, tiered storage, Iceberg Topics, and broker-migration scenarios that mirror the Redpanda design interview.
• Rehearse system design against the design practice library → for streaming-platform-selection, latency-budget, and TCO-defence scenarios that decide the four-broker matrix.
• Warm up on the aggregation practice library → for the tumble / session / rolling-window shapes that dominate Iceberg-Topics dashboarding workloads.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-broker decision matrix, the Seastar shard model, and the Iceberg Topics pipeline collapse against real graded inputs.

Lock in redpanda muscle memory

Docs explain what Redpanda is. PipeCode drills explain when it wins — when Seastar thread-per-core makes the p99 story deterministic, when Tiered Storage flips the CFO's line item, when Iceberg Topics collapse a five-service ETL into one config, when the wire-compatible migration playbook makes a 12-month project into a 6-week rollout. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the streaming trade-offs senior data engineers actually face.

Practice streaming problems →
Practice design problems →