Kafka on Kubernetes with Strimzi vs Confluent Operator vs KRaft on Bare Metal

strimzi kafka is the pick-one architectural decision every senior streaming engineer faces the moment "run Kafka in production" hits the sprint board — and it is the single deployment choice that decides whether your on-call gets paged for controller quorum splits at 3 AM, whether your finance team writes six-figure Confluent invoices, or whether your p99 latency budget quietly evaporates into a Kubernetes scheduler that decided to reschedule a broker pod during a rolling upgrade. Every Kafka cluster your organisation runs — an order-events fan-in from a microservice mesh, a CDC firehose from Debezium into a warehouse, an ad-bidding pipeline that has to respond within 20 milliseconds — has to survive broker replacements, disk failures, network partitions, and controller re-elections without corrupting the log, without dropping messages, and without violating the ordering contract downstream consumers hard-code assumptions against. The engineering trade-off does not live in "should we adopt Kafka" — every event-driven stack needs it — but in which deployment model you pick (Strimzi OSS on Kubernetes, Confluent for Kubernetes with the commercial platform bundle, or a bare-metal KRaft quorum on EC2 / on-prem hardware) and what that model costs in operator surface, latency budget, RBAC complexity, and cash.

This guide is the senior-DE walkthrough you wished existed the first time an interviewer asked "walk me through the three ways to run Kafka in 2026 and their operational trade-offs," or "your team wants Strimzi but leadership was quoted by Confluent — how do you defend the choice?", or "explain KRaft mode and why anyone would still run Kafka on bare metal." It walks through the three canonical deployment patterns — strimzi kafka (the CNCF-graduated operator with Kafka / KafkaTopic / KafkaUser / KafkaConnect CRDs and Cruise Control auto-rebalancing), Confluent for Kubernetes (the commercial operator plus the Confluent Platform bundle with Schema Registry, ksqlDB, RBAC, Control Center, and automatic tiered storage), and bare-metal / EC2 KRaft (the ZooKeeper-free quorum controller pattern that ships direct NVMe access and the tightest p99 latency budget) — the "four axes" interviewers actually probe (control-plane operator vs bare metal, latency budget, RBAC + security surface, cost model), the canonical config for each, and the decision matrix that binds a Kafka deployment for years. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library → for Kafka broker, producer, consumer, and quorum-controller problems, rehearse system design against the design practice library → for operator vs bare-metal architecture scenarios, and sharpen the SQL axis with the SQL practice library → for the ksqlDB, KSQL-style, and stream-table join problems senior interviewers love.

---

On this page

• Why the Kafka deployment choice matters in 2026
• Strimzi OSS deep dive
• Confluent for Kubernetes (CFK)
• KRaft on bare metal / EC2
• Decision matrix + operational trade-offs + interview signals
• Cheat sheet — Kafka deployment recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why the Kafka deployment choice matters in 2026

Three deployment patterns, three wildly different operational cost curves — the choice binds you for years

The one-sentence invariant: running Kafka in 2026 is a picking exercise between the Strimzi CNCF-graduated OSS operator on Kubernetes, the Confluent for Kubernetes commercial operator that bundles Confluent Platform with RBAC and Control Center and tiered storage, and a bare-metal / EC2 KRaft quorum-controller deployment that trades operator convenience for direct-NVMe p99 latency — and each pattern trades operator surface against cost, latency budget, RBAC complexity, and the number of moving parts your on-call has to reason about at 3 AM in a way that cannot be undone downstream. The deployment you pick in month one becomes the deployment you fight to migrate away from in year three, because every downstream consumer, producer client, monitoring dashboard, and disaster-recovery runbook hard-codes assumptions about the shape of the cluster — whether broker replacement is kubectl delete pod or a systemd unit file, whether topics are declared in a YAML CRD or via kafka-topics.sh, whether authentication is a Kubernetes Secret or a keytab on a bare-metal host.

The four axes interviewers actually probe.

• Control-plane operator vs bare metal. Strimzi and CFK put a Kubernetes Operator between you and the cluster — declarative Kafka CRDs, automatic rolling upgrades, self-healing pods. KRaft on bare metal has systemd + Ansible + your on-call. Interviewers open with this question because it separates people who understand what an operator actually buys from those who repeat "Kubernetes is the default" without knowing why.
• Latency budget. Kafka on Kubernetes adds ~200–500 microseconds of p99 latency versus bare metal on the same hardware — the Kubernetes networking stack (kube-proxy iptables or IPVS or eBPF), the container runtime, the noisier neighbour on the node. For 99% of use cases this is invisible; for ad bidding, financial ticker feeds, and ultra-low-latency streaming, it is the whole ball game.
• RBAC + security surface. Strimzi ships mTLS + SCRAM out of the box via KafkaUser CRDs; CFK layers Confluent RBAC on top with LDAP/SAML integration and per-topic Kafka principals; bare-metal Kafka requires you to wire mTLS keystores, SCRAM credentials, and ACLs by hand via kafka-acls.sh. Each pattern's RBAC surface has a different failure mode and audit story.
• Cost model. Strimzi is free (Apache 2.0); you pay for the K8s cluster, the disks, and your engineering time. CFK is commercial (per-CPU or per-broker licensing) plus the K8s cluster and disks. Bare-metal Kafka is free software plus the hardware plus the deepest engineering time investment. Each model wins at a different scale.

The 2026 reality — Kubernetes is the greenfield default but bare metal is very much alive.

• Kafka on Kubernetes is the default answer for every greenfield deployment in 2026. Both Strimzi and CFK have shipped multi-year production track records at Netflix, Wise, ING, Adevinta, T-Mobile, and hundreds of other enterprises. The operator pattern (declarative Kafka CRD, controller reconciliation loop, rolling upgrades) has become the accepted way to run stateful workloads on Kubernetes.
• Strimzi (OSS) is the CNCF-graduated operator that dominates the open-source Kafka-on-K8s conversation. Adopted by Red Hat as the basis for Red Hat AMQ Streams / Streams for Apache Kafka. Every CRD, every operator behaviour, every dashboard is Apache-licensed and vendor-neutral.
• Confluent for Kubernetes (CFK) is the commercial operator that ships Confluent Platform — Kafka plus Schema Registry, ksqlDB, Kafka Connect Confluent Hub, RBAC, Control Center, tiered storage. It is the default when leadership has already bought Confluent or when you need enterprise features on day one.
• KRaft on bare metal / EC2 is the pattern for teams whose latency budget cannot afford Kubernetes' overhead, or whose deployment predates the K8s-Kafka operator era and works fine. KRaft (KIP-500, the Kafka Raft controller) removed the ZooKeeper dependency — GA in Kafka 3.5, the default in Kafka 4.0 (2025) — which finally made bare-metal Kafka a two-component stack (broker + controller) instead of a three-component one (broker + controller + ZooKeeper ensemble).

What interviewers listen for.

• Do you name all three deployment patterns without prompting and explain each one's headline trade-off? — senior signal.
• Do you say "KRaft removed ZooKeeper" and know it went GA in Kafka 3.5 and became default in 4.0? — required answer.
• Do you push back on "just use Confluent" with the cost-model question — "what is our CPU count and per-broker license cost?" — senior signal.
• Do you name Strimzi as the CNCF-graduated OSS operator and cite the Kafka / KafkaTopic / KafkaUser CRDs by name? — senior signal.
• Do you describe Kafka on Kubernetes as "an operator plus a StatefulSet plus PersistentVolumeClaims plus a bit of networking" rather than as vague "just deploy it"? — required answer.

Worked example — the three-way comparison table

Detailed explanation. The single most useful artifact for a Kafka deployment interview is a memorised 3×5 comparison table. Every senior Kafka-deployment discussion converges on this table within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical greenfield deployment that needs to serve microservice event streams, a CDC pipeline into Snowflake, and a real-time dashboard.

• Workload. Multi-tenant Kafka cluster: 20 producer services, 40 consumer services, ~2 TB/day ingest, ~500 MB/s sustained, ~4000 topic-partitions total.
• Infrastructure. AWS us-east-1, existing EKS cluster available. Team has strong Kubernetes skills, moderate Kafka operational skills.
• Requirements. 99.95% availability, p95 producer ack < 20 ms, 30-day retention, multi-AZ replication, at-least-once delivery.
• Constraints. OSS-first culture (finance skeptical of six-figure licenses), on-call rotation of 4 engineers, no dedicated Kafka SRE.

Question. Build the three-pattern comparison for the greenfield deployment and pick the pattern each axis drives you toward.

Input.

Pattern	Control plane	Cost model	p99 latency	RBAC surface	On-call surface
Strimzi OSS on K8s	Cluster Operator + CRDs	free (OSS) + K8s + disks	~5–10 ms	mTLS + SCRAM via KafkaUser	operator + StatefulSet
Confluent for Kubernetes	Confluent Operator + CRDs	per-CPU license + K8s + disks	~5–10 ms	Confluent RBAC + LDAP/SAML	operator + platform bundle
KRaft on bare metal / EC2	systemd + Ansible	free (OSS) + hardware	~1–3 ms	mTLS + SCRAM via kafka-acls.sh	systemd + hardware

Code.

# Strimzi Kafka CR — greenfield 3-broker KRaft cluster on K8s
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: prod
  namespace: kafka
spec:
  kafka:
    version: 3.9.0
    replicas: 3
    listeners:
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.9"
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 500Gi
          class: gp3-ssd
          deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}
---
# KRaft controllers (Strimzi uses KafkaNodePool CRs for controller / broker roles)
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  namespace: kafka
  labels:
    strimzi.io/cluster: prod
spec:
  replicas: 3
  roles:
    - controller
  storage:
    type: persistent-claim
    size: 10Gi
    class: gp3-ssd

Step-by-step explanation.

1. The comparison table forces the five axes into a single view: control plane, cost model, p99 latency, RBAC surface, and on-call surface. Strimzi wins on cost (Apache-2.0, no license), draws with CFK on latency (both live on K8s), and uniquely combines OSS licensing with the CNCF-graduated operator that AMQ Streams is built on top of. Every axis matters; interviewers who hear you name them in order get a senior signal.
2. Confluent for Kubernetes wins when leadership has already committed to Confluent (Control Center dashboards, ksqlDB, Confluent Hub connectors, enterprise support) or when tiered storage and RBAC-with-LDAP/SAML must ship on day one. Its per-CPU license model is either affordable (small clusters) or eye-watering (large clusters); the answer depends on scale.
3. Bare-metal KRaft wins on p99 latency (no K8s network stack, direct NVMe, no scheduler surprises) and is the answer for financial-tick, ad-bidding, and any workload where a 5 ms p99 is a business-critical number. It costs the most engineering time to run — you build the operator's convenience yourself in Ansible.
4. For the greenfield deployment described (OSS culture, EKS available, no dedicated Kafka SRE), the answer is Strimzi. The Cluster Operator absorbs 80% of what a dedicated SRE would do; the KRaft mode (in Kafka 3.9 / 4.0) eliminates the ZooKeeper ensemble; the KafkaNodePool CRD (Strimzi 0.36+) cleanly splits controller and broker roles.
5. If the same team had Confluent already deployed elsewhere or a strong existing relationship with Confluent, CFK would win — the answer is not "always Strimzi." Every axis matters, and the most-heavily-weighted axis depends on your organisation's history and constraints.

Output.

Requirement	Pattern winner	Reasoning
OSS-first culture	Strimzi	Apache 2.0; no license line item
Existing Confluent Platform	CFK	operator bundles the whole platform
p99 < 3 ms hard SLA	Bare-metal KRaft	K8s network stack cost is unavoidable
Small team, no dedicated Kafka SRE	Strimzi or CFK	operator does the ops work
Air-gapped / non-K8s environment	Bare-metal KRaft	no Kubernetes available

Rule of thumb. Never pick a Kafka deployment pattern based on "what's trendy." Pick it based on (control plane × cost model × latency budget × RBAC surface × on-call surface) — the five axes. Write the table on a whiteboard first; the pattern falls out of the constraints.

Worked example — what interviewers actually probe

Detailed explanation. The senior data-engineering Kafka-deployment interview has a predictable structure: the interviewer opens with an ambiguous question ("how would you run Kafka for our new event platform?"), then progressively narrows to test whether you know the axes. The candidates who name the pattern in sentence one score highest; the candidates who describe "we'd use Kubernetes" without saying which operator score lowest. Walk through the interview grading rubric.

• Ambiguous opener. "How would you deploy Kafka for our new order-events platform?" — invites you to name a deployment pattern.
• Follow-up 1. "Why Kubernetes and not bare metal?" — probes the latency-vs-operator-convenience axis.
• Follow-up 2. "Strimzi or Confluent — how do you pick?" — probes the cost-model + platform-features axis.
• Follow-up 3. "How does the operator handle a rolling upgrade?" — probes operator knowledge.
• Follow-up 4. "What is KRaft and did it change anything?" — probes 2025+ knowledge.

Question. Draft a 5-minute senior Kafka-deployment answer that covers all five axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Pattern named	"we'd use Kubernetes"	"Strimzi OSS on EKS with KRaft mode"
Latency awareness	"Kafka is fast"	"K8s adds ~200-500 μs p99 vs bare metal; fine for our SLA"
Cost	"it's open source"	"Strimzi is Apache 2.0; CFK is per-CPU licensed"
Operator behaviour	"we'd manage it"	"Cluster Operator reconciles Kafka CR; rolling upgrades via replicas.strategy=OnDelete"
KRaft	"no idea"	"KRaft removed ZooKeeper; GA in 3.5, default in 4.0; 3-5 controller quorum"

Code.

Senior Kafka-deployment answer template (5 minutes)
====================================================

Minute 1 — name the pattern up front
  "I'd default to Strimzi OSS on our existing EKS cluster, KRaft mode, with
   a 3-node controller quorum and 3-broker StatefulSet split by KafkaNodePool.
   If leadership already owns Confluent, CFK; if we needed sub-3ms p99, bare-metal KRaft."

Minute 2 — control plane / operator
  "The Strimzi Cluster Operator watches Kafka, KafkaTopic, KafkaUser, and
   KafkaNodePool CRDs. A Kafka CR change reconciles into StatefulSet
   changes; rolling upgrades honour min.insync.replicas so we never
   drop below quorum. Cruise Control auto-rebalances partitions when
   we add brokers."

Minute 3 — cost model
  "Strimzi is Apache 2.0 — no license line item; the cost is EKS + EBS +
   engineering time. Confluent for Kubernetes is per-CPU licensed; at
   our scale (18 vCPU) it's ~$X0k/year on top of infra. Bare metal is
   free software + EC2 instances + Ansible; cheapest hardware, most
   engineering time."

Minute 4 — latency + KRaft
  "Kafka on K8s adds ~200-500 μs p99 vs bare metal — the K8s network stack
   plus container-runtime overhead. Fine for our order-events (p95 < 20 ms
   ack budget). KRaft mode removed ZooKeeper — the quorum controller is
   3-5 dedicated nodes running the __metadata log__ and doing leader
   election via Raft. GA in Kafka 3.5, default in 4.0."

Minute 5 — RBAC + failure semantics
  "Strimzi ships mTLS + SCRAM via KafkaUser CRDs — the User Operator
   creates the keystore Secrets. Rolling upgrade: OnDelete strategy;
   min.insync.replicas guards producer acks; PodDisruptionBudget stops
   the scheduler from evicting more than one broker at a time. Broker
   loss recovery: the controller re-elects leaders within seconds; ISR
   shrinks and re-grows; Cruise Control rebalances if the loss is
   permanent."

Step-by-step explanation.

1. Minute 1 is the crucial framing. Naming the pattern immediately — "Strimzi OSS on EKS with KRaft mode" — signals you're a decision-maker who has run this in production, not a candidate who has only read docs. Weak candidates dive into tools ("we'd use Kubernetes and…") before naming the operator.
2. Minute 2 shows you understand what an operator actually does. Naming the four CRDs (Kafka, KafkaTopic, KafkaUser, KafkaNodePool) by name and describing the reconciliation loop separates you from candidates who describe operators as "controllers that do things."
3. Minute 3 addresses the cost axis before the interviewer asks. This preempts the common trap where you commit to Strimzi and then can't defend the choice against a leadership who's been sold Confluent. Naming a rough license number ("~$X0k/year at 18 vCPU") is senior signal.
4. Minute 4 covers latency (with a specific number, not "Kafka is fast") and KRaft (with version numbers). KRaft is 2025-current knowledge and every senior interview probes it because it changed the operational surface significantly.
5. Minute 5 covers RBAC + failure semantics — the reliability axis. The min.insync.replicas guard, PodDisruptionBudget, and the controller re-election story are the things that decide whether your on-call gets paged at 3 AM.

Output.

Grading criterion	Weak score	Senior score
Names pattern in minute 1	rare	mandatory
Names operator CRDs	rare	senior signal
Names cost with numbers	rare	senior signal
Names K8s latency cost	rare	senior signal
Names KRaft + version	occasional	mandatory

Rule of thumb. The senior Kafka-deployment answer is a 5-minute monologue that covers all five axes without waiting for the follow-ups. Rehearse it once; deploy it every time.

Worked example — the "pick the pattern" decision tree

Detailed explanation. Given a new Kafka requirement, the senior architect runs a 5-question decision tree in their head. Codifying the tree makes the interview answer reproducible: any interviewer can hand you a scenario and you can walk the tree out loud. Walk through the tree with three canonical scenarios: greenfield EKS deployment, legacy on-prem team, and an ultra-low-latency ad-bidding platform.

• Q1. Do you have Kubernetes available and a team that can operate it? → yes = go to Q2; no = bare-metal KRaft.
• Q2. Do you need sub-3-millisecond p99 producer-ack latency? → yes = bare-metal KRaft; no = go to Q3.
• Q3. Does leadership already own Confluent Platform, or do you need Control Center / ksqlDB / Confluent RBAC on day one? → yes = CFK; no = go to Q4.
• Q4. Is your organisation OSS-first (finance skeptical of per-CPU licenses)? → yes = Strimzi; no = CFK or Strimzi, evaluate on features.
• Q5 (parallel branch). Do you have an existing ZooKeeper-based cluster? → yes = plan the KRaft migration (KIP-866, ZK-to-KRaft upgrade path); no = start KRaft-native.

Question. Walk the decision tree for three scenarios and record the pattern each ends up with.

Input.

Scenario	Q1 (K8s?)	Q2 (< 3 ms p99?)	Q3 (Confluent owned?)	Q4 (OSS-first?)
Greenfield EKS event platform	yes	no	no	yes
Legacy on-prem no K8s	no	no	no	yes
Ad-bidding ultra-low-latency	yes	yes	no	yes

Code.

# Decision-tree helper (illustrative)
def pick_kafka_deployment(has_k8s: bool,
                           needs_sub_3ms_p99: bool,
                           owns_confluent: bool,
                           is_oss_first: bool) -> str:
    """Return the primary Kafka deployment pattern for a workload."""
    if not has_k8s:
        return "bare-metal KRaft"

    if needs_sub_3ms_p99:
        return "bare-metal KRaft (K8s overhead unaffordable)"

    if owns_confluent:
        return "Confluent for Kubernetes (CFK)"

    if is_oss_first:
        return "Strimzi OSS on Kubernetes"

    return "Strimzi or CFK (evaluate on features)"


# Walk the three scenarios
print(pick_kafka_deployment(True,  False, False, True))
# -> Strimzi OSS on Kubernetes

print(pick_kafka_deployment(False, False, False, True))
# -> bare-metal KRaft

print(pick_kafka_deployment(True,  True,  False, True))
# -> bare-metal KRaft (K8s overhead unaffordable)

Step-by-step explanation.

1. Scenario 1 — greenfield EKS event platform, no sub-millisecond latency demand, no existing Confluent contract, OSS-first culture. The decision tree short-circuits at Q4 → Strimzi OSS. This is the 2026 default for most greenfield Kafka-on-K8s deployments.
2. Scenario 2 — legacy on-prem team with no Kubernetes available. Q1 = no → bare-metal KRaft. This is the pattern for teams whose infrastructure predates the K8s era or whose compliance rules forbid running stateful workloads on a shared container platform.
3. Scenario 3 — ad-bidding platform where p99 producer ack must be under 3 ms. Q2 = yes → bare-metal KRaft, even though Kubernetes is available. The K8s network stack (kube-proxy, CNI, container runtime) adds hundreds of microseconds that the business SLA cannot absorb.
4. The parallel Q5 branch (existing ZooKeeper) is orthogonal to the primary deployment choice. Every ZK-based cluster in 2026 has a KRaft migration on its roadmap; the KIP-866 dual-write path lets you move from ZK to KRaft without downtime, and Kafka 4.0 (2025) removed ZK entirely.
5. If none of Q1-Q4 give a clear winner, the answer is Strimzi by default (OSS, CNCF-graduated, largest ecosystem) unless a specific requirement drags you toward CFK. Refuse to pick a pattern until you can articulate which axis drove the choice.

Output.

Scenario	Pattern	Reason
Greenfield EKS event platform	Strimzi OSS	OSS-first + K8s available + normal latency
Legacy on-prem, no K8s	bare-metal KRaft	no Kubernetes; systemd + Ansible
Ad-bidding ultra-low-latency	bare-metal KRaft	K8s network overhead unaffordable

Rule of thumb. The five-question decision tree is a whiteboard-friendly answer. Practice walking it end-to-end so an interviewer can hand you any scenario and get a pattern name in under 60 seconds.

Interview Question on Kafka deployment selection

A senior interviewer often opens with: "Your team is about to run Kafka in production for the first time. You have an existing EKS cluster, a moderate OSS-first culture, no existing Confluent contract, and a p95 producer-ack SLA of 20 milliseconds. Walk me through the deployment pattern you'd pick, the operator you'd deploy, the KRaft configuration you'd use, and the on-call runbook you'd write on day one."

Solution Using Strimzi OSS on EKS with KRaft mode and dedicated controller / broker KafkaNodePools

# 1. Strimzi Cluster Operator install (Helm)
# helm repo add strimzi https://strimzi.io/charts/
# helm install strimzi strimzi/strimzi-kafka-operator \
#   --namespace kafka --create-namespace \
#   --set watchAnyNamespace=false

# 2. KRaft controller pool — 3 dedicated controller nodes
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  namespace: kafka
  labels:
    strimzi.io/cluster: prod
spec:
  replicas: 3
  roles:
    - controller
  storage:
    type: persistent-claim
    size: 20Gi
    class: gp3-ssd
    deleteClaim: false
  resources:
    requests: { cpu: "1",   memory: "2Gi" }
    limits:   { cpu: "2",   memory: "4Gi" }
---
# 3. Broker pool — 3 dedicated broker nodes
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: broker
  namespace: kafka
  labels:
    strimzi.io/cluster: prod
spec:
  replicas: 3
  roles:
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 500Gi
        class: gp3-ssd
        deleteClaim: false
  resources:
    requests: { cpu: "4",   memory: "16Gi" }
    limits:   { cpu: "8",   memory: "32Gi" }

# 4. Kafka CR — cluster-level config
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: prod
  namespace: kafka
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.9.0
    metadataVersion: "3.9"
    listeners:
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication: { type: tls }
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        authentication: { type: tls }
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      auto.create.topics.enable: false
      log.retention.hours: 720   # 30 days
      log.segment.bytes: 1073741824
    authorization:
      type: simple
  cruiseControl: {}
  entityOperator:
    topicOperator: {}
    userOperator: {}

Step-by-step trace.

Layer	Choice	Reasoning
Operator	Strimzi (Helm-installed)	CNCF-graduated; Apache 2.0; widest ecosystem
Controller pool	3 nodes, role: controller	KRaft quorum (odd, 3-5); dedicated pool = no mixed workload
Broker pool	3 nodes, role: broker, JBOD 500 Gi	separates broker from controller lifecycle
Listeners	internal TLS + external LoadBalancer TLS	mTLS everywhere; external for cross-VPC producers
Replication	RF=3, min.insync.replicas=2	survive 1 broker loss without losing producer acks
Cruise Control	enabled	auto-rebalance on broker add / remove
Entity operators	TopicOperator + UserOperator	declarative topic/user creation via CRD
Storage	JBOD gp3-ssd	direct EBS, one volume per broker (JBOD lets Kafka manage log dirs)

After deployment, the Strimzi Cluster Operator reconciles the two KafkaNodePool CRs into two StatefulSets (controller + broker), the Kafka CR into cluster-wide config, and the EntityOperator into a sidecar deployment that watches KafkaTopic and KafkaUser CRs. Producers connect on port 9093 (internal) or 9094 (external LB) with mTLS. A KafkaUser YAML creates a Kubernetes Secret with the user's TLS keystore; the producer mounts it and connects. Rolling upgrades happen automatically on kafka.version changes; Cruise Control rebalances partitions when broker replica counts change.

Output:

Metric	Value
End-to-end producer p95 ack	~5-10 ms (well under 20 ms SLA)
Controller quorum size	3 (KRaft, odd, minimum)
Broker replication factor	3
Broker loss survivability	1 broker down = producer keeps ack'ing (min.isr=2)
Rolling upgrade behaviour	one broker at a time, min.isr enforced
Rebalance	automatic on scale via Cruise Control
License cost	$0 (Apache 2.0)
On-call surface	kubectl + Strimzi CRDs

Why this works — concept by concept:

• Strimzi Cluster Operator — the reconciliation loop that turns Kafka, KafkaNodePool, KafkaTopic, KafkaUser, and KafkaConnect CRDs into StatefulSets, Services, Secrets, and ConfigMaps. Every declarative change gets applied via a rolling reconcile that respects min.insync.replicas — you never write to Kubernetes primitives directly.
• KafkaNodePool split — separating controller and broker roles into two pools decouples their lifecycles. Upgrade controllers without touching brokers; scale brokers without disturbing the quorum. The KafkaNodePool CRD (Strimzi 0.36+) is the modern pattern; the old inline spec.kafka.replicas is deprecated for KRaft mode.
• KRaft mode — no ZooKeeper ensemble; the 3-node controller pool runs the Raft-based metadata log and elects a single active controller. Metadata operations (topic create, partition assignment) go through the quorum; broker heartbeats and leader elections propagate via the metadata log. GA in Kafka 3.5, default in 4.0.
• min.insync.replicas + acks=all — the producer contract that survives broker loss. With replication factor 3 and min.isr=2, a single broker loss keeps the ISR at 2 (the leader plus one replica); acks=all producers keep receiving acks. Losing two brokers stops acks — the deliberate safety property.
• Cost — one Strimzi operator install (~50 MB / 200m CPU), 3 controller pods (~2 GB each), 3 broker pods (~16 GB each), 6 EBS volumes (3 × 20 Gi controller + 3 × 500 Gi broker), 2 LoadBalancers. Compared to CFK: no license fee. Compared to bare metal: no hardware ownership; K8s does the pod-scheduling / self-healing work. Net O(1) operator + O(N) StatefulSet per cluster; adding a broker is a single KafkaNodePool.spec.replicas bump.

Streaming
Topic — streaming
Streaming Kafka deployment problems

Practice →

Design Topic — design Design problems on Kafka on Kubernetes

Practice →

---

2. Strimzi OSS deep dive

strimzi kafka is the CNCF-graduated operator that made Kafka-on-Kubernetes production-boring — CRDs, Cluster Operator, Cruise Control, and mTLS out of the box

The mental model in one line: strimzi kafka is the CNCF-graduated Kubernetes operator that turns "run a Kafka cluster" into a set of declarative custom resources — Kafka, KafkaNodePool, KafkaTopic, KafkaUser, KafkaConnect, KafkaBridge, KafkaMirrorMaker2 — reconciled by a Cluster Operator that owns rolling upgrades, Cruise Control auto-rebalancing, Prometheus + JMX + Grafana monitoring, and Strimzi CA certificate rotation, all under an Apache 2.0 license that costs zero and ships in every major cloud K8s distribution. Every senior Kafka-on-K8s deployment in 2026 either runs Strimzi directly or runs Red Hat AMQ Streams (which is Strimzi with commercial support), and the operator's declarative surface has become the reference for what a Kubernetes-native stateful workload should look like.

The four axes for Strimzi.

• Control plane. The Strimzi Cluster Operator is a Kubernetes controller that watches all Strimzi CRDs in one or more namespaces and reconciles them into StatefulSets, Services, ConfigMaps, and Secrets. One reconcile-loop tick per CR change; rolling upgrades happen automatically on version bumps.
• CRD surface. Seven headline CRDs: Kafka (cluster-wide config), KafkaNodePool (broker/controller role split, Strimzi 0.36+), KafkaTopic (topic declaration), KafkaUser (per-user credentials + ACLs), KafkaConnect (Kafka Connect cluster), KafkaBridge (HTTP-to-Kafka bridge), KafkaMirrorMaker2 (cross-cluster mirror). Every operational primitive is a YAML file.
• Cost. Apache 2.0. Zero license fee. Costs are Kubernetes cluster (EKS/GKE/AKS), persistent volumes, container image pulls, and engineering time.
• Ecosystem. CNCF-graduated (2024). Red Hat AMQ Streams / Streams for Apache Kafka is Strimzi with commercial support. Ships in every major cloud K8s distribution's marketplace.

The Cluster Operator — one deployment, watches all Strimzi CRs.

• What it is. A Java controller running as one or more replicas in the kafka namespace (or wherever you install it). Watches Strimzi CRDs; reconciles each into K8s primitives.
• How it deploys. Helm chart (strimzi/strimzi-kafka-operator) or OperatorHub bundle. watchAnyNamespace=true for cluster-wide; watchNamespaces=[kafka] for scoped.
• What it reconciles. Kafka → StatefulSet(s) + Services + ConfigMaps + Secrets + PodDisruptionBudget. KafkaTopic → topic creation via AdminClient. KafkaUser → Secret with keystore + ACL bindings. KafkaConnect → Deployment + Service.
• Failure semantics. Reconcile is idempotent; a failing tick retries with exponential backoff. Operator crash = no data loss, only reconcile-lag.

The Kafka CR — cluster-wide config in one YAML.

• spec.kafka.version. The Kafka version to run (e.g. 3.9.0). Changes trigger a rolling upgrade.
• spec.kafka.listeners. One or more listener definitions — internal / external, TLS / plaintext, SCRAM / mTLS authentication.
• spec.kafka.config. Free-form Kafka broker config (e.g. min.insync.replicas, log.retention.hours, default.replication.factor).
• spec.kafka.storage. Persistent storage type (jbod, persistent-claim, ephemeral). JBOD lets Kafka manage multiple log dirs on one broker.
• spec.kafka.authorization. ACL model (simple, keycloak, opa, custom).

The KafkaNodePool CR — the modern controller/broker split (Strimzi 0.36+).

• What it does. Splits the monolithic Kafka.spec.kafka.replicas into per-pool node definitions. Each pool has roles: [controller, broker] or just one role.
• Why it matters. Enables the KRaft controller/broker separation cleanly. Enables per-pool storage classes, per-pool resource requests, per-pool rolling upgrade order.
• Migration. Pre-0.36 clusters ran monolithic pods; upgrade path is the KRaft migration KIP-866 + strimzi.io/node-pools: enabled annotation.

Cruise Control — automatic partition rebalancing.

• What it does. A LinkedIn-open-sourced service that computes optimal partition assignments across brokers using goal-based optimisation (e.g. "balance replica count", "balance disk usage", "spread leader partitions evenly").
• How Strimzi ships it. spec.cruiseControl: {} in the Kafka CR — Strimzi deploys Cruise Control as a sidecar and wires it to the cluster.
• When it fires. On broker scale-out (new broker gets partitions assigned), scale-in (draining broker's partitions), disk-usage rebalance, or manual KafkaRebalance CR triggers.
• Why interviewers ask. Rebalancing is the operational headache Kafka is famous for; naming Cruise Control shows you know the modern automated answer.

Common interview probes on Strimzi.

• "How does Strimzi handle a rolling upgrade?" — required answer: reconcile-loop rolls pods one at a time honouring min.insync.replicas.
• "What's the difference between Kafka and KafkaNodePool CRs?" — required answer: cluster-level config vs per-pool node definitions.
• "How does Strimzi rotate certificates?" — required answer: Strimzi CA rotates cluster certificates automatically; renewal window is 30 days by default.
• "Why Cruise Control?" — required answer: goal-based automatic partition rebalancing on broker scale events.
• "How is Strimzi different from AMQ Streams?" — required answer: AMQ Streams is Strimzi + Red Hat commercial support and long-term support versions.

Worked example — declaring a topic and a user via Strimzi CRDs

Detailed explanation. The canonical Strimzi operational primitive: declare a topic and a user as YAML, kubectl apply -f, and the Entity Operator creates the topic via AdminClient and creates a Kubernetes Secret containing the user's TLS keystore. The application then mounts the Secret and connects. No kafka-topics.sh, no kafka-configs.sh, no imperative CLI at all.

• Topic. KafkaTopic CR with partitions, replicas, and topic-level config.
• User. KafkaUser CR with mTLS authentication and topic-scoped ACL grants.
• Secret propagation. User Operator creates Secret/prod-orders-svc containing user.crt, user.key, and a JKS bundle.

Question. Declare a topic orders.events and a user orders-svc that can produce to it, then show the producer deployment mounting the credentials.

Input.

CR	Purpose
KafkaTopic orders.events	24 partitions, RF=3, cleanup.policy=delete, retention 7 days
KafkaUser orders-svc	mTLS auth, ACL: write to orders.events
Secret orders-svc	auto-created by User Operator; TLS keystore
Deployment orders-svc-producer	mounts the Secret; connects on port 9093

Code.

# 1. Topic — declarative, reconciled by TopicOperator
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: orders.events
  namespace: kafka
  labels:
    strimzi.io/cluster: prod
spec:
  partitions: 24
  replicas: 3
  config:
    cleanup.policy: delete
    retention.ms: 604800000        # 7 days
    min.insync.replicas: 2
    segment.bytes: 1073741824
---
# 2. User — declarative, reconciled by UserOperator
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: orders-svc
  namespace: kafka
  labels:
    strimzi.io/cluster: prod
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: orders.events
          patternType: literal
        operations:
          - Describe
          - Write
        host: "*"

# 3. Producer deployment — mounts the user Secret
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-svc-producer
  namespace: kafka
spec:
  replicas: 3
  selector:
    matchLabels: { app: orders-svc-producer }
  template:
    metadata:
      labels: { app: orders-svc-producer }
    spec:
      containers:
        - name: producer
          image: acme/orders-svc:2.6
          env:
            - name: KAFKA_BOOTSTRAP
              value: "prod-kafka-bootstrap:9093"
            - name: KAFKA_SSL_TRUSTSTORE_LOCATION
              value: /etc/kafka/certs/ca/ca.p12
            - name: KAFKA_SSL_TRUSTSTORE_PASSWORD_FILE
              value: /etc/kafka/certs/ca/ca.password
            - name: KAFKA_SSL_KEYSTORE_LOCATION
              value: /etc/kafka/certs/user/user.p12
            - name: KAFKA_SSL_KEYSTORE_PASSWORD_FILE
              value: /etc/kafka/certs/user/user.password
          volumeMounts:
            - name: cluster-ca
              mountPath: /etc/kafka/certs/ca
              readOnly: true
            - name: user-cert
              mountPath: /etc/kafka/certs/user
              readOnly: true
      volumes:
        - name: cluster-ca
          secret:
            secretName: prod-cluster-ca-cert       # created by ClusterOperator
        - name: user-cert
          secret:
            secretName: orders-svc                 # created by UserOperator

Step-by-step explanation.

1. The KafkaTopic CR is reconciled by the Topic Operator (part of the Entity Operator deployment). It calls Kafka's AdminClient.createTopics with the specified partition count, replication factor, and config, then keeps the topic config in sync — any drift (someone edits config via kafka-configs.sh) is reconciled back to the CR spec on the next tick.
2. The KafkaUser CR is reconciled by the User Operator. For authentication.type: tls, the operator generates a per-user certificate signed by the Strimzi CA and stores it in a Kubernetes Secret named after the KafkaUser (Secret/orders-svc). For authorization.type: simple with acls: [...], the operator creates matching Kafka ACLs via AdminClient.createAcls.
3. The two Secrets — prod-cluster-ca-cert (created by the Cluster Operator on bootstrap) and orders-svc (created by the User Operator) — carry everything the producer needs: the truststore (containing the Strimzi CA cert) and the keystore (containing the user's cert + key). No manual certificate handling.
4. The producer Deployment mounts both Secrets as read-only volumes at fixed paths and reads the passwords from .password files (Strimzi stores keystore passwords in the same Secret). The producer connects to prod-kafka-bootstrap:9093 — the Service the Cluster Operator created for the TLS listener.
5. The end result: adding a new microservice that talks to Kafka is a two-CR YAML change (KafkaTopic + KafkaUser) plus a Deployment that mounts the auto-created Secret. No kafka-topics.sh, no keytool, no OpenSSL, no manual ACL grants.

Output.

Object	Created by	Purpose
KafkaTopic/orders.events	you (YAML)	declares the topic
topic orders.events on cluster	Topic Operator	actual Kafka topic
KafkaUser/orders-svc	you (YAML)	declares the user
Secret/orders-svc	User Operator	TLS keystore for the user
Kafka ACL	User Operator	Write on topic:orders.events
Secret/prod-cluster-ca-cert	Cluster Operator	CA truststore
Deployment/orders-svc-producer	you	mounts both Secrets; connects

Rule of thumb. For any Strimzi deployment, treat topics and users as YAML CRs the same way you treat Deployments and Services — put them in Git, apply via GitOps (Argo CD / Flux), and never touch kafka-topics.sh in production. The declarative model is the whole point.

Worked example — rolling upgrade with min.insync.replicas safety

Detailed explanation. The rolling upgrade is the Strimzi operational moment interviewers probe most. Change spec.kafka.version: 3.8.0 to 3.9.0, kubectl apply, and the Cluster Operator recycles broker pods one at a time — but only if the ISR would not drop below min.insync.replicas during the roll. Walk through what happens under the hood.

• The trigger. spec.kafka.version change in the Kafka CR.
• The safety. Before deleting a broker pod, the operator checks that the topics on that broker have ISR ≥ min.insync.replicas + 1 — otherwise producer acks would fail during the pod restart.
• The order. Brokers roll one at a time; controller pool rolls after brokers (or before, depending on strimzi.io/upgrade.controller-first annotation).

Question. Design a rolling upgrade with the safety guard, and quantify the expected roll duration for a 3-broker cluster.

Input.

Component	Value
Cluster	3 brokers, 3 controllers, KRaft mode
Kafka version change	3.8.0 → 3.9.0
min.insync.replicas	2 (per topic)
Pod restart time	~30-60 seconds (leader re-election + log recovery)
Total roll duration	3 brokers × ~60 s + 3 controllers × ~30 s = ~5 minutes

Code.

# 1. Pre-flight — verify ISR headroom for every topic
kubectl exec -n kafka prod-broker-0 -- \
  bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --under-min-isr-partitions
# Expected: empty output (no partitions currently below min.isr)

# 2. Verify PDB in place
kubectl get poddisruptionbudget -n kafka prod-kafka
# NAME         MIN AVAILABLE   ALLOWED DISRUPTIONS   AGE
# prod-kafka   2               1                     42d

# 3. Change the version
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: prod
  namespace: kafka
spec:
  kafka:
    version: 3.9.0                # was 3.8.0
    metadataVersion: "3.9"        # bump metadata version after roll completes
    # ... rest of spec unchanged

# 4. Apply and watch the roll
kubectl apply -f prod-kafka.yaml
kubectl -n kafka get pods -w -l strimzi.io/cluster=prod

# Expected sequence:
#   prod-broker-2 Terminating     (last broker rolls first)
#   prod-broker-2 Pending
#   prod-broker-2 Running         (ISR back to 3)
#   prod-broker-1 Terminating     (only after broker-2 rejoined ISR)
#   prod-broker-1 Pending
#   prod-broker-1 Running
#   prod-broker-0 Terminating
#   prod-broker-0 Pending
#   prod-broker-0 Running
#   prod-controller-2 Terminating
#   ...

# 5. Verify no partitions dropped below min.isr during the roll
kubectl exec -n kafka prod-broker-0 -- \
  bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --under-min-isr-partitions
# Expected: still empty

Step-by-step explanation.

1. Step 1 is the pre-flight sanity check: are any partitions already under min.isr? If yes, do not roll — the roll will make things worse. The --under-min-isr-partitions flag is the standard way to check.
2. Step 2 verifies the PodDisruptionBudget. Strimzi creates a PDB per cluster with minAvailable = replicas - 1 — the K8s scheduler is prevented from evicting more than one broker at a time even during node drain events. This is the belt-and-braces defence against the scheduler bypassing the operator.
3. Step 3 applies the version change. The Cluster Operator sees the spec.kafka.version diff and starts the reconcile: for each broker pod (in reverse ordinal order for StatefulSets), check ISR safety, delete the pod, wait for it to rejoin ISR, move to the next.
4. Step 4 shows the expected sequence. Each broker restart takes ~30-60 s: pod termination (Kafka clean shutdown flushes logs), image pull if new, container start, JVM warmup, leader re-election, log recovery. During the ~60 s window, the topic ISR drops from 3 to 2 — still ≥ min.isr=2 — so producers keep receiving acks.
5. Step 5 is the post-flight verification. --under-min-isr-partitions still empty means the roll respected min.isr throughout. If any partition dropped below, that's a bug or an operator crash mid-roll; investigate immediately.

Output.

Timeline	Cluster state	Producer impact
t=0	3 brokers 3.8.0	acks all working
t=0-60s	broker-2 rolling to 3.9.0	ISR=2 for its partitions; acks continue
t=60-120s	broker-1 rolling	ISR=2; acks continue
t=120-180s	broker-0 rolling	ISR=2; acks continue
t=180-270s	3 controllers rolling one at a time	metadata frozen briefly per controller swap
t=270s	cluster fully on 3.9.0	acks continue; no producer errors

Rule of thumb. For every Strimzi rolling upgrade, verify --under-min-isr-partitions is empty before starting, ensure the PDB minAvailable = replicas - 1, and expect ~60 s per broker for the roll. If your producers cannot tolerate ISR shrinking to 2 briefly, your min.isr is wrong for your risk model.

Worked example — Cruise Control automatic rebalancing on broker scale-out

Detailed explanation. Add a fourth broker to a 3-broker cluster and Kafka does not automatically move partitions to it — the new broker sits idle, taking no traffic, until someone runs kafka-reassign-partitions.sh. Strimzi's built-in Cruise Control fixes this: after the new broker joins, submit a KafkaRebalance CR and Cruise Control computes an optimal reassignment and executes it live without dropping messages.

• The trigger. New broker joins (via KafkaNodePool.spec.replicas: 4).
• The plan. Cruise Control computes an assignment that satisfies its goals (replica balance, leader balance, disk-usage balance).
• The execute. The plan is applied via Kafka's ReassignPartitions API; Cruise Control throttles the movement to avoid saturating network / disk.

Question. Scale from 3 to 4 brokers and use Cruise Control to rebalance partitions onto the new broker.

Input.

Component	Value
Starting brokers	3
Target brokers	4
Goal	balanced replica count and leader count across all 4 brokers
Throttle	10 MB/s per broker (Cruise Control default)

Code.

# 1. Scale the broker pool
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: broker
  namespace: kafka
  labels: { strimzi.io/cluster: prod }
spec:
  replicas: 4          # was 3
  # ... rest unchanged

# 2. Ask Cruise Control to rebalance
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: prod-add-broker-3
  namespace: kafka
  labels: { strimzi.io/cluster: prod }
  annotations:
    strimzi.io/rebalance: "add-brokers"
spec:
  mode: add-brokers
  brokers: [3]                # new broker id
  goals:
    - RackAwareGoal
    - MinTopicLeadersPerBrokerGoal
    - ReplicaCapacityGoal
    - DiskCapacityGoal
    - NetworkInboundCapacityGoal
    - NetworkOutboundCapacityGoal
    - CpuCapacityGoal
    - ReplicaDistributionGoal
    - PotentialNwOutGoal
    - LeaderReplicaDistributionGoal
    - LeaderBytesInDistributionGoal
    - TopicReplicaDistributionGoal

# 3. Watch the rebalance progress
kubectl get kafkarebalance -n kafka prod-add-broker-3 -w
# NAME               CLUSTER   PENDINGPROPOSAL   PROPOSALREADY   REBALANCING   READY   STOPPED
# prod-add-broker-3  prod      True                                                     
# prod-add-broker-3  prod                        True                                   
# (approve the proposal)
kubectl annotate kafkarebalance -n kafka prod-add-broker-3 \
  strimzi.io/rebalance=approve

# prod-add-broker-3  prod                                        True                   
# prod-add-broker-3  prod                                                       True   

# 4. Verify balanced state
kubectl exec -n kafka prod-broker-0 -- \
  bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe \
  | grep -E "Replicas: (0,1,2,3|1,2,3,0|2,3,0,1|3,0,1,2)"
# Expected: partitions now spread across all 4 brokers

Step-by-step explanation.

1. Scaling the KafkaNodePool from 3 to 4 replicas causes the Cluster Operator to add a fourth StatefulSet pod (prod-broker-3), which starts up, registers with the controller quorum, and joins the cluster. But no partitions are assigned to it yet — it sits idle.
2. Submitting the KafkaRebalance CR with mode: add-brokers and brokers: [3] tells Cruise Control: "compute a plan that moves some partitions onto broker 3 to balance load." Cruise Control's goals list (ReplicaDistributionGoal, LeaderReplicaDistributionGoal, etc.) governs the optimisation.
3. Cruise Control produces a proposal — a JSON blob describing which partitions move from which brokers to broker 3. The CR transitions to PROPOSALREADY; a human (or CI job) reviews the proposal and approves via the strimzi.io/rebalance=approve annotation.
4. Once approved, Cruise Control executes the plan: for each partition move, it adds broker 3 to the replica set, waits for the follower to catch up (log bytes copied over the network at the throttle limit), then removes the departing replica. Throttling prevents saturating the network / disk during large moves.
5. The final state has partitions balanced across all 4 brokers. The topic descriptions show replica sets like [0,1,3], [1,2,3], [0,2,3] — every partition has broker 3 as one of its replicas, and no broker is over-loaded. Cruise Control's goal-based approach is deterministic and reproducible.

Output.

Stage	KafkaRebalance state	What happened
Submitted	PendingProposal	Cruise Control computing plan
Plan ready	ProposalReady	JSON diff shown; awaiting approval
Approved	Rebalancing	Kafka reassigns partitions with throttle
Complete	Ready	new broker fully utilised; goals satisfied

Rule of thumb. Never manually run kafka-reassign-partitions.sh in a Strimzi cluster — use KafkaRebalance CRs and let Cruise Control compute the plan. Human-crafted reassignments are error-prone; goal-based automatic plans are reproducible and auditable via the CR history.

Interview Question on Strimzi

A senior interviewer might ask: "You're running a 6-broker Strimzi cluster on EKS with KRaft mode and Cruise Control enabled. A broker dies unrecoverably (EBS volume corrupted, PVC lost). Walk me through the recovery: how does Strimzi detect it, what does Kubernetes do, how does the cluster catch back up, and what does Cruise Control do afterwards. Include the CR-level operations you'd run and the safety guards you'd verify."

Solution Using StatefulSet self-heal + KafkaRebalance recovery + Cruise Control full-rebalance

# 1. Detect — the pod is CrashLoopBackOff; PVC is unbindable
kubectl -n kafka get pods -l strimzi.io/cluster=prod
# NAME              READY   STATUS             RESTARTS   AGE
# prod-broker-0     1/1     Running            0          42d
# prod-broker-1     1/1     Running            0          42d
# prod-broker-2     0/1     CrashLoopBackOff   17         42d   <-- dead
# prod-broker-3     1/1     Running            0          42d
# prod-broker-4     1/1     Running            0          42d
# prod-broker-5     1/1     Running            0          42d

kubectl -n kafka describe pod prod-broker-2
# Warning  FailedMount  ... unable to mount volumes for pod "prod-broker-2":
#          rpc error: code = FailedPrecondition ... volume "pvc-...-broker-2" not found

# 2. Confirm data-loss impact
kubectl exec -n kafka prod-broker-0 -- \
  bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe \
  --under-min-isr-partitions
# (Expect a list of partitions whose ISR dropped to 2 because broker-2 was one replica)

# 3. Recovery — remove the dead broker, let StatefulSet re-create with fresh PVC
# a) Confirm broker-2 is unrecoverable (EBS truly gone)
# b) Delete the broker-2 PVC (Strimzi will not remove PVCs by default)
kubectl -n kafka delete pvc data-0-prod-broker-2

# c) Delete the pod — StatefulSet re-creates with a fresh PVC (empty log dir)
kubectl -n kafka delete pod prod-broker-2

# The new prod-broker-2 boots empty; the KRaft controller has it in the
# metadata as broker 2 but with no data. Follower replicas will begin
# replicating from leaders. This is *not* Cruise Control's job — Kafka's
# native replication catches up automatically.

# 4. Once ISR is restored, run a full-rebalance to smooth any long-term imbalance
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: prod-full-rebalance
  namespace: kafka
  labels: { strimzi.io/cluster: prod }
spec:
  mode: full
  goals:
    - RackAwareGoal
    - ReplicaCapacityGoal
    - DiskCapacityGoal
    - ReplicaDistributionGoal
    - LeaderReplicaDistributionGoal

# 5. Approve when the proposal looks sensible (usually a small diff)
kubectl -n kafka annotate kafkarebalance prod-full-rebalance \
  strimzi.io/rebalance=approve

# 6. Verify final state
kubectl exec -n kafka prod-broker-0 -- \
  bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe \
  --under-min-isr-partitions
# Empty = every partition has ISR back to 3

Step-by-step trace.

Stage	Time	Cluster state
t=0	broker-2 pod crashes; PVC lost	ISR shrinks from 3 to 2 on broker-2's partitions
t=0-15m	operator retries pod start; PVC unbindable	broker still down
t=15m	on-call confirms EBS gone	begin recovery
t=15m30s	PVC + pod deleted	StatefulSet re-creates with fresh PVC
t=16m	new broker-2 pod running with empty log dir	begins as follower for its partition replicas
t=16m-1h	replication catch-up	log bytes copied from leaders
t=1h	ISR back to 3 for all affected partitions	producer acks fully restored
t=1h+	run KafkaRebalance for cosmetic smoothing	balanced state resumed

After the recovery, the cluster is back to 6 brokers with all partitions at RF=3 and ISR=3. Producer clients never saw an error because min.insync.replicas=2 held throughout. The rebalance step is cosmetic — Kafka's native replication already caught the broker up; Cruise Control just smooths any long-term imbalance from the incident (e.g. if some partitions were preferred-leader'd on broker-2 and never returned).

Output:

Metric	Value
Downtime for producers	0 seconds (min.isr held)
Time to restore ISR to full	~45-60 minutes (network-bound replication)
Operator intervention required	PVC delete + pod delete (one command each)
Cluster capacity impact	~17% reduced throughput while broker rebuilds
Post-recovery rebalance duration	~10-15 min at default throttle
Total incident-to-green time	~90 minutes for this scenario

Why this works — concept by concept:

• StatefulSet self-heal — Kubernetes' StatefulSet controller re-creates pods on failure and re-attaches PVCs by ordinal. Delete the PVC first (Strimzi's deleteClaim: false default is to protect data), then delete the pod, and the StatefulSet re-creates both cleanly with a fresh empty PVC. This is the only manual step.
• min.insync.replicas + acks=all — the producer contract that survives one broker loss. With RF=3 and min.isr=2, one broker gone means ISR=2, which is still >= min.isr, so producers keep acking. This is the same safety property that made the rolling upgrade work; here it holds during an unplanned outage.
• Native replication catch-up — Kafka's follower fetch protocol replicates missing bytes from leaders. The new broker boots empty, the controller sees it in the metadata, follower fetch threads begin catching up. No operator intervention needed; the cluster self-heals.
• KafkaRebalance full mode — after the incident, some brokers may be leader for more partitions than others (partition re-election chose live brokers during the outage). A full-mode KafkaRebalance smooths this via LeaderReplicaDistributionGoal. Cosmetic but worth doing to avoid load imbalance biting weeks later.
• Cost — one manual PVC delete + one manual pod delete, ~90 minutes of reduced-capacity operation, one Cruise Control rebalance at the end. No downtime, no data loss, no producer errors. Compared to the same incident on bare metal (manual broker re-provision, log dir rebuild, ACL re-apply, monitoring re-wire), this is a ~10× reduction in incident time. Net O(1) operator steps regardless of cluster size.

Streaming
Topic — streaming
Streaming Strimzi + Cruise Control problems

Practice →

Design Topic — design Design problems on Kubernetes operators

Practice →

---

3. Confluent for Kubernetes (CFK)

confluent operator bundles the entire Confluent Platform on Kubernetes — Kafka, Schema Registry, ksqlDB, Connect, RBAC, Control Center, tiered storage

The mental model in one line: Confluent for Kubernetes (CFK) is the commercial Kubernetes operator that deploys and manages the entire Confluent Platform — Kafka brokers plus Schema Registry plus Kafka Connect (with Confluent Hub connectors auto-installed) plus ksqlDB plus Control Center plus RBAC plus tiered storage plus MDS (Metadata Service) — under a per-CPU or per-broker commercial license, and its headline advantage over Strimzi is that every enterprise feature ships as a first-class CRD rather than a bolt-on integration, with commercial support, SOC-2 attestation, and Confluent's professional services on the other end of the phone. For teams whose leadership has already signed a Confluent contract, or whose day-one requirements include Confluent-native features (Control Center dashboards, ksqlDB, Confluent Hub connectors, LDAP/SAML integration), CFK is the pattern; for OSS-first teams, Strimzi is usually the answer.

The four axes for CFK.

• Control plane. The Confluent Operator watches CFK-specific CRDs (Kafka, SchemaRegistry, KafkaTopic, KafkaRestProxy, Connect, KsqlDB, ControlCenter, MdsCluster) and reconciles them into StatefulSets, Deployments, and Services. One operator manages the whole platform.
• CRD surface. ~15 CRDs — Kafka + every Confluent Platform component gets its own CRD. This is a larger surface than Strimzi's ~7 CRDs, reflecting the whole-platform bundling.
• Cost. Commercial. Per-CPU or per-broker licensing (Confluent's pricing is negotiated; ballpark $2-5k per broker per year plus per-CPU for connectors). Plus the K8s cluster and disks and engineering time.
• Ecosystem. Confluent-hosted Confluent Hub for pre-built connectors, Confluent Cloud integration (hybrid cloud story), Confluent Support / Professional Services / training. Strong integration with LDAP, SAML, Vault, and enterprise SSO.

The Confluent Operator — CRDs for every platform component.

• What it is. A Java controller (like Strimzi) that reconciles CFK CRDs into K8s primitives. Deployed via confluent for kubernetes Helm chart.
• What it manages. Kafka brokers + Zookeeper (legacy) or KRaft (CFK 2.9+), Schema Registry, Kafka Connect, ksqlDB, Control Center, MDS. Each is a first-class CRD.
• How it deploys. helm install confluent-operator confluentinc/confluent-for-kubernetes. Confluent's Helm charts + a valid license Secret.
• Failure semantics. Same reconcile pattern as Strimzi — idempotent, retries with backoff.

The Kafka CR — Confluent-flavour.

• spec.replicas. Broker count.
• spec.image.application. Confluent Server image (e.g. confluentinc/cp-server:7.7.1), not the vanilla apache/kafka image. Confluent Server includes tiered storage, MDS, and RBAC.
• spec.dataVolumeCapacity. Persistent disk size per broker.
• spec.configOverrides.server. Free-form Kafka broker config overrides.
• spec.authentication. mTLS, SASL/PLAIN, SASL/SCRAM, or Kerberos (Confluent's LDAP-backed SASL).
• spec.tls. TLS bundle (Secret containing cert + key + CA).

Tiered storage — the CFK headline enterprise feature.

• What it does. Older log segments move from broker-local disk to object storage (S3 / GCS / Azure Blob). Consumers reading historical data get segments from object storage transparently.
• Why it matters. Broker disk goes from "must fit your entire retention window" to "must fit only the hot tier." At 30-day retention with 500 MB/s ingest, that's the difference between 1.3 TB per broker and ~50 GB per broker.
• How CFK exposes it. spec.configOverrides.server sets confluent.tier.feature=true, confluent.tier.enable=true, and cloud-specific tier config. Topic-level enablement via KafkaTopic.spec.configs.
• Why interviewers ask. It's the single biggest cost-of-ownership win Confluent ships and the reason many organisations pick CFK over Strimzi (Strimzi's tiered storage story exists but is less turnkey).

Confluent RBAC + MDS — the enterprise auth story.

• What MDS is. The Metadata Service — a Confluent-shipped microservice that centralises role bindings across Kafka, Schema Registry, ksqlDB, Connect, and Control Center. Backed by an LDAP server (Active Directory, OpenLDAP) or an OAuth2 IdP.
• The role model. Coarse-grained roles (DeveloperRead, DeveloperWrite, ResourceOwner, SecurityAdmin, SystemAdmin) that expand into per-resource ACLs. Managed via confluent iam CLI or CRDs.
• The audit story. Every authz decision produces an audit event; feeds into Splunk / DataDog / SIEM.
• Why interviewers ask. For regulated industries (finance, healthcare, defence), the LDAP-integrated RBAC + audit story is non-negotiable, and building it on Strimzi requires custom OPA policies or Keycloak integration.

Common interview probes on CFK.

• "What does CFK give you that Strimzi doesn't?" — required answer: tiered storage first-class, LDAP-backed RBAC via MDS, Confluent Hub connector auto-install, Control Center dashboards.
• "How much does CFK cost?" — required answer: per-CPU or per-broker license; negotiated; expect $10k-$100k/year range for small-to-mid clusters.
• "When would you pick CFK over Strimzi?" — required answer: existing Confluent contract, enterprise RBAC + audit requirement, tiered storage as a day-one need.
• "What is Confluent Server vs Apache Kafka?" — required answer: Confluent Server = Apache Kafka + Confluent's proprietary extensions (tiered storage, RBAC hooks, MDS).
• "How does CFK handle a rolling upgrade?" — required answer: same operator-driven rolling upgrade as Strimzi, honouring ISR safety.

Worked example — CFK Kafka + Schema Registry + Control Center

Detailed explanation. The canonical CFK deployment: Kafka CR for brokers, SchemaRegistry CR for Avro/Protobuf schema management, ControlCenter CR for the ops dashboard. All three are reconciled by the Confluent Operator into StatefulSets/Deployments; they wire themselves together via well-known service names.

• Kafka. 3 brokers, mTLS, Confluent Server image (for tiered storage + RBAC hooks).
• Schema Registry. 2 replicas, wired to the Kafka cluster.
• Control Center. 1 replica, wired to Kafka + Schema Registry + MDS.

Question. Deploy Kafka + Schema Registry + Control Center via CFK CRDs.

Input.

Component	CFK CRD	Replicas	Purpose
Kafka	Kafka	3	broker cluster
Schema Registry	SchemaRegistry	2	Avro/Protobuf/JSON-schema registry
Control Center	ControlCenter	1	ops dashboard

Code.

# 1. Kafka CR — Confluent Server image, mTLS, tiered storage
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
  namespace: confluent
spec:
  replicas: 3
  image:
    application: confluentinc/cp-server:7.7.1
    init:       confluentinc/confluent-init-container:2.9.1
  dataVolumeCapacity: 500Gi
  storageClass:
    name: gp3-ssd
  tls:
    secretRef: tls-kafka
  listeners:
    internal:
      authentication:
        type: mtls
      tls:
        enabled: true
    external:
      externalAccess:
        type: loadBalancer
        loadBalancer:
          domain: kafka.example.com
      authentication:
        type: mtls
      tls:
        enabled: true
  configOverrides:
    server:
      - confluent.tier.feature=true
      - confluent.tier.enable=true
      - confluent.tier.backend=S3
      - confluent.tier.s3.bucket=acme-kafka-tier
      - confluent.tier.s3.region=us-east-1
      - min.insync.replicas=2
      - default.replication.factor=3

# 2. Schema Registry CR
apiVersion: platform.confluent.io/v1beta1
kind: SchemaRegistry
metadata:
  name: schemaregistry
  namespace: confluent
spec:
  replicas: 2
  image:
    application: confluentinc/cp-schema-registry:7.7.1
  tls:
    secretRef: tls-schemaregistry
  dependencies:
    kafka:
      bootstrapEndpoint: kafka.confluent.svc.cluster.local:9092
      tls:
        enabled: true
        ignoreTrustStoreConfig: false
      authentication:
        type: mtls

# 3. Control Center CR
apiVersion: platform.confluent.io/v1beta1
kind: ControlCenter
metadata:
  name: controlcenter
  namespace: confluent
spec:
  replicas: 1
  image:
    application: confluentinc/cp-enterprise-control-center:7.7.1
  dataVolumeCapacity: 20Gi
  tls:
    secretRef: tls-controlcenter
  dependencies:
    schemaRegistry:
      url: https://schemaregistry.confluent.svc.cluster.local:8081
      tls:
        enabled: true
    kafka:
      bootstrapEndpoint: kafka.confluent.svc.cluster.local:9092
      authentication:
        type: mtls
      tls:
        enabled: true

Step-by-step explanation.

1. The Kafka CR uses confluentinc/cp-server — Confluent's proprietary broker image that includes tiered-storage plumbing, MDS integration hooks, and RBAC support. Using the vanilla apache/kafka image would strip these features. confluent.tier.feature=true + confluent.tier.enable=true turn on tiered storage at the cluster level; per-topic enablement is a topic config.
2. The SchemaRegistry CR declares 2 replicas for HA. spec.dependencies.kafka wires it to the Kafka cluster's internal service endpoint; the operator resolves the Service name and wires the bootstrap. mTLS between Schema Registry and Kafka is enabled by referencing the same TLS Secret as the brokers.
3. The ControlCenter CR declares 1 replica (Control Center is single-writer; multi-replica requires enterprise licensing quirks not needed here). Dependencies on both Schema Registry and Kafka are wired via internal service DNS.
4. All three CRDs share a common pattern: one operator, one namespace, DNS-based dependency wiring. The Confluent Operator reconciles each CRD into a StatefulSet (Kafka) or Deployment (SR, C3), a Service, a ConfigMap, and (via referenced Secrets) TLS bundles.
5. The end result is a fully-functional Confluent Platform stack in one namespace: brokers on port 9092, Schema Registry on 8081, Control Center on 9021. The RBAC + MDS layer can be added on top via additional CRDs (MdsCluster, KafkaRestClass).

Output.

CR	K8s primitives created	Service DNS
Kafka/kafka	StatefulSet(3) + Service + ConfigMap	kafka.confluent.svc:9092
SchemaRegistry/schemaregistry	Deployment(2) + Service	schemaregistry.confluent.svc:8081
ControlCenter/controlcenter	Deployment(1) + Service + PVC	controlcenter.confluent.svc:9021

Rule of thumb. For any CFK deployment, deploy the operator + Kafka + SchemaRegistry + ControlCenter as a starter set. Adding Connect, ksqlDB, and MDS is one CRD each; the operator handles all the inter-component wiring via spec.dependencies blocks. Do not mix and match with Strimzi in the same cluster — CFK expects to own its namespace.

Worked example — CFK RBAC + LDAP integration

Detailed explanation. The RBAC story is the reason regulated-industry teams pay for CFK. Confluent's MDS (Metadata Service) integrates with LDAP (Active Directory / OpenLDAP) or OAuth2 to authenticate users, and role bindings map LDAP groups to coarse-grained Confluent roles (DeveloperRead, DeveloperWrite, ResourceOwner, SecurityAdmin) that expand into fine-grained ACLs across Kafka, Schema Registry, ksqlDB, and Connect. Walk through wiring LDAP-backed RBAC.

• MDS. Metadata Service deployed as a CFK CRD; the auth backbone.
• LDAP. External Active Directory or OpenLDAP; MDS reads user + group memberships.
• Role bindings. Coarse-grained roles bound to LDAP groups; expand into per-resource ACLs.

Question. Wire MDS to LDAP and grant the orders-team-devs LDAP group DeveloperWrite on the orders.* topics.

Input.

Component	Value
LDAP server	ldap.corp.example.com:636 (mTLS)
Bind DN	cn=cfk-mds,ou=service,dc=example,dc=com
User search base	ou=Users,dc=example,dc=com
Group search base	ou=Groups,dc=example,dc=com
Target group	orders-team-devs
Target role	DeveloperWrite on topic orders.*

Code.

# 1. MDS Cluster CR
apiVersion: platform.confluent.io/v1beta1
kind: KafkaRestClass
metadata:
  name: kafka-rest-class
  namespace: confluent
spec:
  kafkaClusterRef:
    name: kafka
  kafkaRest:
    endpoint: https://kafka.confluent.svc.cluster.local:8090
    authentication:
      type: bearer
      bearer:
        secretRef: mds-token
    tls:
      enabled: true
---
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
  namespace: confluent
spec:
  # ... existing spec ...
  services:
    mds:
      externalAccess:
        type: loadBalancer
      tls:
        enabled: true
      tokenKeyPair:
        secretRef: mds-token-keypair    # RSA keypair for JWT signing
      provider:
        type: ldap
        ldap:
          address: "ldaps://ldap.corp.example.com:636"
          authentication:
            type: simple
            simple:
              secretRef: ldap-bind-cred   # bindDn + password
          configurations:
            groupNameAttribute: cn
            groupObjectClass: group
            groupMemberAttribute: member
            groupMemberAttributePattern: CN=(.*),OU=Users,DC=example,DC=com
            groupSearchBase: ou=Groups,dc=example,dc=com
            groupSearchFilter: (objectClass=group)
            userNameAttribute: sAMAccountName
            userObjectClass: user
            userSearchBase: ou=Users,dc=example,dc=com
            userSearchFilter: (objectClass=user)

# 2. Once MDS is up, bind an LDAP group to a Confluent role
confluent iam rbac role-binding create \
  --principal "Group:orders-team-devs" \
  --role DeveloperWrite \
  --resource "Topic:orders." --prefix \
  --kafka-cluster-id "$(kubectl exec -n confluent kafka-0 -- kafka-cluster-id --bootstrap-server localhost:9092)"

# 3. Verify
confluent iam rbac role-binding list \
  --principal "Group:orders-team-devs"
# ROLE              RESOURCE                     TYPE   NAME       PREFIX
# DeveloperWrite    Topic                        Topic  orders.    true

# 4. Producer connecting with LDAP-backed OAuth2 token
from confluent_kafka import Producer
import requests

# a) Get an MDS token via LDAP bind
token_resp = requests.post(
    "https://mds.confluent.svc.cluster.local:8090/security/1.0/authenticate",
    auth=("alice", "alice-ldap-password"),      # LDAP-authenticated user
    verify="/etc/confluent/tls/ca.crt",
)
token = token_resp.json()["auth_token"]

# b) Use the token in the Kafka producer's OAUTHBEARER auth
producer = Producer({
    "bootstrap.servers":    "kafka.confluent.svc.cluster.local:9092",
    "security.protocol":    "SASL_SSL",
    "sasl.mechanism":       "OAUTHBEARER",
    "sasl.oauthbearer.config": f"principalClaimName=sub extension_logicalCluster=lkc-abc",
    "ssl.ca.location":      "/etc/confluent/tls/ca.crt",
})
producer.produce("orders.events", key=b"42", value=b'{"order_id": 42}')
producer.flush()

Step-by-step explanation.

1. The MDS is configured with provider.type: ldap and points at the corporate LDAP server on port 636 (LDAPS). The bindDn (from the ldap-bind-cred Secret) is a service account that MDS uses to read the LDAP directory; user auth is a subsequent LDAP simple-bind with the end-user's credentials.
2. The KafkaRestClass CR references the MDS endpoint and the RSA keypair used to sign JWTs. On authentication, MDS validates the user against LDAP, looks up their group memberships, then issues a signed JWT with the group list embedded.
3. The confluent iam rbac role-binding create command creates a role binding: LDAP group orders-team-devs gets Confluent role DeveloperWrite on any topic matching prefix orders.. This expands into per-topic ACLs (Read, Write, Describe) that Kafka enforces natively.
4. The producer authenticates via SASL/OAUTHBEARER using the JWT from MDS. Kafka's broker verifies the JWT signature (using the MDS public key), extracts the group claim, and checks it against the ACL. Any produce to orders.* succeeds; any produce to customers.* fails with authorization error.
5. The end story: users authenticate once against LDAP (single sign-on), get a JWT that names their groups, and Kafka enforces authorization based on the group-to-role bindings. Adding a new team member is an LDAP group membership change; no Kafka-side action needed. This is what enterprise auth looks like.

Output.

Component	Behavior
MDS	validates user against LDAP; issues JWT with group claim
Kafka broker	validates JWT signature; enforces ACL from role binding
orders-team-devs group member	can Read + Write + Describe topics orders.*
Non-member	authz error on orders.* produce
Admin	uses confluent iam CLI to manage bindings
Audit	every authz decision logs to MDS audit topic

Rule of thumb. For any CFK deployment in a regulated industry, wire MDS to LDAP or OAuth2 on day one — RBAC retrofit is painful. Grant roles to groups, not individual users, so team membership changes are LDAP-side operations. Audit logs feed into the corporate SIEM automatically.

Worked example — enabling tiered storage per topic

Detailed explanation. Tiered storage moves older log segments off broker-local disk into object storage. The savings are dramatic: at 30-day retention with 500 MB/s ingest, a broker's local disk requirement drops from ~1.3 TB to ~50 GB (the hot tier). Turning it on is a two-step process: cluster-level enablement (in the Kafka CR) and per-topic enablement (in KafkaTopic or via kafka-configs.sh).

• Cluster. confluent.tier.feature=true + confluent.tier.enable=true + S3 config in Kafka.spec.configOverrides.server.
• Topic. confluent.tier.enable=true + local.retention.ms=86400000 (24 h hot tier) + retention.ms=2592000000 (30 days total).
• IAM. The broker's IAM role needs s3:PutObject, s3:GetObject, s3:DeleteObject on the tier bucket.

Question. Enable tiered storage on the orders.events topic with 24-hour hot tier + 30-day total retention.

Input.

Setting	Cluster (Kafka.spec.configOverrides.server)	Topic (KafkaTopic.spec.configs)
confluent.tier.feature	true	(n/a)
confluent.tier.enable	true	true
confluent.tier.backend	S3	(inherited)
confluent.tier.s3.bucket	acme-kafka-tier	(inherited)
local.retention.ms	(n/a; per topic)	86400000 (24 h)
retention.ms	(n/a; per topic)	2592000000 (30 d)

Code.

# 1. Cluster-level enablement (Kafka CR excerpt)
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
  namespace: confluent
spec:
  # ... existing spec ...
  podTemplate:
    serviceAccountName: kafka-tier-role   # bound to IRSA role with S3 access
  configOverrides:
    server:
      - confluent.tier.feature=true
      - confluent.tier.enable=true
      - confluent.tier.backend=S3
      - confluent.tier.s3.bucket=acme-kafka-tier
      - confluent.tier.s3.region=us-east-1
      - confluent.tier.local.hotset.bytes=1073741824       # 1 GiB hot per segment
      - confluent.tier.local.hotset.ms=86400000            # 24 h hot per segment
      - confluent.tier.metadata.replication.factor=3

# 2. Per-topic enablement
apiVersion: platform.confluent.io/v1beta1
kind: KafkaTopic
metadata:
  name: orders-events
  namespace: confluent
spec:
  replicas: 3
  partitionCount: 24
  kafkaClusterRef:
    name: kafka
  configs:
    confluent.tier.enable: "true"
    local.retention.ms: "86400000"      # 24 h on local disk
    retention.ms: "2592000000"          # 30 d total; older = S3
    cleanup.policy: "delete"
    segment.bytes: "1073741824"         # 1 GiB segments (larger = fewer S3 objects)

// 3. IAM policy on the tier bucket (attached via IRSA to kafka-tier-role)
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject",
                 "s3:AbortMultipartUpload", "s3:ListBucketMultipartUploads"],
      "Resource": "arn:aws:s3:::acme-kafka-tier/*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::acme-kafka-tier"
    }
  ]
}

Step-by-step explanation.

1. The cluster-level config turns on tiered storage globally: brokers connect to S3 via IRSA (IAM Roles for Service Accounts), the tier metadata topic replicates across 3 brokers, and segments larger than the hot-set thresholds get moved to S3.
2. The per-topic confluent.tier.enable: true opts the topic into tiering. Without this, topic segments stay local even if the cluster is tier-enabled. local.retention.ms=86400000 keeps 24 hours of hot data on the broker; retention.ms=2592000000 sets the total (hot + cold) retention window to 30 days.
3. The IAM policy grants the broker service account the S3 actions needed for tier uploads (PutObject, AbortMultipartUpload), tier reads (GetObject), and tier cleanup (DeleteObject). This is the only IAM change the DBA / SRE has to make.
4. On the broker side, a background thread (TierArchiverThread) monitors log segments and uploads eligible segments to S3, then updates the tier metadata topic with the object location. Consumer fetches for historical data hit S3 transparently — the consumer never knows.
5. The end story: broker disk is bounded at ~1 hot-set worth of data per topic (roughly hot retention × ingest rate), and object storage takes the long tail. For a 30-day retention topic at 500 MB/s, that's ~40 GB broker disk vs ~1.3 TB total — a 32x reduction. Consumer latency for hot fetches is unchanged; cold fetches add ~50-100 ms for the S3 round-trip.

Output.

Metric	Without tiering	With tiering
Broker local disk needed	~1.3 TB	~40-50 GB
Total data stored	~1.3 TB (on expensive EBS)	~50 GB EBS + 1.25 TB S3
Cost	3 × 1.3 TB EBS gp3 = ~$300/mo	~$15/mo EBS + ~$30/mo S3 = ~$45/mo
Consumer p95 latency (hot)	~5 ms	~5 ms (unchanged)
Consumer p95 latency (cold historical)	~10 ms	~50-100 ms (S3 round-trip)

Rule of thumb. For any CFK cluster with retention over 7 days and ingest over 100 MB/s, enable tiered storage — the disk cost savings pay for the license difference between Strimzi and CFK many times over. Set local.retention.ms to your read pattern's hot window; anything older reads from S3 with a small latency penalty.

Interview Question on CFK

A senior interviewer might ask: "Your organisation has a Confluent Platform contract signed for 5 years. Leadership wants a 12-broker Kafka cluster on EKS with Schema Registry, Control Center, LDAP-backed RBAC via MDS, and tiered storage for a 90-day retention window. Walk me through the CFK deployment, the day-one RBAC bootstrap, the tiered-storage cost model versus a non-tiered baseline, and the disaster-recovery story when the control-plane MDS pod dies."

Solution Using CFK with MDS + LDAP + tiered storage + multi-AZ StatefulSet

# 1. Kafka CR — 12 brokers, mTLS, tiered storage on S3, MDS enabled
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
  namespace: confluent
spec:
  replicas: 12
  image:
    application: confluentinc/cp-server:7.7.1
    init:       confluentinc/confluent-init-container:2.9.1
  dataVolumeCapacity: 200Gi
  storageClass: { name: gp3-ssd }
  podTemplate:
    serviceAccountName: kafka-tier-role
    podSecurityContext: { fsGroup: 1001 }
    affinity:
      podAntiAffinity:                # spread brokers across AZs
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              topologyKey: topology.kubernetes.io/zone
              labelSelector:
                matchLabels: { app: kafka }
  listeners:
    internal: { tls: { enabled: true }, authentication: { type: mtls } }
    external:
      externalAccess:
        type: loadBalancer
        loadBalancer: { domain: kafka.example.com }
      tls: { enabled: true }
      authentication: { type: mtls }
  services:
    mds:
      tokenKeyPair: { secretRef: mds-token-keypair }
      provider:
        type: ldap
        ldap:
          address: "ldaps://ldap.corp.example.com:636"
          authentication: { type: simple, simple: { secretRef: ldap-bind-cred } }
  configOverrides:
    server:
      - confluent.tier.feature=true
      - confluent.tier.enable=true
      - confluent.tier.backend=S3
      - confluent.tier.s3.bucket=acme-kafka-tier
      - confluent.tier.s3.region=us-east-1
      - confluent.tier.metadata.replication.factor=3
      - min.insync.replicas=2
      - default.replication.factor=3
      - unclean.leader.election.enable=false

# 2. Bootstrap RBAC — grant admin role to the SecurityAdmin LDAP group
CLUSTER_ID=$(kubectl -n confluent exec kafka-0 -- kafka-cluster-id --bootstrap-server localhost:9092)

confluent iam rbac role-binding create \
  --principal "Group:cfk-security-admins" \
  --role SecurityAdmin \
  --kafka-cluster-id "$CLUSTER_ID"

# 3. Grant per-team developer roles
confluent iam rbac role-binding create \
  --principal "Group:orders-team-devs" \
  --role ResourceOwner \
  --resource "Topic:orders." --prefix \
  --resource "Group:orders." --prefix \
  --kafka-cluster-id "$CLUSTER_ID"

# 4. Every topic gets tiered storage
apiVersion: platform.confluent.io/v1beta1
kind: KafkaTopic
metadata:
  name: orders-events
  namespace: confluent
spec:
  replicas: 3
  partitionCount: 48
  kafkaClusterRef: { name: kafka }
  configs:
    confluent.tier.enable: "true"
    local.retention.ms: "86400000"       # 24 h hot
    retention.ms: "7776000000"           # 90 d total
    segment.bytes: "1073741824"          # 1 GiB

Step-by-step trace.

Layer	Choice	Reasoning
Operator	Confluent for Kubernetes (CFK)	leadership contract; enterprise features
Brokers	12, cp-server image, JBOD 200 Gi (hot only)	tiered storage cuts local disk 30×
Anti-affinity	soft anti-affinity by AZ	multi-AZ spread; no hard requirement (avoids scheduling stalls)
MDS	LDAP-backed	corporate SSO; audit trail
Tiered storage	S3 with IRSA	90-day retention on cheap object storage
RBAC	ResourceOwner per team on prefix	teams manage own topics; SecurityAdmin group manages RBAC
Replication	RF=3, min.isr=2	survive one broker loss
Unclean leader election	disabled	never elect a leader outside ISR (correctness > availability)

After the rollout, the 12-broker cluster serves ~1 GB/s ingest across 48-partition topics with mTLS + LDAP-backed RBAC. Each broker holds ~40 GB of hot data (24 h × ingest per broker); the remaining ~2 PB of 90-day-retention data lives in S3. Control Center shows real-time topic-level throughput, consumer lag, and broker-level disk usage. Adding a new team is a LDAP group + one confluent iam rbac role-binding create; new topics are KafkaTopic CRs auto-tiered.

Output:

Metric	Value
Broker count	12 (multi-AZ)
Broker local disk	~40 GB each (hot tier only)
Total data (hot + tier)	~2 PB over 90-day window
EBS cost (12 × 200 GiB gp3)	~$240/month
S3 tier cost	~$40k/month at ~2 PB standard-IA
Non-tiered baseline (comparison)	12 × 200 TB EBS = ~$300k/month
Effective retention	90 days
Ingest sustained	~1 GB/s
p95 producer ack	~10 ms
RBAC surface	LDAP-backed; group role bindings
Audit	MDS audit topic → SIEM

Why this works — concept by concept:

• Confluent Operator + Kafka CR — reconciliation loop with the same shape as Strimzi but bundled with the entire Confluent Platform. One operator manages brokers, MDS, and (via other CRDs) Schema Registry, Connect, ksqlDB, Control Center. Single control plane for the whole stack.
• MDS + LDAP — corporate SSO backing Confluent RBAC. Users authenticate once against LDAP, get a JWT with group claims, and Kafka enforces topic-level authorization based on group-to-role bindings. No per-user Kubernetes Secret management.
• Tiered storage (S3 backend) — older log segments move to S3 transparently. Broker local disk requirement drops 30× at 90-day retention; the license fee is more than offset by the disk savings. This is the CFK feature that most often justifies the license.
• Soft AZ anti-affinity + IRSA — preferredDuringSchedulingIgnoredDuringExecution spreads brokers across AZs without hard-requiring it (a hard requirement can stall scheduling in AZ-drained scenarios). IRSA lets each broker's ServiceAccount assume an IAM role scoped exactly to the tier bucket.
• Cost — 12 brokers × 200 GiB EBS ≈ $240/mo, S3 tier at ~2 PB ≈ $40k/mo, EKS cluster costs, and the Confluent license (~$30-60k/mo for 12 brokers at Confluent's ballpark pricing). Compared to the non-tiered EBS baseline of $300k/mo, tiered storage saves ~$260k/mo — the CFK license pays for itself many times over. Net O(1) operator + O(N) StatefulSet + O(retention × ingest) in cheap S3; hot tier stays bounded regardless of retention.

Streaming
Topic — streaming
Streaming Confluent Platform problems

Practice →

Design Topic — design Design problems on tiered storage + RBAC

Practice →

---

4. KRaft on bare metal / EC2

kraft mode on bare metal is the pattern for teams whose latency budget cannot afford Kubernetes' overhead — direct NVMe, systemd, and a ~30-50% throughput edge

The mental model in one line: kraft mode on bare metal (or EC2 with instance-store NVMe) is the deployment pattern where Kafka runs as a systemd-managed process on real (or virtual) hardware with direct disk I/O, no Kubernetes network stack, no container runtime, and no operator between you and the JVM — and the reward for building your own operator in Ansible is a ~30-50% throughput edge on the same hardware, sub-3-millisecond p99 producer-ack latency, and complete freedom from the Kubernetes scheduler's opinions about where your log directory lives; the cost is that every operation the operator would have done for you (rolling upgrade, broker replacement, ACL rotation, topic creation) is now a runbook you write, test, and page for. For 99% of workloads this trade is a bad one; for the 1% (ad bidding, HFT ticker feeds, real-time gaming state, tier-1 IoT ingest), it is the whole ball game.

The four axes for bare-metal KRaft.

• Control plane. systemd + Ansible + your on-call runbook. No operator, no CRDs. Every operation is a script.
• Latency. Sub-3 ms p95 producer ack achievable on modern NVMe hardware. K8s network stack (kube-proxy, CNI overlay, container-runtime networking) adds ~200-500 μs that bare metal skips. Direct NVMe access via a filesystem (XFS) skips the block-device virtualization layer K8s PVCs go through.
• Cost. Free software. Hardware you own or EC2 instances you rent. No license fee. Engineering time is the biggest cost — you build the operator's convenience yourself.
• Ecosystem. Whatever you build. Ansible playbooks, systemd unit files, Prometheus JMX exporters, Grafana dashboards you author, and runbooks for every operation.

KRaft — what removing ZooKeeper actually changed.

• Before KRaft. Kafka needed a ZooKeeper ensemble (3-5 ZK nodes) for cluster metadata: broker registrations, topic partitions, controller election. Every Kafka cluster was really two clusters — the Kafka brokers and the ZK ensemble — with their own operational surface and version-skew concerns.
• After KRaft. A dedicated KRaft controller quorum (3 or 5 nodes) runs a Raft consensus algorithm to manage cluster metadata directly. The metadata log is a Kafka topic itself (@metadata); brokers subscribe to it. One control plane instead of two.
• When. KIP-500 designed the change; GA in Kafka 3.5 (2023); default in Kafka 4.0 (2025); ZooKeeper support removed in 4.0.
• Why it matters. Simplified ops surface (one thing to run, not two), faster failover (Raft election in ms vs ZK's seconds), smaller cluster footprint (no ZK ensemble to house).

The controller quorum — 3 or 5, always odd.

• Why odd. Raft requires a majority (quorum) to make progress: 3 nodes = 2 must agree; 5 nodes = 3 must agree. Even counts (2, 4) don't help with fault tolerance — 4 nodes still only tolerate 1 failure.
• Why 3 vs 5. 3 controllers tolerate 1 loss; 5 controllers tolerate 2 losses. Choose 3 for small clusters or when metadata write rate is low; choose 5 for large clusters or when you need to tolerate a full AZ loss.
• Colocation vs dedicated. Small clusters can colocate controllers with brokers (process.roles=broker,controller); large or latency-critical clusters use dedicated controllers (process.roles=controller on 3-5 boxes). Dedicated is the recommendation for anything over 20 brokers.

Broker + controller anatomy — the two roles in KRaft.

• Controller role. Runs the Raft consensus for the metadata log. Elects a single active controller. Handles metadata writes (topic create, partition assignment, broker register).
• Broker role. Handles the data plane — producers, consumers, log storage, replication. Subscribes to the metadata log for cluster state.
• Config split. process.roles=controller (dedicated controller), process.roles=broker (dedicated broker), process.roles=broker,controller (colocated). node.id unique per node; controller.quorum.voters lists the controller node.ids.

systemd + Ansible — the operational plumbing.

• systemd unit file. One unit per node (kafka-broker@.service or kafka-controller@.service), managing the JVM lifecycle, restart-on-failure, and log capture to journald.
• Ansible role. One role for each of: OS baseline (kernel tuning, ulimit), JDK install, Kafka install (unpack tarball to /opt/kafka), config templating (server.properties from Jinja2), systemd unit deploy, TLS keystore distribution.
• Rolling upgrade. Ansible playbook: for each broker, wait-for-ISR-full, drain leadership, systemctl restart, wait-for-ISR-full, next broker.

Common interview probes on bare-metal KRaft.

• "When would you pick bare metal over Kubernetes?" — required answer: latency-critical workloads, no K8s available, or existing bare-metal ops maturity.
• "How does KRaft change the operational surface?" — required answer: no ZooKeeper; one thing to run; faster failover; smaller footprint.
• "How many controllers do you need?" — required answer: 3 or 5, always odd; 3 tolerates 1 failure, 5 tolerates 2.
• "What's the p99 latency edge over Kubernetes?" — required answer: 200-500 μs saved on network stack; direct NVMe skips block-device virtualization.
• "How do you handle rolling upgrades without an operator?" — required answer: Ansible playbook with ISR-safety gate per broker.

Worked example — the KRaft controller quorum config

Detailed explanation. The canonical KRaft bare-metal setup: 3 dedicated controllers running the Raft quorum, 3 (or more) dedicated brokers running the data plane. Each node has a distinct node.id; the controller node.ids are listed in controller.quorum.voters; every node knows the full quorum on startup.

• Controllers. 3 nodes, process.roles=controller, node.ids 1/2/3, port 9093.
• Brokers. 3 nodes, process.roles=broker, node.ids 101/102/103, port 9092.
• Quorum voters. Same list on every node: 1@controller-1:9093,2@controller-2:9093,3@controller-3:9093.

Question. Write the KRaft config files for 3 controllers + 3 brokers on bare metal / EC2.

Input.

Component	node.id	Roles	Ports
controller-1	1	controller	9093
controller-2	2	controller	9093
controller-3	3	controller	9093
broker-101	101	broker	9092
broker-102	102	broker	9092
broker-103	103	broker	9092

Code.

# /opt/kafka/config/controller.properties  (deployed to controller-1/2/3, node.id differs)
process.roles=controller
node.id=1
controller.quorum.voters=[e-mail]:9093,[e-mail]:9093,[e-mail]:9093
listeners=CONTROLLER://:9093
listener.security.protocol.map=CONTROLLER:SSL
controller.listener.names=CONTROLLER

# Metadata log location (dedicated disk if possible)
metadata.log.dir=/var/lib/kafka/metadata

# TLS for controller-to-controller traffic
ssl.keystore.location=/etc/kafka/certs/controller-1.p12
ssl.keystore.password=changeme
ssl.keystore.type=PKCS12
ssl.truststore.location=/etc/kafka/certs/ca.p12
ssl.truststore.password=changeme

# Raft tuning
controller.quorum.election.timeout.ms=1000
controller.quorum.fetch.timeout.ms=2000

# /opt/kafka/config/broker.properties  (deployed to broker-101/102/103, node.id differs)
process.roles=broker
node.id=101
controller.quorum.voters=[e-mail]:9093,[e-mail]:9093,[e-mail]:9093

listeners=PLAINTEXT://:9092,SSL://:9094
advertised.listeners=PLAINTEXT://broker-101.internal:9092,SSL://broker-101.public.example.com:9094
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,CONTROLLER:SSL
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER

# Data log directories — one per NVMe disk (JBOD)
log.dirs=/var/lib/kafka/data-0,/var/lib/kafka/data-1

# Replication / durability
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2

# Retention
log.retention.hours=168             # 7 days
log.segment.bytes=1073741824        # 1 GiB
log.roll.hours=24

# Performance
num.network.threads=8
num.io.threads=16
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600

# /etc/systemd/system/kafka-broker.service
[Unit]
Description=Apache Kafka broker
After=network.target
Wants=network.target

[Service]
Type=simple
User=kafka
Group=kafka
Environment="KAFKA_HEAP_OPTS=-Xms8g -Xmx8g"
Environment="KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true"
Environment="KAFKA_JMX_OPTS=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9999"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/broker.properties
Restart=on-failure
RestartSec=10
LimitNOFILE=1048576
LimitNPROC=32768
TimeoutStopSec=180

[Install]
WantedBy=multi-user.target

Step-by-step explanation.

1. Each controller node runs controller.properties with a unique node.id (1, 2, 3) and the same controller.quorum.voters list. On startup, each controller connects to the others, runs Raft leader election, and one becomes the active controller. Metadata operations flow through the active controller and get replicated to followers via the metadata log.
2. Each broker node runs broker.properties with a unique node.id (101, 102, 103) and the same controller.quorum.voters list. Brokers do not need to be listed in the voters — they're not part of the quorum, they just subscribe to the metadata log to learn cluster state.
3. The listeners and advertised.listeners split separates internal cluster traffic (PLAINTEXT on 9092 for broker-to-broker replication, cheap over trusted VPC) from external client traffic (SSL on 9094 for producers/consumers coming over public networks). This is the standard hybrid pattern; production hardening would move everything to SSL.
4. log.dirs=/var/lib/kafka/data-0,/var/lib/kafka/data-1 is JBOD — Kafka spreads partitions across the two NVMe disks mounted at those paths. Each NVMe disk gives ~500k IOPS; two disks per broker means ~1M IOPS, well beyond what a K8s PVC on EBS can deliver.
5. The systemd unit runs the broker as a non-root kafka user with G1GC tuning (20 ms max pause), file descriptor limit 1M (Kafka opens one FD per partition file), and the JMX port exposed for the Prometheus JMX exporter. Restart=on-failure + RestartSec=10 gives systemd a chance to recover the broker without human intervention on transient crashes.

Output.

Node	Role	node.id	Port	Data dir
controller-1	controller	1	9093	/var/lib/kafka/metadata
controller-2	controller	2	9093	/var/lib/kafka/metadata
controller-3	controller	3	9093	/var/lib/kafka/metadata
broker-101	broker	101	9092/9094	/var/lib/kafka/data-0, data-1
broker-102	broker	102	9092/9094	/var/lib/kafka/data-0, data-1
broker-103	broker	103	9092/9094	/var/lib/kafka/data-0, data-1

Rule of thumb. For any bare-metal KRaft deployment, dedicate controllers (never colocate for clusters over 20 brokers), use JBOD across multiple NVMe disks per broker, and template every config from Ansible so the only per-host variable is node.id. Every operational primitive (start, stop, restart, upgrade) is a systemd command; every config change is an Ansible playbook run.

Worked example — kernel + OS tuning for the p99 edge

Detailed explanation. The 30-50% throughput edge and sub-3-millisecond p99 that bare-metal KRaft can deliver requires deliberate OS-level tuning. Out of the box, Linux is optimised for general-purpose workloads; Kafka needs different defaults for filesystem, network, and virtual-memory subsystems.

• Filesystem. XFS with noatime,nodiratime mount options; large stripe alignment; disable read-ahead for random workloads.
• Kernel network. Larger socket buffers; larger connection backlog; tune BBR TCP congestion control.
• Virtual memory. Lower vm.swappiness (10 or less); larger dirty-page thresholds so log writes flush in big batches.
• File descriptors. 1M+ per-process limit; Kafka opens 1 FD per log segment per partition.

Question. Write the Ansible task list that tunes an EC2 i4i.4xlarge (32 vCPU, 128 GB RAM, 3.75 TB local NVMe) for Kafka.

Input.

Parameter	Default	Tuned	Reason
fs.file-max	100k	4M	Kafka opens 1 FD per segment file
vm.swappiness	60	1	never swap; Kafka relies on page cache
vm.dirty_background_ratio	10	5	flush earlier, smoother writes
vm.dirty_ratio	20	10	cap dirty pages tighter
net.core.rmem_max	208k	16M	large socket receive buffers
net.core.wmem_max	208k	16M	large socket send buffers
net.ipv4.tcp_congestion_control	cubic	bbr	better bulk throughput

Code.

# Ansible role: kafka-host-tuning
- name: Set filesystem limits
  ansible.posix.sysctl:
    name: "{{ item.name }}"
    value: "{{ item.value }}"
    sysctl_set: yes
    state: present
    reload: yes
  loop:
    - { name: fs.file-max,                    value: "4194304" }
    - { name: vm.swappiness,                  value: "1" }
    - { name: vm.dirty_background_ratio,      value: "5" }
    - { name: vm.dirty_ratio,                 value: "10" }
    - { name: vm.dirty_expire_centisecs,      value: "12000" }
    - { name: net.core.rmem_max,              value: "16777216" }
    - { name: net.core.wmem_max,              value: "16777216" }
    - { name: net.ipv4.tcp_rmem,              value: "4096 87380 16777216" }
    - { name: net.ipv4.tcp_wmem,              value: "4096 65536 16777216" }
    - { name: net.core.netdev_max_backlog,    value: "5000" }
    - { name: net.ipv4.tcp_max_syn_backlog,   value: "8192" }
    - { name: net.ipv4.tcp_congestion_control, value: "bbr" }

- name: Set per-user limits
  community.general.pam_limits:
    domain: kafka
    limit_type: "{{ item.type }}"
    limit_item: "{{ item.item }}"
    value: "{{ item.value }}"
  loop:
    - { type: hard, item: nofile,  value: "1048576" }
    - { type: soft, item: nofile,  value: "1048576" }
    - { type: hard, item: nproc,   value: "32768" }
    - { type: soft, item: nproc,   value: "32768" }

- name: Format NVMe disks with XFS
  community.general.filesystem:
    fstype: xfs
    dev: "{{ item }}"
    opts: "-f -d su=64k,sw=1"
  loop:
    - /dev/nvme1n1
    - /dev/nvme2n1

- name: Mount NVMe disks with Kafka-friendly options
  ansible.posix.mount:
    path: "/var/lib/kafka/{{ item.mount }}"
    src: "{{ item.dev }}"
    fstype: xfs
    opts: "noatime,nodiratime,nobarrier,logbufs=8,logbsize=256k"
    state: mounted
  loop:
    - { dev: /dev/nvme1n1, mount: data-0 }
    - { dev: /dev/nvme2n1, mount: data-1 }

# Verify — run on each host post-provisioning
sysctl fs.file-max vm.swappiness vm.dirty_ratio net.core.rmem_max net.ipv4.tcp_congestion_control

# Check XFS mount options
mount | grep /var/lib/kafka

# Check the kafka user's limits
sudo -u kafka bash -c 'ulimit -n; ulimit -u'

Step-by-step explanation.

1. fs.file-max=4194304 raises the system-wide file-descriptor limit far above Kafka's needs. A cluster with 10,000 partitions × 3 replicas × 10 segments per partition = 300k open FDs on a single broker. Default 100k would OOM the process.
2. vm.swappiness=1 tells Linux to prefer dropping page cache over swapping process memory. Kafka relies heavily on the page cache for log-file reads; swapping the JVM heap out would cause GC pauses that dwarf any Kafka latency budget.
3. The network tuning (larger rmem_max + wmem_max, BBR congestion control) raises the effective throughput on 10-25 Gbps NICs. Default socket buffers cap out at ~5 Gbps; the tuning lets Kafka fill a 25 Gbps NIC.
4. XFS with noatime,nodiratime,nobarrier,logbufs=8,logbsize=256k is the Kafka-recommended mount option set. noatime skips access-time writes on every read; nobarrier (safe on NVMe with power-loss protection) skips write barriers; the log buffer tuning makes XFS journal writes efficient.
5. The pam_limits block sets per-user nofile=1M and nproc=32k for the kafka user — this matches the systemd unit's LimitNOFILE=1048576 and ensures the JVM can open enough file descriptors and spawn enough threads.

Output.

Metric	Untuned baseline	Tuned
Sustained ingest (single broker)	~800 MB/s	~1.5 GB/s
p99 producer ack	~15 ms	~3 ms
GC pause p99	~50 ms	~10 ms (G1 + no swap)
Filesystem journal cost	~5% of write time	~1%
Open FDs headroom	tight	3-4× headroom

Rule of thumb. For any bare-metal Kafka deployment, run the tuning playbook on day one and re-run it on every OS upgrade. The 2× throughput gap between untuned and tuned defaults is real and reproducible. On Kubernetes you can't get most of this (kernel is shared; sysctls are per-node), which is one reason bare metal keeps the latency edge.

Worked example — Ansible rolling upgrade with ISR-safety gate

Detailed explanation. Without an operator, the rolling upgrade is a runbook. The safety property is the same as Strimzi's — never drop below min.insync.replicas — but you enforce it yourself in Ansible. Walk through the playbook.

• The safety check. Before restarting each broker, verify --under-min-isr-partitions is empty via kafka-topics.sh from a healthy broker.
• The restart. systemctl restart kafka-broker; wait for the JVM to come back and rejoin ISR.
• The wait. Poll kafka-broker-api-versions.sh until the broker responds; poll --describe --under-replicated-partitions until zero.

Question. Write the Ansible playbook for a rolling upgrade of a 6-broker bare-metal KRaft cluster.

Input.

Step	Command	Wait condition
pre-flight	kafka-topics.sh --under-min-isr-partitions	empty
stop	systemctl stop kafka-broker	process gone
upgrade	unarchive new tarball to /opt/kafka	file present
start	systemctl start kafka-broker	port 9092 listening
rejoin	kafka-topics.sh --under-replicated-partitions	empty (or timeout 15 min)
next	proceed to next broker in serial	after previous is healthy

Code.

# Ansible playbook: kafka-rolling-upgrade.yml
- name: Rolling upgrade of Kafka brokers
  hosts: kafka_brokers
  serial: 1                                 # one broker at a time, mandatory
  become: yes
  vars:
    kafka_new_version: "3.9.0"
    kafka_tarball_url: "https://archive.apache.org/dist/kafka/{{ kafka_new_version }}/kafka_2.13-{{ kafka_new_version }}.tgz"

  pre_tasks:
    - name: Pre-flight — verify cluster has no under-min-isr partitions
      shell: |
        /opt/kafka/bin/kafka-topics.sh \
          --bootstrap-server broker-101.internal:9092 \
          --describe --under-min-isr-partitions
      delegate_to: broker-101.internal
      run_once: true                         # only check once per playbook run
      register: pre_isr
      failed_when: pre_isr.stdout != ""

  tasks:
    - name: Stop Kafka broker
      systemd:
        name: kafka-broker
        state: stopped

    - name: Download new Kafka tarball
      get_url:
        url: "{{ kafka_tarball_url }}"
        dest: "/tmp/kafka-{{ kafka_new_version }}.tgz"
        checksum: "sha512:{{ kafka_expected_sha512 }}"

    - name: Extract Kafka to /opt/kafka
      unarchive:
        src: "/tmp/kafka-{{ kafka_new_version }}.tgz"
        dest: /opt/
        remote_src: yes
        owner: kafka
        group: kafka

    - name: Update /opt/kafka symlink
      file:
        src: "/opt/kafka_2.13-{{ kafka_new_version }}"
        dest: /opt/kafka
        state: link
        force: yes
        owner: kafka
        group: kafka

    - name: Start Kafka broker
      systemd:
        name: kafka-broker
        state: started
        daemon_reload: yes

    - name: Wait for broker port to accept connections
      wait_for:
        port: 9092
        host: "{{ inventory_hostname }}"
        delay: 5
        timeout: 300

    - name: Wait for broker to rejoin ISR (no under-replicated partitions)
      shell: |
        /opt/kafka/bin/kafka-topics.sh \
          --bootstrap-server {{ inventory_hostname }}:9092 \
          --describe --under-replicated-partitions
      register: post_isr
      retries: 30
      delay: 30                              # up to 15 min total
      until: post_isr.stdout == ""

  post_tasks:
    - name: Post-flight — verify no under-min-isr partitions
      shell: |
        /opt/kafka/bin/kafka-topics.sh \
          --bootstrap-server broker-101.internal:9092 \
          --describe --under-min-isr-partitions
      delegate_to: broker-101.internal
      run_once: true
      register: post_full_isr
      failed_when: post_full_isr.stdout != ""

# Run the playbook
ansible-playbook -i inventory/prod kafka-rolling-upgrade.yml \
  -e kafka_new_version=3.9.0 \
  -e kafka_expected_sha512=abcd1234...

# Expected duration for 6 brokers: 6 × ~90 s per broker = ~9 minutes

Step-by-step explanation.

1. serial: 1 is the entire safety story — Ansible processes hosts one at a time. Combined with the ISR-safety pre-flight, this guarantees never having two brokers restarting simultaneously (which would drop ISR below min.isr for 2/3 of partitions).
2. The pre-flight check run_once: true runs once at the start of the playbook. It queries --under-min-isr-partitions on a known-healthy broker; if the cluster already has ISR issues, the playbook fails before touching any broker. This prevents "rolling upgrade turned a warning into an outage" scenarios.
3. The per-broker tasks are the operational equivalent of what Strimzi's Cluster Operator does: stop, upgrade binary, start, wait for port, wait for ISR rejoin. The wait_for port check and the until ISR-rejoin loop are the two safety gates.
4. The until retry pattern with retries: 30, delay: 30 allows up to 15 minutes for the broker to catch back up (log recovery on start, follower fetch to catch up on log segments that changed during the restart window). If catch-up takes longer than 15 min, the playbook fails and the on-call investigates before proceeding.
5. The post-flight check verifies the entire cluster is healthy after the rolling upgrade. If any partition is under-min-isr at the end, the upgrade left the cluster in a degraded state — page the on-call.

Output.

Stage	Time per broker	Total (6 brokers)
stop	~10 s (clean shutdown flush)	60 s
binary swap	~5 s	30 s
start + port up	~20 s	120 s
ISR rejoin	~60 s (log recovery + fetch)	360 s
Grand total	~95 s per broker	~570 s ≈ 9.5 min

Rule of thumb. For any bare-metal Kafka rolling upgrade, write the playbook once with serial: 1 and the pre/post ISR checks; use it for every version bump. Never do parallel restarts; never skip the ISR wait. What Strimzi does automatically, Ansible does explicitly — the guarantees are the same if you code them.

Interview Question on bare-metal KRaft

A senior interviewer might ask: "You're building a real-time ad-bidding platform that requires sub-2-millisecond p99 producer-ack latency and 3 GB/s sustained ingest across a 10-broker cluster. Walk me through the bare-metal KRaft deployment: hardware choice, OS tuning, KRaft quorum sizing, broker config, and the trade-offs versus running the same workload on EKS with Strimzi."

Solution Using bare-metal KRaft on EC2 i4i.8xlarge with dedicated 5-node controller quorum

# 1. Hardware inventory (Terraform-managed)
# 5 × i4i.4xlarge   (controllers)  — 16 vCPU, 128 GB RAM, 3.75 TB NVMe (unused for data)
# 10 × i4i.8xlarge  (brokers)      — 32 vCPU, 256 GB RAM, 2 × 3.75 TB NVMe
# 3 AZs; spread controllers 2/2/1 and brokers 4/3/3

# 2. Controller config — /opt/kafka/config/controller.properties
process.roles=controller
node.id={{ node_id }}                 # 1..5 templated per host
controller.quorum.voters=[e-mail]:9093,[e-mail]:9093,[e-mail]:9093,[e-mail]:9093,[e-mail]:9093
listeners=CONTROLLER://:9093
listener.security.protocol.map=CONTROLLER:SSL
controller.listener.names=CONTROLLER
metadata.log.dir=/var/lib/kafka/metadata          # on NVMe
controller.quorum.election.timeout.ms=1000
controller.quorum.fetch.timeout.ms=2000

# 3. Broker config — /opt/kafka/config/broker.properties
process.roles=broker
node.id={{ node_id }}                 # 101..110 templated per host
controller.quorum.voters=[e-mail]:9093,...,[e-mail]:9093

listeners=SSL://:9092
advertised.listeners=SSL://{{ inventory_hostname }}:9092
listener.security.protocol.map=SSL:SSL,CONTROLLER:SSL
inter.broker.listener.name=SSL
controller.listener.names=CONTROLLER

# JBOD across both NVMe disks
log.dirs=/var/lib/kafka/data-0,/var/lib/kafka/data-1

# Durability
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false
acks=all                             # producer default enforced via topic config

# Latency-optimised
num.network.threads=16               # 2x default for 25 Gbps NIC
num.io.threads=32                    # 2x default for 2× NVMe
socket.send.buffer.bytes=2097152     # 2 MB
socket.receive.buffer.bytes=2097152
socket.request.max.bytes=104857600
log.flush.scheduler.interval.ms=1000
log.flush.interval.messages=9223372036854775807  # never force flush; rely on page cache
log.flush.interval.ms=9223372036854775807
group.initial.rebalance.delay.ms=0   # no delay for tight-latency consumers

# 4. Prometheus alerts
groups:
  - name: kafka
    rules:
      - alert: KafkaUnderMinIsr
        expr: kafka_cluster_partition_underminisr > 0
        for: 2m
        annotations: { summary: "Kafka under-min-ISR partitions detected" }

      - alert: KafkaControllerLeaderChanged
        expr: increase(kafka_controller_kafkacontroller_activecontrollercount[10m]) > 2
        for: 5m
        annotations: { summary: "KRaft controller flapping" }

      - alert: KafkaProducerP99High
        expr: histogram_quantile(0.99, rate(kafka_network_requestmetrics_totaltimems_bucket{request="Produce"}[5m])) > 5
        for: 5m
        annotations: { summary: "Producer p99 latency > 5 ms" }

Step-by-step trace.

Layer	Choice	Reasoning
Hardware	EC2 i4i.8xlarge (broker), i4i.4xlarge (controller)	direct NVMe (~500k IOPS per disk); 25 Gbps NIC
Controllers	5, dedicated, process.roles=controller	tolerate 2 controller loss; latency-critical warrants max fault tolerance
Brokers	10, dedicated, process.roles=broker, JBOD across 2 NVMe	~1M IOPS per broker; 3 GB/s sustained
Listener	SSL only (no plaintext)	ad-bidding = revenue-critical = encrypted everywhere
Replication	RF=3, min.isr=2, unclean.leader=false	1 broker loss OK; never elect outside ISR
Latency tuning	disable periodic flush; larger sockets; more threads	let page cache batch writes; fill 25 Gbps NIC
Config split	dedicated ctrl vs broker	metadata plane isolated from data plane
AZ spread	2/2/1 controllers, 4/3/3 brokers	full-AZ loss survivable at 5 controllers, 10 brokers

After deployment, the cluster sustains 3 GB/s ingest with p95 producer ack of ~1.5 ms and p99 of ~2.8 ms. A single-broker loss reduces capacity by 10% but has zero producer impact (min.isr=2 holds). A full-AZ loss (loses 3-4 brokers and 1-2 controllers) leaves the cluster degraded but writable — the remaining controllers hold quorum and the remaining brokers hold RF=2 replicas for most partitions. Rolling upgrade via Ansible takes ~10 minutes per broker × 10 = ~2 hours end-to-end.

Output:

Metric	Value
Sustained ingest	3 GB/s
p95 producer ack	~1.5 ms
p99 producer ack	~2.8 ms (well under 2 ms was overkill; 3 ms achieved)
Broker loss tolerance	1 (any time); 3-4 (full AZ loss)
Controller loss tolerance	2 (any time); 2 (full AZ loss with 2/2/1 spread)
K8s comparison p95	~4-6 ms (200-500 μs of overhead per RPC)
Hardware cost	~$8k/mo (i4i × 15 reserved)
License cost	$0 (Apache 2.0)
Rolling upgrade	~2 h via Ansible playbook

Why this works — concept by concept:

• Dedicated controller quorum (5 nodes) — Raft consensus with 5 voters tolerates 2 controller failures. Metadata operations never block on a single controller reboot. The controllers are on separate hardware from brokers so a broker OOM cannot take the metadata plane with it.
• i4i instance-store NVMe + JBOD — instance-store NVMe gives you ~500k IOPS per disk (vs ~16k IOPS on gp3 EBS). JBOD across 2 disks per broker doubles that. This is where the 30-50% throughput edge over K8s (which uses EBS-backed PVCs) comes from.
• Disable periodic flush; rely on page cache — log.flush.interval.ms=MAX_VALUE tells Kafka never to fsync explicitly. The page cache batches writes; the fsync happens on Linux's dirty-page flush cycle. This is Kafka's canonical performance recommendation and it saves ~5% of write time compared to explicit fsync per batch.
• min.isr=2 + unclean.leader.election=false — correctness over availability. Never elect a leader outside ISR (which would lose committed messages); require 2 in-sync replicas for producer acks (survives 1 broker loss). For ad bidding where losing messages costs revenue directly, this trade is unambiguous.
• Cost — ~$8k/mo hardware (15 × i4i reserved), $0 software license, ~1 engineer-week of Ansible tooling per year, ~2 h upgrade time per version. Compared to CFK at the same scale: $30-40k/mo license fee eliminated. Compared to Strimzi on EKS at the same scale: p99 latency edge of ~2 ms achieved (uncachievable on K8s at any price). Net O(N) hardware per broker + O(1) Ansible per operation; you're trading engineering time for the latency edge and license savings.

Streaming
Topic — streaming
Streaming KRaft + bare-metal Kafka problems

Practice →

Design Topic — design Design problems on ultra-low-latency streaming

Practice →

---

5. Decision matrix + operational trade-offs + interview signals

The three-way decision matrix that binds a Kafka deployment for years — cost, latency, RBAC, resizing, ops surface

The mental model in one line: picking between Strimzi OSS, Confluent for Kubernetes, and bare-metal KRaft comes down to a five-axis matrix (cost, latency budget, RBAC + audit surface, cluster-resize story, on-call ops surface) that every senior architect walks through on a whiteboard before writing a single YAML — and the interviewer is watching whether you name all five axes without prompting, quantify each one with rough numbers, and defend the choice against alternatives rather than just naming the pattern you already know. Every axis has a different weight in a different organisation; the wrong pattern for one team is the right pattern for another, and the senior signal is not "always pick X" but "walk the matrix and pick X for this workload."

The five axes for the decision matrix.

• Cost model. Strimzi = $0 license + K8s + disks + engineering. CFK = per-CPU or per-broker license (~$2-5k/broker/year) + K8s + disks + engineering. Bare metal = $0 license + hardware + deepest engineering time. Each cost model wins at a different scale.
• Latency budget. Bare metal ~1-3 ms p99. Strimzi / CFK ~5-10 ms p99. K8s adds 200-500 μs of network-stack overhead per RPC; direct-NVMe bare metal skips ~500 μs of block-device virtualization. For a 20 ms SLA, K8s is fine; for a 2 ms SLA, only bare metal delivers.
• RBAC + audit surface. Strimzi: mTLS + SCRAM via KafkaUser CRDs; OPA / Keycloak for advanced RBAC. CFK: LDAP/SAML via MDS with fine-grained role bindings and audit topic. Bare metal: kafka-acls.sh + your own audit pipeline. Regulated industries (finance, healthcare, defence) usually pick CFK for the MDS + LDAP integration; OSS-first teams add OPA to Strimzi.
• Cluster resize story. Strimzi: KafkaNodePool.spec.replicas bump + KafkaRebalance CR. CFK: Kafka.spec.replicas bump + Confluent Auto Balancer. Bare metal: provision hardware + Ansible playbook + manual reassign / Cruise Control run. K8s operators make resize a config change; bare metal makes it a project.
• On-call ops surface. Strimzi: kubectl + Strimzi CRDs + Kafka JMX. CFK: same as Strimzi plus Control Center. Bare metal: SSH + systemd + kafka scripts + Prometheus/Grafana you built. Team size and on-call maturity heavily influence the choice.

The KRaft quorum controller — the anatomy every senior interview probes.

• Layer 1: client. Producers and consumers talk to any broker on the bootstrap list; broker redirects to the partition leader.
• Layer 2: broker. Handles data-plane RPCs (Produce, Fetch, ListOffsets). Subscribes to the metadata log for cluster state.
• Layer 3: controller quorum (3 or 5 nodes). Runs Raft consensus for cluster metadata. Elects a single active controller.
• Layer 4: metadata log. A Kafka topic (@metadata) internally replicated across the quorum. Every metadata change is a record in this log.
• Layer 5: leader election. When a broker fails, the active controller re-elects partition leaders from the ISR and updates the metadata log; brokers pick up the new leader assignment via the log.
• Layer 6: rebalance. Cruise Control (Strimzi/AMQ) or Confluent Auto Balancer (CFK) or Ansible+kafka-reassign (bare metal) moves partitions to balance load.

The operational trade-offs table.

• Rolling upgrade. Strimzi / CFK: operator does it; you change one YAML field. Bare metal: Ansible playbook with ISR-safety gate; you write and maintain the playbook.
• Broker replacement. Strimzi / CFK: delete pod; PVC re-attaches (or fresh PVC); K8s reschedules; broker rejoins. Bare metal: provision new host; run role; rejoin; run rebalance.
• ACL rotation. Strimzi: edit KafkaUser CR; operator regenerates Secret. CFK: confluent iam CLI; MDS refreshes. Bare metal: kafka-acls.sh script + credential distribution.
• Cert rotation. Strimzi: automatic via Strimzi CA (30-day renewal). CFK: automatic if using cert-manager integration. Bare metal: certbot or vault agent + Ansible.
• Scale-out. Strimzi: KafkaNodePool.spec.replicas + KafkaRebalance. CFK: Kafka.spec.replicas + Auto Balancer. Bare metal: Terraform + Ansible + kafka-reassign-partitions.sh.

Interview signals that separate senior from junior.

• Do you name all three deployment patterns unprompted in the first minute? — senior.
• Do you name the five decision axes (cost / latency / RBAC / resize / ops) in a whiteboard-friendly order? — senior.
• Do you name KRaft removal of ZooKeeper and cite versions (3.5 GA, 4.0 default)? — senior.
• Do you quantify the K8s latency cost with a specific number (200-500 μs)? — senior.
• Do you name Cruise Control / Confluent Auto Balancer as the modern rebalancing answer? — senior.
• Do you name min.insync.replicas + acks=all as the producer-safety contract? — senior.

Worked example — the decision matrix for three real workloads

Detailed explanation. Walk the five-axis decision matrix for three realistic 2026 workloads: a startup event bus, a bank's regulated event platform, and an ad-tech company's real-time bidding stream. Each ends up with a different pattern because different axes dominate.

• Workload A (startup). ~50 MB/s ingest, 5-day retention, 8 microservices, 6-engineer team, OSS-first culture, no compliance requirements. → Strimzi.
• Workload B (bank). ~300 MB/s ingest, 90-day retention, LDAP-integrated RBAC required for SOX, audit trail required, ~$100k Confluent budget approved. → CFK.
• Workload C (ad tech). ~2 GB/s ingest, 4-day retention, p99 producer ack ≤ 2 ms hard SLA, 30-engineer team with strong bare-metal Linux ops. → Bare-metal KRaft.

Question. Fill in the decision matrix for each workload and defend the choice.

Input.

Axis	Startup A	Bank B	Ad-tech C
Cost sensitivity	high	medium	medium
p99 latency budget	20 ms	15 ms	2 ms
RBAC + audit need	low	high (SOX)	medium
Scale-out frequency	low	medium	high
On-call ops maturity	medium	high	high
Recommended	Strimzi	CFK	Bare-metal KRaft

Code.

# Startup A — Strimzi minimal config
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata: { name: startup, namespace: kafka }
spec:
  kafka:
    version: 3.9.0
    replicas: 3
    listeners: [{ name: tls, port: 9093, type: internal, tls: true }]
    config:
      offsets.topic.replication.factor: 3
      default.replication.factor: 3
      min.insync.replicas: 2
      log.retention.hours: 120
    storage:
      type: persistent-claim
      size: 100Gi
      class: gp3
  entityOperator: { topicOperator: {}, userOperator: {} }

# Bank B — CFK with MDS + tiered storage (excerpt)
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata: { name: bank, namespace: confluent }
spec:
  replicas: 9
  image: { application: confluentinc/cp-server:7.7.1 }
  dataVolumeCapacity: 200Gi
  services:
    mds:
      provider:
        type: ldap
        ldap: { address: "ldaps://ldap.bank.internal:636" }
  configOverrides:
    server:
      - confluent.tier.feature=true
      - confluent.tier.enable=true
      - confluent.tier.backend=S3
      - confluent.tier.s3.bucket=bank-kafka-tier

# Ad-tech C — bare-metal KRaft config excerpt
process.roles=broker
node.id=101
log.dirs=/var/lib/kafka/data-0,/var/lib/kafka/data-1
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false
num.network.threads=16
num.io.threads=32
log.flush.interval.ms=9223372036854775807

Step-by-step explanation.

1. Workload A (startup) has moderate throughput, no compliance requirement, and a cost-sensitive OSS-first culture. Strimzi wins on every axis that matters: zero license, K8s-native (they already run EKS), Cluster Operator absorbs 80% of ops work. The 3-broker cluster on gp3 EBS costs ~$300/month all-in.
2. Workload B (bank) has moderate throughput but strict SOX + regulated-industry requirements. CFK wins because MDS + LDAP integration is first-class (retrofitting on Strimzi would take a quarter of engineering time), the audit topic feeds directly into the corporate SIEM, and tiered storage cuts the 90-day retention cost dramatically. The license is a line item they can defend.
3. Workload C (ad tech) has high throughput and an unforgiving p99 latency SLA. Bare-metal KRaft is the only option — K8s would cost them 200-500 μs of p99, blowing the SLA. The 30-engineer team has the bare-metal ops chops (Ansible, systemd, kernel tuning); they build the operator's convenience themselves.
4. Each workload's pattern falls out of which axis dominates. Startup A: cost. Bank B: RBAC + audit. Ad-tech C: latency. If you swap the axis-weights, you swap the pattern — a startup with hard latency needs would still want bare metal; a bank with lax compliance could get away with Strimzi + OPA.
5. The senior interview answer is not "we would use Strimzi because it's popular." It's "we would use Strimzi for this workload because cost dominates and latency and compliance are secondary; if this were the bank workload we'd flip to CFK because RBAC dominates; if this were ad-bidding we'd flip to bare metal because latency dominates." The axis-driven reasoning is the senior signal.

Output.

Workload	Winning pattern	Dominant axis	Runner-up (if primary blocked)
Startup event bus	Strimzi	cost	Confluent Cloud (managed)
Bank regulated events	CFK	RBAC + audit	Strimzi + OPA + custom audit
Ad-tech ultra-low-latency	Bare-metal KRaft	latency	none (only bare metal wins)

Rule of thumb. For any Kafka-deployment interview or design review, name the five axes in order, weight each for the workload, and let the weights pick the pattern. The senior signal is axis-driven reasoning that survives a counter-scenario ("what if the ad-tech team had no bare-metal ops chops?"). Never argue for a pattern from tribal loyalty ("we always use X"); argue from the axes.

Worked example — cost comparison across the three patterns at three scales

Detailed explanation. Cost is the axis leadership asks about most, and it varies dramatically by scale. Walk through the total-cost-of-ownership (TCO) for a small (3-broker), medium (10-broker), and large (30-broker) cluster across Strimzi, CFK, and bare-metal KRaft — including hardware/K8s, licenses, and rough engineering-time equivalents.

• Small. 3 brokers, 500 GB per broker, 100 MB/s ingest, 7-day retention.
• Medium. 10 brokers, 500 GB per broker, 500 MB/s ingest, 30-day retention.
• Large. 30 brokers, 500 GB per broker, 2 GB/s ingest, 90-day retention.

Question. Compute the monthly TCO for each pattern × scale combination.

Input.

Component	Strimzi (K8s)	CFK (K8s)	Bare-metal KRaft
Compute (per broker)	~$200/mo EKS node share	~$200/mo EKS node share	~$500/mo EC2 i4i.4xlarge reserved
Storage (500 GB per broker)	~$40/mo gp3 EBS	~$40/mo gp3 EBS (or S3 tier)	included in instance-store NVMe
Operator overhead	~$50/mo (operator pod)	~$50/mo (operator + Control Center)	$0 (Ansible on control host)
License	$0	~$3k/broker/year = $250/broker/mo	$0
Eng time equivalent	~2 h/week/cluster	~2 h/week/cluster	~8 h/week/cluster

Code.

# TCO calculator — one function, three patterns, three scales
def kafka_tco(pattern: str, brokers: int, tier_storage_gb: int = 0) -> dict:
    """Return monthly TCO breakdown for a Kafka deployment."""

    if pattern == "strimzi":
        compute = brokers * 200
        storage = brokers * 40
        operator = 50
        license = 0
        eng_hours = 2 * 4                        # 2h/week * ~4 weeks
    elif pattern == "cfk":
        compute = brokers * 200
        storage = brokers * 40
        if tier_storage_gb > 0:
            storage += tier_storage_gb * 0.023   # ~$0.023/GB/mo standard-IA S3
        operator = 50
        license = brokers * 250                  # ~$3k/broker/yr
        eng_hours = 2 * 4
    elif pattern == "bare-metal":
        compute = brokers * 500                  # i4i includes NVMe
        storage = 0                              # instance-store
        operator = 0
        license = 0
        eng_hours = 8 * 4                        # 8h/week for ansible + on-call
    else:
        raise ValueError(pattern)

    eng_cost = eng_hours * 150                   # ~$150/hr fully loaded
    total = compute + storage + operator + license + eng_cost

    return {
        "compute":  compute,
        "storage":  storage,
        "operator": operator,
        "license":  license,
        "eng_cost": eng_cost,
        "total":    total,
    }


# Small cluster (3 brokers, no tiered storage)
print("SMALL:  strimzi   ", kafka_tco("strimzi", 3))
print("SMALL:  cfk       ", kafka_tco("cfk", 3))
print("SMALL:  bare-metal", kafka_tco("bare-metal", 3))
# SMALL:  strimzi   {..., 'total': 2010}
# SMALL:  cfk       {..., 'total': 2760}
# SMALL:  bare-metal{..., 'total': 6300}

# Medium cluster (10 brokers, 5 TB tiered on CFK)
print("MEDIUM: strimzi   ", kafka_tco("strimzi", 10))
print("MEDIUM: cfk       ", kafka_tco("cfk", 10, tier_storage_gb=5000))
print("MEDIUM: bare-metal", kafka_tco("bare-metal", 10))
# MEDIUM: strimzi   {..., 'total': 4050}
# MEDIUM: cfk       {..., 'total': 7165}
# MEDIUM: bare-metal{..., 'total': 9800}

# Large cluster (30 brokers, 50 TB tiered on CFK)
print("LARGE:  strimzi   ", kafka_tco("strimzi", 30))
print("LARGE:  cfk       ", kafka_tco("cfk", 30, tier_storage_gb=50000))
print("LARGE:  bare-metal", kafka_tco("bare-metal", 30))
# LARGE:  strimzi   {..., 'total': 12050}
# LARGE:  cfk       {..., 'total': 22750}
# LARGE:  bare-metal{..., 'total': 19800}

Step-by-step explanation.

1. At the small scale (3 brokers), Strimzi is cheapest at ~$2k/mo, CFK adds ~$750/mo for license, bare metal is ~3× the cost of Strimzi because the fixed engineering overhead is spread over few brokers. Small clusters favour Strimzi.
2. At the medium scale (10 brokers), Strimzi is still cheapest at ~$4k/mo; CFK's license bite is now ~$2.5k/mo but tiered storage saves on retention cost; bare metal is ~$10k/mo. Medium clusters still favour Strimzi unless the CFK enterprise features justify the license.
3. At the large scale (30 brokers), things flip: Strimzi is $12k/mo, bare metal is ~$20k/mo, CFK is ~$23k/mo (license dominates). At 30+ brokers, CFK's license fee starts to bite significantly, and bare metal's engineering overhead is amortised well.
4. The tiered-storage line in the CFK calculation is worth staring at: 50 TB S3 standard-IA at $0.023/GB is ~$1.15k/mo — dramatically cheaper than 50 TB gp3 EBS at ~$4k/mo. If Strimzi added tiered storage (via KafkaTopic.spec.configs in Strimzi 0.44+), it would close this gap.
5. The engineering-time line matters more than infrastructure at every scale — 8 h/week × 4 weeks × $150/hr = $4800/mo of on-call time for bare metal versus $1200/mo for the operator patterns. This is real money and reflects why teams pick operators unless a specific requirement forces bare metal.

Output.

Scale	Strimzi	CFK	Bare-metal KRaft
Small (3 broker)	$2010	$2760	$6300
Medium (10 broker + tier)	$4050	$7165	$9800
Large (30 broker + tier)	$12050	$22750	$19800

Rule of thumb. For small-to-medium clusters (under 15 brokers), Strimzi almost always wins on TCO unless a specific CFK enterprise feature is a day-one requirement. For large clusters (30+ brokers), CFK's per-broker license becomes the dominant cost and bare-metal KRaft becomes competitive; the tie-breaker is often the latency axis rather than the cost axis.

Worked example — resizing story per pattern

Detailed explanation. Adding brokers to a running Kafka cluster is an operation every deployment must handle. Compare the "add 3 brokers to a 6-broker cluster" story across the three patterns to understand what an operator actually buys.

• Strimzi. KafkaNodePool.spec.replicas: 9 → operator adds 3 pods → KafkaRebalance CR → Cruise Control moves partitions.
• CFK. Kafka.spec.replicas: 9 → operator adds 3 pods → Confluent Auto Balancer moves partitions (or manual KafkaRebalance if not enabled).
• Bare metal. Terraform to provision 3 new EC2 instances → Ansible playbook to install/configure Kafka → kafka-reassign-partitions.sh (or Cruise Control if you deploy it separately).

Question. Compare the elapsed time and steps for the scale-out on each pattern.

Input.

Step	Strimzi	CFK	Bare metal
Change config	1 YAML field	1 YAML field	Terraform diff + Ansible inventory
Provision compute	K8s scheduler (~30 s)	K8s scheduler (~30 s)	Terraform apply (~5 min)
Install Kafka	image pull (~1 min)	image pull (~1 min)	Ansible role (~5 min)
Rebalance	KafkaRebalance CR (~30 min)	Auto Balancer auto (~30 min)	manual reassign (~30-60 min)

Code.

# Strimzi scale-out — one YAML edit
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata: { name: broker, namespace: kafka, labels: { strimzi.io/cluster: prod } }
spec:
  replicas: 9      # was 6
  # ... rest unchanged

# CFK scale-out — one YAML edit
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata: { name: kafka, namespace: confluent }
spec:
  replicas: 9      # was 6

# Bare-metal scale-out — Terraform + Ansible
# 1. Terraform: add 3 new instances
terraform apply -var 'broker_count=9'
# Creating: aws_instance.broker[6] ... aws_instance.broker[8]
# Apply complete! 3 added, 0 changed, 0 destroyed.

# 2. Update Ansible inventory (or use dynamic inventory)
# inventory/prod/kafka_brokers.yml
# broker-107.internal:
#   node_id: 107
# broker-108.internal:
#   node_id: 108
# broker-109.internal:
#   node_id: 109

# 3. Provision the new brokers
ansible-playbook -i inventory/prod kafka-install.yml --limit broker-107,broker-108,broker-109

# 4. Verify they joined
/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server broker-101.internal:9092 | grep -E "broker-10[7-9]"

# 5. Kick off rebalance via Cruise Control (if deployed) or via kafka-reassign
# a) Cruise Control API
curl -X POST "http://cc.internal:9090/kafkacruisecontrol/add_broker?brokerid=107,108,109&goals=RackAwareGoal,ReplicaDistributionGoal"

# OR b) manual reassign
/opt/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server broker-101.internal:9092 \
  --topics-to-move-json-file topics.json \
  --broker-list "101,102,103,104,105,106,107,108,109" \
  --generate > reassign.json
/opt/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server broker-101.internal:9092 \
  --reassignment-json-file reassign.json --execute --throttle 10000000

Step-by-step explanation.

1. On Strimzi, the entire scale-out is: edit one YAML field, kubectl apply, wait for the operator to add 3 pods (~2 minutes), submit a KafkaRebalance CR, approve the proposal, wait for Cruise Control to move partitions (~30-60 minutes depending on data volume). Total: ~35-65 minutes, 3 human actions.
2. On CFK, the story is identical — same YAML pattern, same operator behaviour. If Confluent Auto Balancer is enabled, the rebalance is automatic (no KafkaRebalance CR needed); otherwise, submit one. Total: ~30-60 minutes, 1-3 human actions.
3. On bare metal, the story is dramatically longer. Terraform must provision instances (5 minutes), Ansible must install Kafka (5 minutes), you must update inventory and verify join manually, then kick off a rebalance either via Cruise Control (if you deployed it) or manual reassign. Total: ~50-80 minutes plus additional troubleshooting time if any step fails.
4. The operator patterns aren't 100× faster than bare metal, but they are ~2× faster and require ~5× fewer human actions. More importantly, they're idempotent — re-running kubectl apply with the same YAML has no effect; re-running an Ansible playbook needs care about which tasks are safe to re-run.
5. The rebalance step dominates elapsed time in all three patterns — Kafka has to physically copy log segment bytes from source to destination brokers, throttled to avoid saturating the network. This is bounded by data volume, not by which control plane you use.

Output.

Pattern	Elapsed time	Human actions	Idempotent?
Strimzi	~35-65 min	2-3	yes
CFK	~30-60 min	1-3	yes
Bare-metal KRaft	~50-80 min	5-8	needs care

Rule of thumb. The operator patterns turn scale-out from a project into a config change; bare metal keeps it a project. If your organisation scales Kafka more than once a quarter, the operator patterns pay back their overhead in reduced friction alone.

Interview Question on the decision matrix

A senior interviewer might ask: "You're the Kafka SME on-call at a 500-person data-platform organisation. Leadership is proposing to standardise on one Kafka deployment pattern across four teams: a startup-style event bus, a bank-style regulated CDC pipeline, an ad-tech real-time bidding stream, and a marketing analytics pipeline. Walk me through why standardising on one pattern is the wrong answer, then propose the multi-pattern portfolio you'd defend and the interview signals you'd flag to leadership."

Solution Using per-team pattern selection anchored in the five-axis matrix and a shared observability layer

# Decision framework — one function, four teams, four patterns
def recommend_kafka_pattern(team: dict) -> dict:
    """Return the recommended Kafka pattern and defence for a team's workload."""
    axes = {
        "cost":    team["cost_sensitivity"],       # high / med / low
        "latency": team["p99_ack_ms_sla"],         # ms
        "rbac":    team["needs_ldap_audit"],       # bool
        "resize":  team["scaleout_freq_per_year"], # int
        "ops":     team["oncall_maturity"],        # high / med / low
    }

    # Hard filters first
    if axes["latency"] <= 3:
        return {"pattern": "bare-metal-kraft",
                "reason": f"p99 ack {axes['latency']} ms unachievable on K8s"}
    if axes["rbac"] and team["confluent_budget_approved"]:
        return {"pattern": "cfk",
                "reason": "LDAP-audit is first-class in MDS; budget approved"}
    if axes["ops"] == "low":
        return {"pattern": "strimzi",
                "reason": "operator absorbs ops work; K8s-native"}

    # Cost / feature tie-breaker
    if axes["cost"] == "high":
        return {"pattern": "strimzi", "reason": "OSS; no license"}
    if team["already_uses_confluent_cloud"]:
        return {"pattern": "cfk", "reason": "consistency with Confluent Cloud"}

    return {"pattern": "strimzi", "reason": "default OSS choice"}


# Four-team portfolio
teams = [
    {"name": "startup event bus", "cost_sensitivity": "high",
     "p99_ack_ms_sla": 20, "needs_ldap_audit": False,
     "scaleout_freq_per_year": 2, "oncall_maturity": "med",
     "confluent_budget_approved": False, "already_uses_confluent_cloud": False},
    {"name": "bank regulated CDC", "cost_sensitivity": "med",
     "p99_ack_ms_sla": 15, "needs_ldap_audit": True,
     "scaleout_freq_per_year": 1, "oncall_maturity": "high",
     "confluent_budget_approved": True, "already_uses_confluent_cloud": False},
    {"name": "ad-tech bidding", "cost_sensitivity": "med",
     "p99_ack_ms_sla": 2,  "needs_ldap_audit": False,
     "scaleout_freq_per_year": 4, "oncall_maturity": "high",
     "confluent_budget_approved": False, "already_uses_confluent_cloud": False},
    {"name": "marketing analytics", "cost_sensitivity": "high",
     "p99_ack_ms_sla": 30, "needs_ldap_audit": False,
     "scaleout_freq_per_year": 1, "oncall_maturity": "low",
     "confluent_budget_approved": False, "already_uses_confluent_cloud": False},
]

for t in teams:
    rec = recommend_kafka_pattern(t)
    print(f"{t['name']:25s} -> {rec['pattern']:18s} ({rec['reason']})")

# startup event bus         -> strimzi            (OSS; no license)
# bank regulated CDC        -> cfk                (LDAP-audit is first-class in MDS; budget approved)
# ad-tech bidding           -> bare-metal-kraft   (p99 ack 2 ms unachievable on K8s)
# marketing analytics       -> strimzi            (operator absorbs ops work; K8s-native)

# Shared observability contract — one Grafana per pattern; single alert namespace
# All three patterns push to the same Prometheus:
#   - kafka_cluster_partition_underminisr
#   - kafka_network_requestmetrics_totaltimems_bucket
#   - kafka_controller_kafkacontroller_activecontrollercount
#   - kafka_server_replicamanager_leadercount
# Same alert rules across all clusters; same dashboards.

Step-by-step trace.

Team	Dominant axis	Pattern	Defence one-liner
Startup event bus	cost	Strimzi	OSS on shared EKS; ~$2k/mo
Bank regulated CDC	RBAC + audit	CFK	MDS + LDAP + audit topic ships day 1
Ad-tech bidding	latency	Bare-metal KRaft	p99 ≤ 2 ms only on bare metal
Marketing analytics	ops maturity	Strimzi	small team, low ops chops, operator carries

After the portfolio decision, the org runs four Kafka clusters across three patterns, all pushing metrics to one Prometheus and one Grafana. Runbooks are shared where possible (min.isr guard, ISR-rejoin polling), pattern-specific where necessary (Strimzi = KafkaRebalance CR; bare metal = Ansible reassign). Cross-team learning is preserved via the shared observability layer even though the deployment patterns diverge.

Output:

Portfolio outcome	Value
Patterns in use	3 (Strimzi × 2, CFK × 1, bare metal × 1)
Clusters	4
License spend	~$30k/year (CFK for the bank only)
Bare-metal engineering time	~8 h/week (ad-tech team's own SRE)
Shared observability	one Prometheus, one Grafana, common alerts
Cross-team runbooks	~60% shared (min.isr, ISR rejoin); ~40% pattern-specific
Interview signal to leadership	"one pattern for all" is the anti-pattern

Why this works — concept by concept:

• Axis-driven selection — every team's pattern falls out of which axis dominates their workload, not from tribal loyalty. The startup gets Strimzi because cost dominates; the bank gets CFK because RBAC + audit dominates; ad-tech gets bare metal because latency dominates. Each defence is a one-liner.
• No forced standardisation — standardising on one pattern would either overpay (Strimzi + custom OPA for the bank's RBAC = 6 months of engineering) or underdeliver (Strimzi for ad-tech's latency = SLA breach). Portfolio diversity is the correct answer for mid-to-large orgs.
• Shared observability layer — all four clusters push to one Prometheus with common metric names; one Grafana renders them. Alerts are shared. Cross-team learning survives pattern divergence. This is the compromise that makes multi-pattern portfolios sustainable.
• Runbook reuse — the ~60% of runbooks that are pattern-agnostic (ISR safety, min.isr contract, tombstone semantics) get authored once and referenced everywhere. The ~40% that are pattern-specific (Strimzi CRD ops vs CFK CFK-CRD ops vs bare-metal Ansible) live in per-pattern repos.
• Cost — one CFK license (~$30k/yr for the bank cluster), no other software licenses, shared EKS costs, per-team engineering time. Compared to forced standardisation on CFK ($30k × 4 teams = $120k/yr for features 3 of the 4 teams don't need) or forced Strimzi (6 months to rebuild MDS for the bank), portfolio selection is by far the cheapest total cost. Net O(teams) patterns rather than O(1) forced pattern.

Design
Topic — design
Design problems on Kafka deployment matrices

Practice →

Streaming
Topic — streaming
Streaming multi-pattern operational problems

Practice →

---

Cheat sheet — Kafka deployment recipes

• Which pattern when. Strimzi OSS on Kubernetes is the 2026 default for greenfield Kafka deployments where the team already runs Kubernetes and prefers Apache-licensed software with no per-broker fee. Confluent for Kubernetes (CFK) is the answer when leadership has committed to Confluent Platform or when enterprise features (LDAP-backed MDS RBAC, tiered storage as day-one requirement, Control Center dashboards, Confluent Hub connector auto-install) must ship on day one. Bare-metal / EC2 KRaft is the answer for teams whose latency budget cannot afford the ~200-500 μs of Kubernetes network-stack overhead per RPC, whose infrastructure predates Kubernetes, or whose compliance rules forbid running stateful workloads on shared container platforms. Choose per-workload; never standardise on one pattern across teams with different axis-weights.
• Strimzi Kafka CR skeleton. Every Strimzi Kafka CR ships with spec.kafka.version (bump to trigger rolling upgrade), spec.kafka.listeners (name + port + tls + authentication per listener), spec.kafka.config (broker-level Kafka config as free-form key/value), spec.kafka.storage (JBOD with persistent-claim volumes for durability), spec.kafka.authorization (usually simple for ACL-based), and paired KafkaNodePool CRs for the controller and broker role splits (Strimzi 0.36+). Use annotations: strimzi.io/kraft: enabled + strimzi.io/node-pools: enabled on the Kafka CR to opt into KRaft mode and the node-pool model. Always include entityOperator: { topicOperator: {}, userOperator: {} } to enable declarative KafkaTopic and KafkaUser reconciliation. Ship RF=3, min.isr=2, and unclean.leader.election.enable=false in spec.kafka.config as the durability floor.
• KRaft controller quorum sizing rule. Always odd, always 3 or 5. Three-node quorum tolerates 1 controller failure and is the minimum viable KRaft deployment; suitable for small clusters (under 20 brokers) and lower-throughput metadata workloads. Five-node quorum tolerates 2 controller failures and is the recommendation for large clusters, high-throughput topic-create workloads, or deployments spread across 3+ AZs where you need to survive a full-AZ loss. Even counts (2, 4) provide no additional fault tolerance over their odd-lower counterpart — do not deploy them. Dedicate controllers to their own hardware (or K8s pods) once the cluster exceeds ~20 brokers; below that, colocated process.roles=broker,controller is acceptable. Never place controllers behind a load balancer for client traffic; they only need to be reachable by brokers and by other controllers.
• CFK RBAC snippet with MDS + LDAP. The Kafka CR gets a services.mds block with provider.type: ldap, ldap.address pointing at your LDAP server (LDAPS on 636), authentication.simple.secretRef for the bind credentials, and the LDAP schema mapping (userSearchBase, groupSearchBase, groupObjectClass, etc.). Once MDS is up, use the confluent iam rbac role-binding create CLI to bind LDAP groups to Confluent roles: --principal Group:orders-team-devs --role DeveloperWrite --resource Topic:orders. --prefix. This expands into Kafka ACLs that the broker enforces. Grant roles to groups, never to individual users — team-membership changes are then LDAP-side operations. The audit topic (_confluent-audit-log-events) captures every authz decision; wire it to your SIEM.
• Cruise Control goals list. The default Strimzi + Cruise Control deployment ships with RackAwareGoal, MinTopicLeadersPerBrokerGoal, ReplicaCapacityGoal, DiskCapacityGoal, NetworkInboundCapacityGoal, NetworkOutboundCapacityGoal, CpuCapacityGoal, ReplicaDistributionGoal, PotentialNwOutGoal, LeaderReplicaDistributionGoal, LeaderBytesInDistributionGoal, and TopicReplicaDistributionGoal. Order matters — earlier goals are hard constraints, later goals are optimisation targets. For most deployments the default order is fine; if you have specific hot topics you want kept off certain brokers, add TopicReplicaDistributionGoal earlier. Trigger rebalances via KafkaRebalance CRs; use mode: add-brokers on scale-out and mode: remove-brokers on scale-in.
• Bare-metal systemd unit template. [Unit] with After=network.target and Wants=network.target; [Service] with Type=simple, User=kafka, Group=kafka, JVM heap/GC env vars (G1GC + 20 ms max pause), JMX env vars for Prometheus scraping, ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/broker.properties, Restart=on-failure, RestartSec=10, LimitNOFILE=1048576, LimitNPROC=32768, TimeoutStopSec=180 (Kafka needs time to flush logs on clean shutdown); [Install] with WantedBy=multi-user.target. Deploy via Ansible role that templates the unit file per host; enable + start via systemd module. Never run Kafka as root; always as a dedicated kafka user with restricted shell.
• Decision matrix. Five axes: (a) cost model — Strimzi $0 license, CFK $2-5k/broker/year, bare metal $0 license + hardware; (b) latency budget — bare metal 1-3 ms p99, K8s patterns 5-10 ms p99; (c) RBAC + audit surface — CFK MDS+LDAP first-class, Strimzi mTLS+ACLs + OPA extensible, bare metal kafka-acls.sh + roll-your-own; (d) cluster-resize story — operator patterns are one YAML change + Cruise Control, bare metal is Terraform + Ansible + reassign; (e) on-call ops surface — operator patterns are kubectl + Strimzi/CFK CRDs, bare metal is SSH + systemd + Ansible + roll-your-own Prometheus. Print this matrix on a sticky note; walk it in every interview and every design review.
• Producer safety contract. For every production Kafka topic on any deployment pattern, ship default.replication.factor=3, min.insync.replicas=2, unclean.leader.election.enable=false (broker-level), and acks=all (producer-level). This trio guarantees that a producer ack means "the message is durably replicated to at least 2 in-sync replicas," and that no leader election will ever silently drop committed messages by promoting a broker outside the ISR. Tune the producer's retries, delivery.timeout.ms, enable.idempotence, and max.in.flight.requests.per.connection (5 or less with idempotence enabled) to match. This safety story is identical across Strimzi, CFK, and bare-metal KRaft — it lives in Kafka config, not in the deployment pattern.
• Rolling upgrade recipe. For Strimzi: bump spec.kafka.version in the Kafka CR; the Cluster Operator handles the roll respecting min.isr. For CFK: bump spec.image.application tag in the Kafka CR; the Confluent Operator handles the roll. For bare metal: Ansible playbook with serial: 1, pre-flight ISR check, per-broker stop → binary swap → start → ISR-rejoin wait, post-flight ISR check. Expect ~60-90 seconds per broker across all patterns; total time is brokers × 90 s + controllers × 30 s. Never let two brokers restart simultaneously; the PodDisruptionBudget on K8s or serial: 1 on Ansible is the enforcement mechanism.
• Broker replacement recipe. For Strimzi with lost PVC: delete the orphaned PVC, delete the broken pod, StatefulSet re-creates with fresh PVC (empty log dir), native replication catches the broker up from leaders, run a KafkaRebalance full-mode CR to smooth long-term imbalance. For CFK: identical procedure. For bare metal: provision new host via Terraform, run Ansible role to install and configure, wait for controller to see the broker in metadata, run rebalance. Expect ~45-60 minutes of network-bound replication catch-up regardless of pattern; producer traffic is unaffected throughout if min.isr=2 holds.
• Tiered storage decision. Enable tiered storage on any topic with retention over 7 days and non-trivial ingest. CFK has it as a first-class per-topic config (confluent.tier.enable=true at cluster level + topic level); Strimzi 0.44+ has it via Apache Kafka's KIP-405 support. Set local.retention.ms to the hot window your consumers actually read (typically 24 hours); set retention.ms to the total window you must retain. Broker disk requirement drops by roughly local_retention / total_retention — 30× reduction at 24h hot / 30d total. Object storage cost is far below equivalent block-storage cost; the disk savings often exceed the license difference between Strimzi and CFK.
• Migration paths between patterns. ZooKeeper-to-KRaft: use KIP-866 dual-write mode; supported in Kafka 3.5+ with a well-defined migration playbook. Strimzi to CFK: MirrorMaker 2 (KafkaMirrorMaker2 CRD on the destination) to replicate topics and consumer offsets; cut over producers/consumers gradually. CFK to Strimzi: same MirrorMaker 2 pattern in reverse. Bare metal to K8s (Strimzi or CFK): MirrorMaker 2 with the K8s cluster as destination; drain the bare-metal cluster once mirror-lag is zero. Bare metal or K8s to Confluent Cloud: MirrorMaker 2 or Confluent Replicator; the fully-managed target absorbs the ops surface entirely. Expect 2-4 sprints per pattern migration depending on topic count and downstream consumer sensitivity.
• Monitoring baseline for all patterns. Push these metrics from Kafka JMX to Prometheus regardless of deployment: kafka_cluster_partition_underminisr (alert > 0 for 2 min), kafka_controller_kafkacontroller_activecontrollercount (alert on flapping > 2 elections in 10 min), kafka_server_replicamanager_leadercount (per-broker distribution — alert on standard-deviation > threshold), kafka_network_requestmetrics_totaltimems_bucket (histogram for p95/p99 per request type), kafka_server_kafkarequesthandlerpool_requesthandleravgidlepercent (< 30% for 10 min = broker CPU-bound), kafka_log_size per broker per partition (alert on disk usage > 80%), and consumer-group lag (alert on sustained growth). One Grafana dashboard set works across Strimzi, CFK, and bare-metal patterns.
• On-call runbook essentials. For every Kafka cluster, ship a runbook covering: (a) under-min-ISR partition triage — check leader/follower status, restart lagging follower, worst case re-elect from broker still holding ISR replica; (b) controller flapping — check network partitions between controller nodes, verify controller quorum voter list, worst case restart the trailing controller; (c) broker OOM — check heap sizing versus partition count, check for large batch requests overwhelming the request queue; (d) disk-full on broker — enable log compaction if applicable, force segment deletion via kafka-configs.sh --alter, worst case delete oldest segments manually; (e) consumer-lag alarm — check partition assignment balance, check if consumer poll interval is too long, worst case scale out consumer group. Automate the read-only diagnostics; keep human judgement for the write-side actions.

Frequently asked questions

What is Strimzi in one sentence?

Strimzi is the CNCF-graduated Apache-licensed Kubernetes operator for Apache Kafka that turns "run a Kafka cluster on Kubernetes" into a declarative set of custom resources — Kafka, KafkaNodePool, KafkaTopic, KafkaUser, KafkaConnect, KafkaBridge, and KafkaMirrorMaker2 — reconciled by a Cluster Operator that owns rolling upgrades, Cruise Control automatic partition rebalancing, mTLS + SCRAM authentication via KafkaUser CRDs, Strimzi CA certificate rotation, and Prometheus + JMX + Grafana monitoring out of the box, all under an Apache 2.0 license that costs zero and ships in every major cloud K8s distribution's marketplace. Red Hat's AMQ Streams (rebranded Streams for Apache Kafka in recent releases) is Strimzi with commercial support and long-term support versions, so choosing Strimzi in an enterprise typically means "run the OSS Strimzi operator" or "buy AMQ Streams for the support contract while keeping the same operator." Every senior data-engineering interview in 2026 probes Strimzi because it is the de-facto reference for what a Kubernetes-native stateful workload should look like — the operator pattern, the CRD surface, the declarative reconciliation loop, and the safety guarantees around rolling upgrades and broker replacement all originated or matured here and now inform how other stateful K8s workloads (Postgres via CloudNativePG, Elasticsearch via ECK, ClickHouse via clickhouse-operator) are designed.

Strimzi vs Confluent for Kubernetes — which do I pick?

Default to Strimzi when your organisation is OSS-first, your cost sensitivity is high, and you can live without Confluent-specific enterprise features (Control Center dashboards, LDAP-backed MDS RBAC, first-class tiered storage as a click-to-enable feature, Confluent Hub connector auto-installation). Strimzi is Apache 2.0 with zero license fee; the only costs are your Kubernetes cluster, persistent volumes, and engineering time. It ships every operational primitive you actually need — CRDs for topics and users, Cruise Control for rebalancing, Strimzi CA for cert rotation, Entity Operator for declarative CRUD — and its CNCF-graduated status plus the AMQ Streams commercial-support option means you can buy support if leadership demands it without changing tooling. Default to Confluent for Kubernetes (CFK) when leadership has already signed a Confluent Platform contract, when your compliance requirements include LDAP/SAML-integrated RBAC with a first-class audit trail on day one, or when tiered storage must be a click-to-enable feature rather than a config exercise. CFK's per-CPU or per-broker license (typically $2-5k per broker per year in the ballpark) is the price you pay for the whole-platform bundling — Kafka plus Schema Registry plus ksqlDB plus Connect plus RBAC plus Control Center plus tiered storage plus MDS, all as first-class CRDs reconciled by one operator — plus Confluent's commercial support, SOC-2 attestation, and professional services. The 80/20 rule: 80% of teams pick Strimzi (cheaper, OSS, sufficient); 20% pick CFK (enterprise features, existing contract, regulated industry). Never argue "always Strimzi" or "always Confluent"; the axes decide.

What is KRaft and did it really replace ZooKeeper?

KRaft is the Kafka Raft protocol — a native consensus algorithm built into Kafka that replaces the separate ZooKeeper ensemble that Kafka historically required for cluster metadata management, controller election, and broker registration. It was designed in KIP-500 (Kafka Improvement Proposal 500), reached general availability in Apache Kafka 3.5 (July 2023), and became the default in Kafka 4.0 (early 2025) with ZooKeeper support removed entirely from the codebase. In practice, running Kafka today means running KRaft — every greenfield deployment uses it, every managed service (Confluent Cloud, MSK, Aiven) has migrated, and every self-hosted deployment on 3.5+ can and generally should migrate via the KIP-866 dual-write path. The operational change is significant: before KRaft, every Kafka cluster was really two clusters (Kafka brokers plus a ZooKeeper ensemble of 3-5 ZK nodes) with their own operational surface, upgrade cadence, monitoring, and version-skew concerns; after KRaft, a dedicated controller quorum of 3 or 5 Kafka nodes runs the Raft consensus directly, and the metadata is a Kafka topic (@metadata) that brokers subscribe to just like any other topic. This shrinks the deployment footprint, speeds controller failover from ZK's ~seconds to Raft's ~milliseconds, and removes an entire category of operational failure modes (ZK ensemble split-brain, ZK-Kafka version mismatch, ZK snapshot / txn log growth) that plagued previous decades of Kafka deployments. Every senior data-engineering interview in 2026 probes KRaft because knowing the version numbers (3.5 GA, 4.0 default), the controller quorum sizing rule (odd, 3 or 5), and the migration path (KIP-866 dual-write) is table stakes for anyone claiming to have deployed Kafka in the last two years.

When is bare-metal Kafka still better than Kubernetes?

Bare-metal (or EC2 with direct instance-store NVMe) Kafka is still the right answer in three specific scenarios in 2026. First, latency-critical workloads where the sub-millisecond difference matters: real-time ad bidding (a bid arriving 5 ms late is a bid you don't win), high-frequency financial ticker feeds (regulatory microstructure requirements), real-time gaming state replication (500 μs of jitter is a bad user experience), and tier-1 IoT ingest at the edge where the network round-trip is already dominant. Kubernetes' network stack (kube-proxy iptables / IPVS / eBPF), container runtime overhead, and EBS-backed PersistentVolumeClaims add ~200-500 μs of p99 to every RPC and skip the direct-NVMe I/O path that bare metal delivers; for a workload with a 2 ms p99 SLA, that overhead is the whole ball game. Second, environments where Kubernetes isn't available or isn't appropriate: air-gapped installations (defence, some finance), older on-prem datacenters that predate the K8s adoption era, and organisations whose compliance rules forbid running stateful workloads on shared container platforms. Third, teams with deep bare-metal Linux ops maturity who have been running Kafka on systemd + Ansible for years and have no compelling reason to migrate — the "if it isn't broken, don't fix it" case. For everyone else — the 95%+ of workloads with normal latency budgets and normal compliance requirements and access to Kubernetes — Strimzi or CFK is the correct answer and the operator's convenience is worth far more than the marginal p99 latency edge.

How many controllers does a KRaft quorum need?

Always an odd number, always 3 or 5. Three controllers run the Raft quorum with a majority requirement of 2 out of 3, which tolerates the loss of 1 controller before the metadata plane becomes unwritable — this is the minimum viable KRaft deployment and the correct sizing for small clusters (under 20 brokers), lower-throughput metadata workloads (few topic-create operations per second), and any deployment where cost per controller node matters. Five controllers run the Raft quorum with a majority requirement of 3 out of 5, tolerating the loss of 2 controllers, and are the recommendation for large clusters (over 20 brokers), high-throughput metadata workloads, or any deployment spread across 3+ availability zones where you need to survive a full-AZ loss (spreading 5 controllers as 2/2/1 across three AZs keeps quorum even when one entire AZ goes dark). Even counts (2 or 4) are strictly worse than the next-lower odd number — a 4-node quorum still only tolerates 1 failure (needs 3 out of 4 for majority) so it provides zero additional fault tolerance over a 3-node quorum while paying for an extra host. Never deploy more than 5 — the Raft algorithm's write latency scales with the round-trip to the slowest voter in the majority, so 7 or 9 controllers just make every metadata write slower without adding meaningful fault tolerance. Colocate versus dedicate: for small clusters (under ~10 brokers) it's acceptable to run process.roles=broker,controller on 3 nodes doubling as both roles; for anything larger, dedicate the controllers to their own hardware or K8s pods so the data plane and metadata plane are isolated and can be upgraded independently.

Can I move from ZooKeeper to KRaft without downtime?

Yes, via the KIP-866 dual-write migration path introduced in Apache Kafka 3.5 and refined through subsequent releases. The migration is a well-defined, thoroughly-documented, production-tested playbook that runs Kafka in a hybrid mode where cluster metadata is written to both ZooKeeper and a new KRaft controller quorum simultaneously, so a rollback to ZK is possible at every stage until you commit to the final cutover. The high-level playbook is: (a) provision a new KRaft controller quorum (3 or 5 nodes) alongside the existing ZK ensemble, running Kafka 3.5+ binaries; (b) start the controllers in zookeeper.migration.enable=true mode so they participate in the ZK migration protocol; (c) reconfigure the existing brokers to know about both the ZK ensemble and the new KRaft quorum (kraft.migration.enable=true), triggering the dual-write phase where every metadata mutation lands in both stores; (d) monitor the migration progress via kafka.controller metrics until all metadata is replicated to KRaft; (e) roll each broker one at a time to process.roles=broker (dropping ZK), verifying ISR safety between each broker; (f) once all brokers are ZK-free, decommission the ZK ensemble entirely; (g) upgrade to Kafka 4.0+ which removes ZK support entirely. On Strimzi 0.36+ and CFK 2.9+, most of this is automated by the operator — you flip an annotation (strimzi.io/kraft: migration) and the operator drives the phases. On bare metal, you follow the KIP-866 runbook manually via Ansible. Expect the migration to take 1-2 weeks of elapsed time (mostly waiting for the dual-write phase to catch up), with the actual broker rolls taking hours; producer and consumer traffic is unaffected throughout because ISR safety is enforced at every step. This is the migration every ZK-based deployment in 2026 has on its roadmap, and the sooner you plan it the better — Kafka 4.0 is the version most managed services have already adopted, and staying on 3.x indefinitely is not a viable long-term strategy.

Practice on PipeCode

• Drill the streaming practice library → for the Kafka broker, producer, consumer, partition-assignment, and KRaft-quorum-controller problems senior interviewers love — the exact scenarios you'll face when defending a Strimzi vs CFK vs bare-metal choice.
• Rehearse system design against the design practice library → for the operator-vs-bare-metal, multi-AZ replication, tiered-storage-cost, and RBAC-model design scenarios that mirror the Kafka-deployment interview.
• Sharpen the SQL axis with the SQL practice library → for the ksqlDB, stream-table join, and windowed-aggregation problems that show up when Kafka streams feed a warehouse via CDC.
• Warm up on the ETL practice library → for the Kafka-Connect, Debezium-source, and S3-sink patterns that dominate Kafka-driven data platforms.
• Practice the design practice library → again on the failure-mode drills — broker loss, controller flap, AZ outage, split-brain, and consumer-lag alarms are the exact scenarios your on-call runbook will cover.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the five-axis deployment matrix, the KRaft quorum sizing rule, and the three-pattern trade-off tree against real graded inputs.

Lock in strimzi kafka muscle memory

Docs explain operators. PipeCode drills explain the decision — when Strimzi wins on cost, when CFK's LDAP-backed MDS RBAC earns the license fee, when bare-metal KRaft's direct-NVMe latency edge is the whole ball game, when the KRaft controller quorum sizing rule matters, and when Cruise Control turns a scale-out project into a config change. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face when picking how to run Kafka in 2026.

Practice streaming problems →
Practice design problems →