Amazon MSK & MSK Serverless: Managed Kafka Without the Ops

amazon msk — Amazon Managed Streaming for Apache Kafka — is the AWS service that runs real, open-source Apache Kafka for you, so you keep the exact Kafka API, protocol, and ecosystem you already know while handing AWS the parts nobody enjoys: provisioning brokers, patching them, replacing failed nodes, spreading them across availability zones, and scaling storage. You do not get a Kafka-flavoured reimplementation and you do not get a proprietary lock-in protocol; you get the same brokers, the same kafka-topics.sh, the same client libraries, with the undifferentiated operations lifted out.

That boundary — Kafka you own, operations AWS owns — is the whole reason the service exists, and it splits into two very different shapes. Provisioned MSK is a cluster you size yourself: you pick broker instance types, a broker count, storage, and a replication factor, and you pay for those brokers whether they are busy or idle. MSK Serverless removes sizing entirely: no brokers to choose, no storage to scale, throughput that auto-scales within published ceilings, and a bill measured in partition-hours and gigabytes rather than instance-hours. This guide walks the five things an interviewer will actually probe — the managed/owned boundary, provisioned sizing and replication, the serverless throughput model, authentication and encryption, and MSK Connect plus the build-versus-buy call against self-managed Kafka, Redpanda, and WarpStream — and pairs each with a Solution-Tail answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library →, rehearse the consume-and-transform patterns on the event-processing practice set →, and harden your failure handling on the fault-tolerance practice set →.

---

On this page

• Why Amazon MSK changes managed Kafka in 2026
• Provisioned brokers, sizing & replication factor
• MSK Serverless — throughput, partitions & zero capacity planning
• Authentication & encryption — IAM, TLS, SASL/SCRAM
• MSK Connect & choosing your Kafka
• Cheat sheet — Amazon MSK recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Amazon MSK changes managed Kafka in 2026

Amazon MSK is managed Apache Kafka — the ops boundary is the entire value proposition

The one-sentence invariant: MSK runs genuine open-source Apache Kafka and moves only the operational lifecycle behind an AWS control plane, so you keep every Kafka API and lose the pager duty. Everything that makes MSK attractive follows from that. There is no new query language, no proprietary broker, no client rewrite; a topic you created on self-managed Kafka behaves identically on MSK, and a Kafka consumer library from any language connects the same way once it can authenticate.

What Amazon MSK actually is.

• Real Kafka, not a clone. MSK provisions actual Apache Kafka brokers on EC2-class hardware inside AWS's account boundary, running a Kafka version you select (recent versions run in KRaft mode; older ones used AWS-managed ZooKeeper). The wire protocol is stock Kafka, so Kafka Streams, Kafka Connect, Schema Registry, and librdkafka clients all just work.
• A managed control plane. AWS handles broker provisioning, OS and Kafka patching, automatic replacement of unhealthy brokers, multi-AZ placement, CloudWatch metrics, and (for provisioned) storage auto-scaling. You never SSH into a broker.
• Two shapes, one API. Provisioned clusters expose sizing knobs (broker type, count, storage, config); Serverless clusters hide them and auto-scale. Both speak identical Kafka.

The managed / owned split — say it crisply.

• AWS owns: the brokers, the host OS, Kafka patch level, node failure recovery, inter-AZ networking for replication, and the metadata quorum (KRaft/ZooKeeper).
• You own: topics, partition counts, replication factor and min.insync.replicas, retention, producer/consumer code, consumer-group design, serialization, and client-side auth configuration.
• The trap: MSK does not save you from a bad partition count, a replication factor of 1, or acks=1 on data you cannot lose. Managed brokers do not make an under-replicated topic durable.

Where MSK sits against the alternatives.

• vs self-managed Kafka on EC2. Self-managed gives total control — any Kafka version, any broker config, custom JVM tuning — at the cost of owning every upgrade, every failed disk, and every 3 a.m. AZ failover. MSK trades that control for AWS running the lifecycle.
• vs Redpanda. Redpanda is a Kafka-API-compatible broker rewritten in C++ with no JVM and no ZooKeeper, tuned for low, predictable tail latency. MSK is the actual Kafka; Redpanda is a compatible re-implementation optimised for latency and simpler ops.
• vs WarpStream. WarpStream is a Kafka-compatible, diskless system that writes straight to object storage (S3), eliminating inter-AZ replication traffic and local disks — dramatically cheaper at high throughput, at the cost of higher end-to-end latency (typically hundreds of milliseconds).

What interviewers listen for.

• Do you say "MSK is managed Apache Kafka, not a Kafka-like service" in the first sentence? — senior signal.
• Do you name the managed/owned boundary unprompted — AWS runs brokers, you own durability config? — required framing.
• Do you distinguish provisioned (you size it) from serverless (ceilings, no sizing)? — the core mental model.
• Do you reach for MSK when the answer is "we want Kafka but not the broker ops", and reach past it (self-managed, Redpanda, WarpStream) when latency or cost forces the issue? — senior signal.

Worked example — stand up a cluster in one CLI call

Detailed explanation. A provisioned MSK cluster is declared, not hand-built. You describe the broker node group — instance type, how many brokers, which subnets (one per AZ), and storage — and hand it to aws kafka create-cluster. AWS then provisions the brokers, forms the cluster, and returns an ARN; nothing else is yours to install. The same declaration scales from a 3-broker starter to a 15-broker production cluster by changing two numbers.

Question. Create a 3-broker provisioned cluster named orders-stream on Kafka 3.6.0, one broker per AZ across three subnets, kafka.m7g.large brokers with 100 GiB of storage each, and show what AWS returns.

Input.

field	value
broker type	kafka.m7g.large
number of brokers	3 (one per AZ)
storage per broker	100 GiB EBS
Kafka version	3.6.0

Code.

aws kafka create-cluster \
  --cluster-name "orders-stream" \
  --kafka-version "3.6.0" \
  --number-of-broker-nodes 3 \
  --broker-node-group-info '{
    "InstanceType": "kafka.m7g.large",
    "ClientSubnets": ["subnet-aza","subnet-azb","subnet-azc"],
    "SecurityGroups": ["sg-kafka"],
    "StorageInfo": {"EbsStorageInfo": {"VolumeSize": 100}}
  }' \
  --encryption-info '{"EncryptionInTransit":{"InCluster":true,"ClientBroker":"TLS"}}'

Step-by-step explanation. create-cluster sends a declarative spec to the MSK control plane; it returns immediately with a ClusterArn and state CREATING. AWS provisions three kafka.m7g.large brokers, one in each named subnet (so one per AZ), attaches a 100 GiB EBS volume to each, forms the KRaft metadata quorum, and enables TLS both client-to-broker and in-cluster. When the state reaches ACTIVE, aws kafka get-bootstrap-brokers yields the connection endpoints. You wrote no install scripts and touched no host.

Output.

AWS returned	value
ClusterArn	arn:aws:kafka:...:cluster/orders-stream/...
initial state	CREATING → ACTIVE
brokers	3 (one per AZ), TLS in-cluster + client-broker
bootstrap	available via get-bootstrap-brokers once ACTIVE

Rule of thumb. If you can describe the broker node group in JSON, MSK can build it — the starter cluster and the production cluster differ only in instance type, broker count, and storage, never in the process.

Amazon MSK interview question on the managed boundary

Question. You have just created a provisioned MSK cluster with IAM access control enabled. Without SSHing into anything, produce your first message to a new topic from an EC2 client and explain exactly which parts of that path AWS operates versus which parts you configured.

Solution Using the IAM bootstrap and a console producer

Code.

 # 1. Get the IAM (SASL/OAUTHBEARER) bootstrap string — note port 9098
aws kafka get-bootstrap-brokers --cluster-arn "$ARN" \
  --query BootstrapBrokerStringSaslIam --output text
 # -> b-1...:9098,b-2...:9098,b-3...:9098

 # 2. Client config using the AWS IAM auth handler (aws-msk-iam-auth on classpath)
cat > client.properties <<'EOF'
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
EOF

 # 3. Create the topic and produce (partitions/RF are YOURS to choose)
kafka-topics.sh --bootstrap-server "$BOOT" --command-config client.properties \
  --create --topic orders --partitions 6 --replication-factor 3
kafka-console-producer.sh --bootstrap-server "$BOOT" --producer.config client.properties \
  --topic orders

Step-by-step trace.

step	who runs it	what happens
1	AWS control plane	returns the :9098 IAM bootstrap endpoints
2	you (client)	client presents its EC2 role via AWS_MSK_IAM; broker checks IAM policy
3	AWS brokers	topic orders created with your 6 partitions, RF 3
4	AWS brokers	leader elected per partition across the 3 AZs
5	you (producer)	message routed to a partition leader over TLS, replicated by the brokers

1. The bootstrap endpoint and the brokers behind it are AWS-operated; you only fetch the address.
2. Authentication is SASL_SSL with mechanism AWS_MSK_IAM: the client signs with its IAM role and the broker evaluates your IAM policy — AWS runs the check, you wrote the policy.
3. --partitions 6 --replication-factor 3 are your durability and parallelism decisions; MSK executes them but never chooses them for you.
4. Leader election, replication across AZs, and fsync are broker-side, fully managed.
5. The producer only needs the bootstrap string and auth config; everything after the TLS handshake is Kafka doing Kafka.

Output:

concern	operated by
broker provisioning, patching, leader election, replication	AWS
topic name, partitions, replication factor, retention	you
IAM policy + client auth config	you (AWS enforces)

Why this works — concept by concept:

• Managed control plane — AWS owns the broker lifecycle and the metadata quorum, so "produce a message" never involves installing or patching Kafka.
• IAM access control — authentication piggybacks on AWS IAM via AWS_MSK_IAM, so client identity is an IAM role, not a Kafka password you rotate by hand.
• You still own durability — partitions and replication factor are your call; managed brokers execute but do not second-guess an unsafe replication-factor 1.
• Same Kafka tooling — kafka-topics.sh and kafka-console-producer.sh work unchanged, proving MSK is real Kafka rather than a look-alike API.
• Cost — producing is O(message size) network I/O to a leader; managed replication adds a bounded per-message fan-out to replication.factor − 1 followers.

Streaming
Topic — streaming
Kafka produce-and-consume streaming problems

Practice →

Reliability Topic — fault-tolerance Broker-failure and recovery problems

Practice →

---

2. Provisioned brokers, sizing & replication factor

Size the brokers, spread them across AZs, and replicate every partition three ways

Provisioned MSK hands you the knobs, and the interview is really about turning them correctly. Say the durability rule in one breath: replication factor 3 across three AZs, min.insync.replicas=2, producers on acks=all — that trio is what makes an acknowledged write survive a full availability-zone outage with zero data loss.

Broker instance types.

• Dev / small. kafka.t3.small is a burstable, cheap broker for non-production and light loads. Fine to learn on, wrong for sustained throughput.
• Production. The kafka.m7g / kafka.m5 families (Graviton m7g is the modern default) scale from .large up to very large sizes; bigger instances give more network, CPU, and page cache, which is what actually caps Kafka throughput.
• Right-size by ingest, not by row count. AWS publishes per-broker throughput guidance; pick the smallest instance that comfortably clears your peak MB/s per broker with headroom for a broker being down.

Brokers must be a multiple of the AZ count.

• MSK places brokers evenly across the subnets (AZs) you supply — 2 or 3 AZs. Three AZs is the durable default.
• Broker count is a multiple of AZs: 3, 6, 9, 12… for a 3-AZ cluster, so each AZ holds an equal share and losing one AZ loses a predictable fraction.
• More brokers = more total throughput and more partition capacity, but also more inter-AZ replication traffic (a real cost line).

Replication factor and in-sync replicas — the durability core.

• replication.factor=3. Each partition has one leader and two followers, one copy per AZ. Losing a broker or an entire AZ still leaves the data on the survivors.
• min.insync.replicas=2. With acks=all, a write is only acknowledged once at least 2 replicas have it. If only one replica is in sync, producers get an error instead of a silent single-copy write.
• acks=all on the producer. The client half of the contract: acknowledge only when the in-sync set has the record. RF=3 + min.insync=2 + acks=all tolerates one broker/AZ failure with no acknowledged-data loss and still accepts writes.

Storage and tiered storage.

• EBS per broker. Each broker gets an EBS volume (up to 16 TiB); MSK can auto-scale storage up as you fill it. You can also provision extra storage throughput for hot workloads.
• Tiered storage. MSK provisioned supports a two-tier model: a local (hot) tier on EBS governed by local.retention.ms / local.retention.bytes, and a remote (cold) tier where older segments age out to low-cost managed storage while overall retention.ms stays long. You get weeks or months of retention without paying EBS prices for cold data, and consumers still read old offsets transparently.

Worked example — sizing a 3-broker cluster for a target ingest

Detailed explanation. Sizing is throughput arithmetic tempered by a failure margin. You start from peak ingest, multiply by the replication factor to get the write load the cluster actually carries, divide across brokers, and then check that the cluster still copes when one broker is gone. The instance type is whatever clears that per-broker number with headroom.

Question. You expect 60 MB/s of peak producer ingest into topics configured replication.factor=3. You want to run 3 brokers across 3 AZs and survive one broker being down. Roughly what write load does each surviving broker carry, and is a small burstable instance appropriate?

Input.

parameter	value
peak producer ingest	60 MB/s
replication factor	3
brokers	3 (1 per AZ)
failure margin	survive 1 broker down

Code.

replicated write load = ingest x replication.factor = 60 x 3 = 180 MB/s
per-broker (all healthy)   = 180 / 3 = 60 MB/s
per-broker (one down => 2) = 180 / 2 = 90 MB/s   <-- size for THIS

Step-by-step explanation. Producers push 60 MB/s, but every byte is written three times (leader + 2 followers), so the cluster moves 180 MB/s of replicated writes. Split across 3 healthy brokers that is 60 MB/s each — but you must survive a broker loss, so size for the degraded case where 2 brokers share 180 MB/s, i.e. 90 MB/s per broker. A kafka.t3.small burstable instance cannot sustain that; you choose an m7g size whose documented per-broker throughput clears 90 MB/s with headroom for consumers and replication.

Output.

scenario	per-broker write load	verdict
all 3 healthy	60 MB/s	comfortable
1 broker down (2 left)	90 MB/s	the number you size for
burstable t3.small	cannot sustain 90 MB/s	reject → pick m7g

Rule of thumb. Always size brokers for the degraded cluster, not the healthy one; a cluster that only copes when every broker is up is one instance failure away from an outage.

Amazon MSK interview question on surviving an AZ outage

Question. Marketing can tolerate a slightly slower producer, but finance cannot lose a single acknowledged payment event even if an entire availability zone goes dark. Configure the payments topic so an acknowledged write is guaranteed to survive a full-AZ failure, and explain why a lazier config silently loses data.

Solution Using RF3 + min.insync.replicas=2 + acks=all

Code.

 # Topic: 3 copies, one per AZ; require 2 in-sync before ack
kafka-topics.sh --bootstrap-server "$BOOT" --command-config client.properties \
  --create --topic payments \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=1209600000        # 14 days

 # Producer must request full acknowledgement
 #   acks=all
 #   enable.idempotence=true   (no duplicate on retry)
 #   retries=2147483647, max.in.flight.requests.per.connection=5

Step-by-step trace.

event	in-sync replicas	acks=all outcome
all healthy	3 (a,b,c)	write acked after 2+ have it
AZ-c down	2 (a,b)	still ≥ min.insync ⇒ writes continue, acked
AZ-c and AZ-b down	1 (a)	< min.insync ⇒ producer errors, no silent 1-copy write
AZ-c returns	2→3	follower catches up, rejoins ISR

1. replication.factor=3 puts one copy of every partition in each of the three AZs.
2. min.insync.replicas=2 with acks=all means the leader only acknowledges once a second replica also has the record — so an acked payment exists in at least two AZs.
3. Lose one AZ and the in-sync set drops to 2, which still meets the floor: writes keep flowing and stay durable.
4. Lose two AZs and the floor is breached — the producer receives an error rather than committing a single-copy write that a final failure could erase.
5. enable.idempotence=true ensures the retry after any transient error does not create a duplicate payment.

Output:

config	AZ-loss behaviour	acknowledged-data loss
RF3 + min.insync=2 + acks=all	tolerates 1 AZ, errors past that	none
RF3 + acks=1	leader acks alone	loses in-flight on leader failover
RF1	no redundancy	loses everything on the one broker

Why this works — concept by concept:

• Replication factor 3 — three copies, one per AZ, means a whole-AZ outage still leaves two live copies of every acknowledged record.
• min.insync.replicas — turns "how many copies must confirm before ack" into a hard floor; below it the broker refuses the write instead of accepting a fragile single copy.
• acks=all — the producer-side half of the contract; without it min.insync.replicas is toothless because the leader would ack alone.
• Idempotent producer — makes the mandatory retries safe, so durability does not come at the cost of duplicated payments.
• Cost — durability is O(replication.factor) in write amplification and inter-AZ bytes; RF3 triples the network and storage of RF1, which is the price of zero-loss.

Streaming
Topic — streaming
Partitioning and replication streaming problems

Practice →

Reliability Topic — fault-tolerance AZ-failure and zero-data-loss problems

Practice →

---

3. MSK Serverless — throughput, partitions & zero capacity planning

Serverless removes sizing — you stop choosing brokers and start living within throughput ceilings

MSK Serverless is the same Kafka with the capacity decisions deleted. Say what disappears in one breath: no broker instance type, no broker count, no storage to provision, no rebalancing — throughput auto-scales and you pay for what flows. In exchange you accept published per-partition and per-cluster ceilings and a smaller set of options (IAM auth only), which is exactly the right trade for spiky or unpredictable workloads.

What disappears.

• No sizing. You do not pick kafka.m7g.large or a broker count; there are no brokers to see. AWS provisions and scales capacity under an opaque cover.
• No storage management. Storage and retention are effectively unlimited and elastic; you do not pre-provision EBS or worry about filling a disk.
• No manual scaling. Throughput scales up and down with load automatically, so a 10x spike does not require a cluster resize the night before.

The throughput model — the part interviews probe.

• Per-partition ceilings. Each partition has an instantaneous write and read ceiling (published as roughly 5 MB/s in, 10 MB/s out per partition at the time of writing). More throughput on a topic means more partitions, not a bigger broker.
• Per-cluster ceilings. A serverless cluster has aggregate ingest/egress ceilings and a maximum partition count (published around 2,400 partitions per cluster). Design your partition count against that budget.
• Partitions are the scaling unit. Because you cannot add brokers, you scale by adding partitions up to the cluster limit — which makes the per-partition ceiling times partition count your real throughput headroom.

Auth, encryption, and billing.

• IAM only. Serverless supports only IAM access control (SASL/OAUTHBEARER, port :9098) — no SASL/SCRAM, no mTLS. Simpler, and it forces good practice.
• Always encrypted. TLS in transit is mandatory; there is no plaintext option.
• Pay per use. Billing is a cluster-hour plus partition-hours plus storage plus data in/out — no idle-broker charge, which is why serverless wins for bursty, low-average-throughput streams.

Worked example — partition math against the serverless ceiling

Detailed explanation. With no brokers to add, serverless throughput planning is partition planning. You take the required write throughput per topic, divide by the per-partition write ceiling to get a minimum partition count, then check the total against the per-cluster partition budget. If you need more headroom you add partitions, not hardware.

Question. A topic must sustain 30 MB/s of writes on MSK Serverless, where the per-partition write ceiling is ~5 MB/s. What is the minimum partition count, and how does it sit against the ~2,400-partition cluster limit if you run 10 similar topics?

Input.

parameter	value
required write throughput / topic	30 MB/s
per-partition write ceiling	~5 MB/s
topics	10
cluster partition limit	~2,400

Code.

min partitions/topic = ceil(30 / 5) = 6      # plus headroom -> use 8
partitions for 10 topics = 8 x 10 = 80
80  <<  2,400  cluster limit  => plenty of room

Step-by-step explanation. 30 MB/s divided by the ~5 MB/s per-partition ceiling needs at least 6 partitions; you round up to 8 for headroom and skew tolerance. Ten such topics use 80 partitions — comfortably inside the ~2,400 cluster budget, so serverless handles the load without you ever choosing a broker. If a single topic needed 400 MB/s you would need ~80 partitions for it alone, and you would start checking the per-cluster ingest ceiling, not just the partition count.

Output.

metric	value
min partitions per topic	6 (use 8 with headroom)
total partitions (10 topics)	80
against ~2,400 limit	~3% used — fine

Rule of thumb. On serverless, throughput is bought in partitions: partitions ≥ ceil(target MB/s ÷ per-partition ceiling), then sanity-check the sum against the cluster partition and ingest limits.

Amazon MSK interview question on provisioned vs serverless

Question. You have two workloads: (a) a steady 250 MB/s clickstream that runs 24/7, and (b) a batch-triggered ingestion that is idle most of the day but spikes hard for two hours each night. Which MSK flavour for each, and how do you justify it on cost and operations?

Solution Using serverless for the spiky workload, provisioned for the steady one

Code.

 # Spiky, unpredictable, mostly-idle workload -> Serverless (pay per partition-hour)
aws kafka create-cluster-v2 --cluster-name nightly-ingest \
  --serverless '{
    "VpcConfigs":[{"SubnetIds":["subnet-aza","subnet-azb","subnet-azc"],
                   "SecurityGroupIds":["sg-kafka"]}],
    "ClientAuthentication":{"Sasl":{"Iam":{"Enabled":true}}}
  }'

 # Steady, high, 24/7 throughput -> Provisioned (cheaper per MB at constant load)
aws kafka create-cluster --cluster-name clickstream \
  --kafka-version 3.6.0 --number-of-broker-nodes 6 \
  --broker-node-group-info '{"InstanceType":"kafka.m7g.2xlarge", ...}'

Step-by-step trace.

workload	shape	pick	why
clickstream	250 MB/s constant 24/7	provisioned	brokers run hot all day; per-MB cost of owned instances beats partition-hour billing
nightly ingest	idle ~22h, spikes 2h	serverless	no idle broker charge; auto-scales the spike; IAM-only is fine

1. Serverless bills partition-hours and data, so a mostly-idle cluster costs almost nothing off-peak and absorbs the nightly spike without a pre-scaled cluster.
2. Provisioned bills per broker-hour regardless of load, which is wasteful for idle time but the cheapest per MB when brokers are saturated around the clock.
3. The steady 250 MB/s clickstream keeps 6 m7g.2xlarge brokers busy continuously, so owned instances amortise well and you also unlock SASL/SCRAM or mTLS if compliance wants them.
4. The nightly job's operational win is bigger than the cost win: nobody resizes a cluster before the spike, because serverless has no size.

Output:

decision	clickstream	nightly ingest
flavour	provisioned (6 × m7g.2xlarge)	serverless
billing basis	broker-hours	partition-hours + data
ops burden	size + monitor brokers	none

Why this works — concept by concept:

• Utilisation decides — provisioned is cheapest at high, constant utilisation; serverless is cheapest when average load is low relative to peak.
• Partition-hour billing — serverless charges for what flows, so idle time is nearly free — perfect for spiky, scheduled ingestion.
• No capacity planning — serverless removes the pre-spike resize entirely, which is often the real reason to choose it over a marginal cost saving.
• Auth trade-off — serverless is IAM-only; if a workload needs SASL/SCRAM or mTLS, that alone can force provisioned.
• Cost — provisioned ≈ O(brokers × hours) flat; serverless ≈ O(partition-hours + bytes), so the crossover is a utilisation ratio, not a raw throughput number.

Streaming
Topic — streaming
Throughput and partition-scaling streaming problems

Practice →

Events Topic — event-processing Bursty event-ingestion and scaling problems

Practice →

---

4. Authentication & encryption — IAM, TLS, SASL/SCRAM

Four ways in, one encryption story — pick IAM first, and everything is TLS in transit and KMS at rest

Security on MSK is two questions with clean answers: how does a client prove who it is (four options, each on its own bootstrap port) and how is data protected on the wire and on disk (TLS in transit, KMS at rest, always). Say the default in one breath: prefer IAM access control on port 9098 — identity becomes an IAM role and access becomes an IAM policy, with no passwords to rotate.

The four authentication methods (provisioned).

• IAM access control — :9098. SASL/OAUTHBEARER backed by AWS IAM. Clients authenticate with their AWS credentials (an EC2/EKS role, for instance) and a fine-grained IAM policy authorises specific topics and actions. This is the recommended path and the only option on Serverless.
• SASL/SCRAM — :9096. Username/password auth where credentials live in AWS Secrets Manager. The secret must be encrypted with a customer-managed KMS key (not the default AWS-managed key) and its name must start with AmazonMSK_. Good when clients cannot assume an IAM role.
• mTLS — :9094. Mutual TLS with client certificates issued by AWS Certificate Manager Private CA. The broker validates the client cert; authorisation is via Kafka ACLs. Right for legacy clients that already speak certificate auth.
• Plaintext / unauthenticated — :9092 (plaintext), :9094 (TLS). No client identity; only acceptable inside a locked-down VPC for development. Never for production data.

Encryption — in transit and at rest.

• In transit. TLS encrypts client-to-broker traffic and, when enabled, in-cluster broker-to-broker replication traffic too. The IAM, SCRAM, and mTLS ports are all TLS; only :9092 is plaintext.
• At rest. Broker EBS volumes are encrypted with AWS KMS — an AWS-managed key by default, or your own customer-managed key (CMK) for tighter control and auditability.

Authorisation — who can do what.

• With IAM auth, an IAM policy grants actions like kafka-cluster:WriteData, kafka-cluster:ReadData, kafka-cluster:CreateTopic scoped to specific cluster, topic, and consumer-group ARNs — least privilege expressed in IAM.
• With SCRAM or mTLS, authorisation uses Kafka ACLs (kafka-acls.sh) mapped to the SCRAM username or the certificate principal.
• The principle: grant a producer WriteData on exactly its topics and nothing else; never hand out cluster-wide admin to an app.

Worked example — a SASL/SCRAM secret in Secrets Manager

Detailed explanation. SASL/SCRAM on MSK has one non-obvious rule that trips people in interviews: the secret cannot use the default KMS key and its name must carry the AmazonMSK_ prefix, then it must be associated with the cluster. Miss any of those and clients fail to authenticate with a confusing error.

Question. Create a SCRAM credential app1 for an MSK cluster and attach it so a client can log in on port 9096.

Input.

requirement	value
secret name	AmazonMSK_app1 (prefix required)
KMS key	customer-managed (not default)
payload	{"username":"app1","password":"..."}
step after create	associate secret with cluster

Code.

 # 1. Secret name MUST start with AmazonMSK_ and use a customer-managed KMS key
aws secretsmanager create-secret \
  --name "AmazonMSK_app1" \
  --kms-key-id "$CMK_ARN" \
  --secret-string '{"username":"app1","password":"S3cretPass!"}'

 # 2. Associate the secret with the cluster (enables SCRAM login for app1)
aws kafka batch-associate-scram-secret \
  --cluster-arn "$ARN" --secret-arn-list "$SECRET_ARN"

 # 3. Client connects on :9096 with SCRAM-SHA-512
 #   security.protocol=SASL_SSL
 #   sasl.mechanism=SCRAM-SHA-512
 #   sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule \
 #       required username="app1" password="S3cretPass!";

Step-by-step explanation. The secret is created with the mandatory AmazonMSK_ name prefix and a customer-managed KMS key — MSK rejects secrets under the default AWS-managed key. batch-associate-scram-secret links the secret to the cluster so the brokers accept that username. The client then connects on :9096 using SASL_SSL + SCRAM-SHA-512; authorisation for what app1 may read or write is granted separately via Kafka ACLs.

Output.

step	result
create-secret	AmazonMSK_app1 stored under CMK
associate	cluster now accepts app1 on :9096
client login	SASL_SSL + SCRAM-SHA-512 succeeds

Rule of thumb. SCRAM on MSK = customer-managed KMS key + AmazonMSK_ name prefix + associate-with-cluster; if any one is missing, authentication fails — prefer IAM unless a client genuinely cannot assume a role.

Amazon MSK interview question on least-privilege producer access

Question. A microservice running on EKS must write only to the orders topic on an IAM-enabled MSK cluster — nothing else, no read, no admin. Write the IAM policy and the client config, and explain why this is safer than a shared SCRAM password.

Solution Using a scoped IAM policy + IAM client config

Code.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ConnectCluster",
      "Effect": "Allow",
      "Action": ["kafka-cluster:Connect"],
      "Resource": "arn:aws:kafka:REGION:ACCT:cluster/orders-stream/*"
    },
    {
      "Sid": "WriteOrdersOnly",
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:WriteData",
        "kafka-cluster:DescribeTopic"
      ],
      "Resource": "arn:aws:kafka:REGION:ACCT:topic/<cluster>/<uuid>/orders"
    }
  ]
}

 # client.properties — IAM auth on :9098, no password anywhere
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Step-by-step trace.

action attempted	policy allows?	broker result
Connect to cluster	yes	handshake succeeds
WriteData to orders	yes	produce accepted
WriteData to payments	no (resource scoped to orders)	authorization failed
ReadData from orders	no (no read action)	authorization failed

1. The EKS pod assumes an IAM role (via IRSA); the client signs the connection with AWS_MSK_IAM — no username or password exists to leak.
2. kafka-cluster:Connect on the cluster ARN lets it open the connection; without it the handshake is rejected.
3. WriteData is scoped to the .../orders topic ARN, so a produce to any other topic is denied by the broker's IAM check.
4. No ReadData and no * admin action means a compromised producer cannot consume data or reshape the cluster.

Output:

capability	granted
connect + write orders	yes
write any other topic	no
read / admin	no

Why this works — concept by concept:

• IAM access control — identity is an IAM role and permission is an IAM policy, so there is no shared secret to distribute, rotate, or accidentally commit to git.
• Resource-scoped actions — WriteData bound to a single topic ARN is least privilege in its purest form; the broker enforces it, not the app.
• No standing password — unlike a SCRAM secret shared across pods, IAM credentials are short-lived and role-scoped, shrinking the blast radius of a leak.
• TLS everywhere — SASL_SSL means the write is encrypted in transit and the at-rest copy is KMS-encrypted, closing both channels.
• Cost — the IAM authorisation check is O(1) per connection/action against the policy, negligible next to message I/O.

Streaming
Topic — streaming
Secured producer/consumer streaming problems

Practice →

Reliability Topic — fault-tolerance Access-control and failure-isolation problems

Practice →

---

5. MSK Connect & choosing your Kafka

Land data with managed connectors — then pick MSK, self-managed, Redpanda, or WarpStream by the trade-off

The last thing interviews probe is the ecosystem and the platform decision. MSK Connect runs managed Kafka Connect so you move data in and out of Kafka without operating a consumer fleet; and the honest answer to "which Kafka?" is a trade-off across ops burden, latency, and cost — MSK is not always the winner.

MSK Connect — managed Kafka Connect.

• What it is. A fully managed Kafka Connect environment. You supply a connector plugin (a JAR, uploaded to S3 as a custom plugin) and a worker configuration; AWS runs the Connect workers.
• Autoscaling workers. Capacity is measured in MCUs (MSK Connect Units — vCPU + memory); connectors can autoscale worker count between a min and max, so throughput follows load.
• Common connectors. An S3 sink to archive every event as Parquet/JSON to a bucket; a Debezium source for change-data-capture from a database into Kafka; JDBC, OpenSearch, and many community connectors.
• Why use it. You get Connect's exactly-once-ish delivery, offset management, and schema handling without running and patching a Connect cluster yourself.

The cost / throughput levers.

• Inter-AZ replication is a real bill. On provisioned MSK, RF3 across AZs means every write crosses AZ boundaries twice — durable, but a meaningful networking cost at high throughput.
• Serverless trades flat cost for partition-hour cost. Great for bursty; can be pricier than saturated provisioned brokers at constant high load.
• Tiered storage cuts retention cost. Long retention lives in the cheap remote tier instead of expensive EBS.

MSK vs self-managed Kafka on EC2.

• Self-managed gives total control: any Kafka version the day it ships, custom broker configs, bespoke JVM tuning, and no AWS-imposed limits. You pay in operational ownership — upgrades, failed disks, AZ failover, monitoring.
• MSK removes that ownership for a service premium. Choose self-managed only when you genuinely need control MSK does not expose; otherwise the ops savings dominate.

Redpanda and WarpStream — the Kafka-API alternatives.

• Redpanda. A Kafka-protocol-compatible broker written in C++ with no JVM and no ZooKeeper, using a thread-per-core design for low, predictable tail latency and simpler single-binary ops. Compelling when latency and operational simplicity matter more than running the reference Kafka.
• WarpStream. A Kafka-compatible, diskless system whose stateless agents write directly to object storage (S3) — no local disks and no inter-AZ replication traffic, which slashes cost at high throughput. The trade is higher end-to-end latency (typically hundreds of ms), so it suits high-volume, latency-tolerant pipelines (logs, analytics) rather than low-latency request paths.

Worked example — the four-way platform comparison

Detailed explanation. The platform choice is not "which is best" but "which trade-off fits this workload." Score each option on the three axes interviewers care about — operational burden, latency, and cost at high throughput — and let the workload pick.

Question. For a high-volume, latency-tolerant log-analytics pipeline that a small team must run cheaply, rank MSK, self-managed Kafka, Redpanda, and WarpStream on ops, latency, and cost, and name a winner.

Input.

option	ops burden	tail latency	cost at high throughput
self-managed EC2 Kafka	high	low	medium
Amazon MSK	low	low	medium–high (inter-AZ)
Redpanda	low–medium	very low	medium
WarpStream	low	high (100s ms)	low (S3, no inter-AZ)

Code.

weights for THIS workload: cost x3, ops x2, latency x1 (latency-tolerant)
score = 3*cost + 2*ops + 1*latency   (higher = better on that axis)

WarpStream : cost=BEST(3) ops=GOOD(2) latency=OK(1)  -> strong on the 2 heavy axes
MSK        : cost=OK      ops=BEST    latency=BEST    -> loses on the heaviest axis (cost)
=> latency-tolerant + cost-first  =>  WarpStream wins THIS workload

Step-by-step explanation. Because the pipeline tolerates latency, WarpStream's biggest weakness (hundreds of ms) barely counts, while its diskless, no-inter-AZ design wins the heavily-weighted cost axis. MSK and Redpanda win on latency, which this workload does not need. Self-managed loses on ops for a small team. Flip the weights — a low-latency request path for a team that wants zero ops — and the winner becomes MSK or Redpanda instead.

Output.

workload	best fit	why
high-volume, latency-tolerant logs, cost-first	WarpStream	diskless S3, no inter-AZ cost
low-latency, zero-ops, real Kafka	Amazon MSK	managed, low latency
lowest tail latency, simple ops	Redpanda	C++, no JVM/ZooKeeper
need total control / custom version	self-managed	full ownership

Rule of thumb. Choose by the axis that dominates the workload: MSK for managed real-Kafka at low latency, Redpanda for tail latency, WarpStream for cost at high throughput when latency is negotiable, self-managed only for control you cannot otherwise get.

Amazon MSK interview question on landing events in S3

Question. Every event on the clicks topic must be archived to S3 as Parquet for the data lake, continuously, without your team running or patching a consumer fleet. How do you do it on MSK, and what makes it resilient to restarts?

Solution Using an MSK Connect S3 sink connector

Code.

{
  "connector.class": "io.confluent.connect.s3.S3SinkConnector",
  "tasks.max": "4",
  "topics": "clicks",
  "s3.bucket.name": "lake-raw-clicks",
  "s3.region": "us-east-1",
  "storage.class": "io.confluent.connect.s3.storage.S3Storage",
  "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
  "partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
  "path.format": "'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH",
  "partition.duration.ms": "3600000",
  "flush.size": "100000",
  "rotate.interval.ms": "60000"
}

 # Deploy as an MSK Connect connector:
 #  - custom plugin: the S3 sink JAR uploaded to S3
 #  - worker auth: IAM (SASL/OAUTHBEARER) to the cluster on :9098
 #  - autoscaling: MCU count min=1 max=4 by CPU

Step-by-step trace.

moment	connector behaviour	offset state
start	joins consumer group for clicks, reads from last committed offset	resumes, no gap
steady	buffers records, writes Parquet at flush.size / rotate.interval.ms	commits offsets after each S3 write
worker restart	MSK Connect restarts the task; it rejoins the group	continues from last committed offset
spike	autoscales toward tasks.max / max MCUs	keeps up

1. The connector is just a managed Kafka consumer group, so it tracks progress via committed offsets exactly like any consumer.
2. Records are buffered and flushed to S3 as Parquet partitioned by event time (year/month/day/hour), which is what the data lake wants.
3. Offsets are committed after a successful S3 write, so a crash re-reads the uncommitted tail and re-writes it rather than losing it.
4. On a restart or scale event, MSK Connect reassigns tasks and they resume from the last committed offset — no team member is paged.

Output:

property	result
destination	S3 Parquet, time-partitioned
team ops	none (managed workers)
restart behaviour	resumes from committed offset, no data loss
scaling	autoscales MCUs / tasks with load

Why this works — concept by concept:

• MSK Connect — a managed Kafka Connect runtime means the S3 sink runs on AWS-operated workers, so archiving needs no consumer fleet of yours.
• Consumer-group offsets — the connector's durability comes from committing offsets after each write, giving at-least-once delivery across restarts.
• Time-based partitioning — writing year/month/day/hour paths makes the lake queryable and cheap to prune, aligning ingestion with query patterns.
• Autoscaling MCUs — worker capacity follows load, so a traffic spike does not require a manual resize.
• Cost — throughput is O(events) with S3 PUTs batched by flush.size, so large flushes amortise request cost against object count.

Streaming
Topic — streaming
Sink-connector and stream-to-lake problems

Practice →

Events Topic — event-processing Event archival and consumer-group problems

Practice →

---

Cheat sheet — Amazon MSK recipes

Create a provisioned cluster (AWS CLI).

aws kafka create-cluster --cluster-name orders-stream \
  --kafka-version 3.6.0 --number-of-broker-nodes 3 \
  --broker-node-group-info '{"InstanceType":"kafka.m7g.large",
    "ClientSubnets":["subnet-aza","subnet-azb","subnet-azc"],
    "SecurityGroups":["sg-kafka"],
    "StorageInfo":{"EbsStorageInfo":{"VolumeSize":100}}}'

Durable topic (survives one AZ, zero acked-data loss).

kafka-topics.sh --bootstrap-server "$BOOT" --command-config client.properties \
  --create --topic payments --partitions 12 --replication-factor 3 \
  --config min.insync.replicas=2
 # producer: acks=all, enable.idempotence=true

IAM client config (:9098).

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Create a serverless cluster (IAM only).

aws kafka create-cluster-v2 --cluster-name nightly-ingest \
  --serverless '{"VpcConfigs":[{"SubnetIds":["subnet-aza","subnet-azb","subnet-azc"],
    "SecurityGroupIds":["sg-kafka"]}],
    "ClientAuthentication":{"Sasl":{"Iam":{"Enabled":true}}}}'

Tiered storage topic (long retention, cheap cold tier).

kafka-configs.sh --bootstrap-server "$BOOT" --command-config client.properties \
  --alter --entity-type topics --entity-name clicks \
  --add-config 'remote.storage.enable=true,local.retention.ms=3600000,retention.ms=2592000000'
 # hot tier: last 1h on EBS; total retention: 30 days in remote tier

MSK Connect S3 sink (essentials).

{"connector.class":"io.confluent.connect.s3.S3SinkConnector",
 "topics":"clicks","s3.bucket.name":"lake-raw-clicks",
 "format.class":"io.confluent.connect.s3.format.parquet.ParquetFormat",
 "flush.size":"100000","tasks.max":"4"}

Provisioned vs Serverless picker.

Situation	Choice
Steady, high, 24/7 throughput	provisioned (owned brokers)
Spiky / unpredictable / mostly idle	serverless (partition-hour billing)
Need SASL/SCRAM or mTLS auth	provisioned (serverless is IAM-only)
Zero capacity planning, fast start	serverless
Fine-grained broker config / tuning	provisioned (or self-managed)

---

Frequently asked questions

What is Amazon MSK?

Amazon MSK (Managed Streaming for Apache Kafka) is a fully managed AWS service that runs genuine open-source Apache Kafka. AWS provisions and patches the brokers, replaces failed nodes, spreads them across availability zones, and manages the metadata quorum, while you keep the standard Kafka API, protocol, and ecosystem (Kafka Connect, Streams, standard client libraries). It comes in two flavours — provisioned, where you size brokers, and serverless, where capacity auto-scales.

What is the difference between MSK Provisioned and MSK Serverless?

Provisioned MSK is a cluster you size: you choose broker instance types, a broker count (a multiple of your AZ count), storage, replication factor, and configs, and you pay per broker-hour whether busy or idle. Serverless removes all of that — no brokers, no storage to provision, throughput that auto-scales within published per-partition and per-cluster ceilings, IAM-only authentication, and billing by partition-hours plus storage and data. Provisioned wins at steady high utilisation; serverless wins for spiky, unpredictable, or mostly-idle workloads.

How do I authenticate to an MSK cluster?

Provisioned MSK offers four methods, each on its own port: IAM access control (SASL/OAUTHBEARER on :9098, the recommended default), SASL/SCRAM (:9096, username/password stored in AWS Secrets Manager under a customer-managed KMS key with an AmazonMSK_ name prefix), mTLS (:9094, client certificates from ACM Private CA), and plaintext/unauthenticated (:9092, dev only). MSK Serverless supports only IAM. All authenticated ports use TLS in transit, and broker storage is encrypted at rest with KMS.

What replication factor should I use in Amazon MSK?

Use replication.factor=3 across three availability zones for production, paired with min.insync.replicas=2 on the topic and acks=all on the producer. That combination keeps three copies of every partition (one per AZ) and only acknowledges a write once at least two replicas hold it, so an acknowledged record survives a full-AZ outage with zero data loss while writes keep flowing. A replication factor of 1 or acks=1 can silently lose in-flight data on a broker failure.

When should I choose MSK over self-managed Kafka, Redpanda, or WarpStream?

Choose Amazon MSK when you want real, managed Apache Kafka at low latency without owning broker operations. Choose self-managed EC2 Kafka only when you need control MSK does not expose, such as a specific version the day it ships or bespoke tuning. Choose Redpanda for the lowest, most predictable tail latency and single-binary simplicity, and choose WarpStream for high-throughput, latency-tolerant workloads where its diskless, S3-backed design (no inter-AZ replication cost) makes it dramatically cheaper.

Does Amazon MSK support Kafka Connect?

Yes — MSK Connect is a managed Kafka Connect environment. You upload a connector plugin (a JAR) to S3 as a custom plugin, supply a worker configuration, and AWS runs autoscaling Connect workers measured in MCUs. Common uses are an S3 sink to archive events as Parquet, or a Debezium source for change-data-capture from a database into Kafka. Because connectors track progress via committed consumer-group offsets, they resume cleanly from the last committed offset after a restart.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every Amazon MSK idea above, from the RF3 + min.insync.replicas durability contract to the serverless throughput ceiling and the MSK Connect S3 sink, maps to a hands-on practice room where you build the streaming logic against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you make this pipeline survive an AZ outage?" holds up under a senior interviewer's depth probes.

Practice streaming problems now →
Event-processing drills →