Learn Kafka by Doing: Build a 3-Broker Kafka Cluster with Docker Compose

Overview

This configuration deploys a 3-broker Kafka cluster with automatic failover, data replication, and a management UI.

---

Docker Compose Configuration

services:
  kafka1:
    image: apache/kafka:latest
    container_name: kafka1
    ports:
      - "9093:9093"
      - "29092:29092"
    volumes:
      - ./kafka-data/kafka1:/var/lib/kafka/data
    networks:
      - kafka-bridge
    environment:
      KAFKA_BROKER_ID: 1
      CLUSTER_ID: t7jxO1XIQwWtRhIzV5PE4w
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:29092,EXTERNAL://localhost:9093
      KAFKA_LISTENERS: CONTROLLER://:9092,EXTERNAL://0.0.0.0:9093,INTERNAL://:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data

  kafka2:
    image: apache/kafka:latest
    container_name: kafka2
    ports:
      - "9094:9093"
    volumes:
      - ./kafka-data/kafka2:/var/lib/kafka/data
    networks:
      - kafka-bridge
    environment:
      KAFKA_BROKER_ID: 2
      CLUSTER_ID: t7jxO1XIQwWtRhIzV5PE4w
      KAFKA_PROCESS_ROLES: broker
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:29092,EXTERNAL://localhost:9094
      KAFKA_LISTENERS: EXTERNAL://0.0.0.0:9093,INTERNAL://:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data

  kafka3:
    image: apache/kafka:latest
    container_name: kafka3
    ports:
      - "9095:9093"
    volumes:
      - ./kafka-data/kafka3:/var/lib/kafka/data
    networks:
      - kafka-bridge
    environment:
      KAFKA_BROKER_ID: 3
      CLUSTER_ID: t7jxO1XIQwWtRhIzV5PE4w
      KAFKA_PROCESS_ROLES: broker
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:29092,EXTERNAL://localhost:9095
      KAFKA_LISTENERS: EXTERNAL://0.0.0.0:9093,INTERNAL://:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8000:8080"
    networks:
      - kafka-bridge
    environment:
      KAFKA_CLUSTERS_0_NAME: kafkamultinodecluster
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka1:29092,kafka2:29092,kafka3:29092"
    depends_on:
      - kafka1
      - kafka2
      - kafka3

networks:
  kafka-bridge:
    driver: bridge

---

Key Configuration Breakdown

Port Strategy

• 9093-9095: External access (your apps connect here)
• 29092: Internal broker communication
• 9092: Controller coordination
• 8000: Kafka UI web interface

Listeners Explained

CONTROLLER (9092) - Controllers coordinate cluster operations
INTERNAL (29092)  - Brokers communicate with each other
EXTERNAL (9093+)  - External clients connect

Critical Environment Variables

CLUSTER_ID: Must be identical across all brokers (forms the cluster)

BROKER_ID: Unique identifier per broker (1, 2, 3)

PROCESS_ROLES:

• kafka1: broker,controller (both roles)
• kafka2/3: broker (worker only)

CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092 (only kafka1 can be controller)

ADVERTISED_LISTENERS:

• INTERNAL://kafka1:29092 - Used by other brokers
• EXTERNAL://localhost:9093 - Used by external clients

Replication Factors:

• KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 - Consumer offsets replicated 3x
• KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 - Transaction logs replicated 3x
• KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 - Minimum in-sync replicas (trade-off between availability and durability)

Data Persistence

./kafka-data/kafka1 → /var/lib/kafka/data (in container)
./kafka-data/kafka2 → /var/lib/kafka/data
./kafka-data/kafka3 → /var/lib/kafka/data

Data survives container restarts.

---

Producer Script

producer-read.sh:

#!/bin/bash
BOOTSTRAP_SERVERS="kafka2:29092"
TOPIC_NAME=$1
CONTAINER_NAME="kafka1"

if [ -z "$TOPIC_NAME" ]; then
  echo "Usage: $0 <topic-name>"
  exit 1
fi

for i in {1..10}; do
  MESSAGE="Message $i from producer"
  docker exec -it $CONTAINER_NAME bash -c "echo \"$MESSAGE\" | /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server $BOOTSTRAP_SERVERS --topic $TOPIC_NAME"
  echo "Sent: $MESSAGE"
done

Usage: ./producer-read.sh customer-orders

Sends 10 messages to the specified topic through kafka2, running from kafka1 container.

---

Consumer Script

consumer-read.sh:

#!/bin/bash
BOOTSTRAP_SERVERS="kafka1:29092"
TOPIC_NAME=$1
CONTAINER_NAME="kafka2"

if [ -z "$TOPIC_NAME" ]; then
  echo "Usage: $0 <topic-name>"
  exit 1
fi

docker exec -it $CONTAINER_NAME bash -c "/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server $BOOTSTRAP_SERVERS --topic $TOPIC_NAME --from-beginning"

Usage: ./consumer-read.sh customer-orders

Reads all messages from the beginning, running from kafka2 container, connecting to kafka1.

---

Quick Start

1. Start cluster: docker-compose up -d
2. Access UI: http://localhost:8000
3. Send messages: ./producer-read.sh my-topic
4. Read messages: ./consumer-read.sh my-topic
5. Stop cluster: docker-compose down

---

Architecture Summary

┌─────────────────────────────────────────────┐
│  kafka1 (broker + controller)               │
│  Ports: 9093 (external), 29092 (internal)   │
└─────────────────────────────────────────────┘
                    ↕
┌─────────────────────────────────────────────┐
│  kafka2 (broker)                            │
│  Ports: 9094 (external), 29092 (internal)   │
└─────────────────────────────────────────────┘
                    ↕
┌─────────────────────────────────────────────┐
│  kafka3 (broker)                            │
│  Ports: 9095 (external), 29092 (internal)   │
└─────────────────────────────────────────────┘
                    ↕
┌─────────────────────────────────────────────┐
│  kafka-ui (management interface)            │
│  Port: 8000                                 │
└─────────────────────────────────────────────┘

All brokers share the same CLUSTER_ID and communicate on the kafka-bridge network.