Multi node KRaft based Kafka cluster with monitoring.

As of Kafka 7.5 Zookeeper is deprecated as an option to start new kafka cluster and default strategy is to use the KRaft to have a zookeeper less kafka.

As of now there is not much available online over how to setup this new style of Kafka cluster without zookeeper. Even ChatGPT doesn't know if you were to ask it, see below

Most of the examples that you will find online are from 7.0 to 7.2 which required you to run some scripts to not only generate a cluster id before the cluster starts but also run a pre hook script to disable or remove zookeeper options from the configs before the cluster would come online and not check for zookeeper.

Many examples use bitnami/kafka image to show how to setup a kraft based examples but I found them to be inconsistent in their setup and use non standard flags.

ChatGPT use wurstmeister/kafka-docker in its example so I decided to check it but its based on zookeeper. Nonetheless it is quite good to know what you need to setup a kafka cluster as the settings are still the same and only zookeeper is no more in the kraft mode.

For Kafka-UI I used provectus/kafka-ui which has a lot of good examples over how to setup kafka-ui with a kraft based kafka cluster but they are all pre 7.5.x where you required lot of tinkering before you can have the cluster up and running.

To setup monitoring kafka cluster I refered to this amazing repo streamthoughts/kafka-monitoring-stack-docker-compose that has detailed examples over how to use jmx-exporter to have the metrics available for prometheus scrapping and highly recommend to refer to it over other information available which are not in depth

One thing to take note of is for volume mapping Kafka requires that you create a folder and its owner is set to 1000:1000 before you can crate the mapping for your container.

Below is the config to setup a 2 node (leaving the 3rd node as an exercise) cluster with monitoring and UI to monitor your cluster. Not adding grafana and prometheus to the post as that is out of the scope. To try out you can use Grafana cloud before setting it up on your own

#KAFAK 10.0.3.52
  kafka_00:
    image: "confluentinc/cp-kafka:7.5.1"
    hostname: "kafak_00"
    user: "0:0"
    container_name: kakashi_kafka_00
    networks:
      network:
        ipv4_address: 10.0.3.52
        aliases:
          - kakashi_kafka_00
    expose:
      - "9092"
      - "9093"
      - "9094"
      - "9101"
    volumes:
      - "/opt/kafka-00-data:/var/lib/kafka/data"
    environment:
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
      - KAFKA_BROKER_ID=0
      - KAFKA_ADVERTISED_LISTENERS=INTERNAL://kafka_00:9092,EXTERNAL://kafka_00:9094
      - KAFKA_CONTROLLER_QUORUM_VOTERS=0@kafka_00:9093,1@kafka_01:9093
      - KAFKA_DEFAULT_REPLICATION_FACTOR=2
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT
      - KAFKA_LISTENERS=CONTROLLER://kafka_00:9093,EXTERNAL://kafka_00:9094,INTERNAL://kafka_00:9092
      - KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
      - KAFKA_NODE_ID=0
      - KAFKA_NUM_PARTITIONS=2
      - KAFKA_PROCESS_ROLES=broker,controller
      - KAFKA_ENABLE_KRAFT=yes
      - CLUSTER_ID=45vUBEgyQjSywv_811biQw
      - KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0
      - KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1
      - KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
      - KAFKA_JMX_PORT=9101
      - KAFKA_JMX_HOSTNAME=kafka_00
      - KAFAK_JMX_OPTS=-Djava.rmi.server.hostname=kafka_00 -Djava.rmi.server.port=9101 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false
    cpu_shares: 64
    mem_reservation: 1024m
    mem_limit: 1512m
    restart: unless-stopped

  #KAFAK_01 10.0.3.54
  kafka_01:
    image: "confluentinc/cp-kafka:7.5.1"
    hostname: "kafka_01"
    user: "0:0"
    container_name: kakashi_kafka_01
    networks:
      network:
        ipv4_address: 10.0.3.54
        aliases:
          - kakashi_kafka_01
    expose:
      - "9092"
      - "9093"
      - "9094"
      - "9101"
    volumes:
      - "/opt/kafka-01-data:/var/lib/kafka/data"
    environment:
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
      - KAFKA_BROKER_ID=1
      - KAFKA_ADVERTISED_LISTENERS=INTERNAL://kafka_01:9092,EXTERNAL://kafka_01:9094
      - KAFKA_CONTROLLER_QUORUM_VOTERS=0@kafka_00:9093,1@kafka_01:9093
      - KAFKA_DEFAULT_REPLICATION_FACTOR=2
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT
      - KAFKA_LISTENERS=CONTROLLER://kafka_01:9093,EXTERNAL://kafka_01:9094,INTERNAL://kafka_01:9092
      - KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
      - KAFKA_NODE_ID=1
      - KAFKA_NUM_PARTITIONS=2
      - KAFKA_PROCESS_ROLES=broker,controller
      - KAFKA_ENABLE_KRAFT=yes
      - CLUSTER_ID=45vUBEgyQjSywv_811biQw
      - KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0
      - KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1
      - KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
      - KAFKA_JMX_PORT=9101
      - KAFKA_JMX_HOSTNAME=kafka_01
      - KAFAK_JMX_OPTS=-Djava.rmi.server.hostname=kafka_01 -Djava.rmi.server.port=9101 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false
    cpu_shares: 64
    mem_reservation: 1024m
    mem_limit: 1512m
    restart: unless-stopped

  #KAFAK_UI 10.0.3.56
  kafka_ui:
    image: "provectuslabs/kafka-ui:53a6553765a806eda9905c43bfcfe09da6812035"
    container_name: kakashi_kafka_ui
    networks:
      network:
        ipv4_address: 10.0.3.56
        aliases:
          - kakashi_kafka_ui
    depends_on:
      - kafka_00
      - kafka_01
    expose:
      - "8080"
    environment:
      - AUTH_TYPE=LOGIN_FORM
      - DYNAMIC_CONFIG_ENABLED=true
      - SPRING_SECURITY_USER_NAME=${KAFKA_UI_USERNAME}
      - SPRING_SECURITY_USER_PASSWORD=${KAFKA_UI_PASSWORD}
      - SERVER_SERVLET_CONTEXT_PATH=/ka5ka-u1
      - KAFKA_CLUSTERS_0_NAME=kakashi-kafka-00
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka_00:9094
      - KAFKA_CLUSTERS_0_METRICS_PORT=9101
      - KAFKA_CLUSTERS_1_NAME=kakashi-kafka-01
      - KAFKA_CLUSTERS_1_BOOTSTRAPSERVERS=kafka_01:9094
      - KAFKA_CLUSTERS_1_METRICS_PORT=9101
    cpu_shares: 64
    mem_reservation: 512m
    mem_limit: 786m
    restart: unless-stopped

  #KAFKA_00_JMX 10.0.3.58
  kafka_00_jmx:
    image: "sscaling/jmx-prometheus-exporter:0.12.0"
    container_name: kakashi_kafka_00_jmx
    networks:
      network:
        ipv4_address: 10.0.3.58
        aliases:
          - kakashi_kafka_00_jmx
    depends_on:
      - kafka_00
    expose:
     - "5556"
    environment:
      - CONFIG_YML=/etc/jmx_exporter/config.yml
      - JVM_OPTS=-Xmx128M
    volumes:
     - ./kafka_jmx_exporter/config/jmx_exporter/config_kafka_00.yml:/etc/jmx_exporter/config.yml
    cpu_shares: 64
    mem_reservation: 196m
    mem_limit: 256m
    restart: unless-stopped


  #KAFKA_01_JMX 10.0.3.60
  kafka_01_jmx:
    image: "sscaling/jmx-prometheus-exporter:0.12.0"
    container_name: kakashi_kafka_01_jmx
    networks:
      network:
        ipv4_address: 10.0.3.60
        aliases:
          - kakashi_kafka_01_jmx
    depends_on:
      - kafka_01
    expose:
     - "5556"
    environment:
      - CONFIG_YML=/etc/jmx_exporter/config.yml
      - JVM_OPTS=-Xmx128M
    volumes:
     - ./kafka_jmx_exporter/config/jmx_exporter/config_kafka_01.yml:/etc/jmx_exporter/config.yml
    cpu_shares: 64
    mem_reservation: 196m
    mem_limit: 256m
    restart: unless-stopped

Kafka without zookeeper is surely a big step forward and easier to manage.