Running Kafka with Docker Compose

Confluent is the primary sponsor of Apache Kafka and not surprisingly they provide great Docker images for Kafka (and its dependency, ZooKeeper) via their Confluent Platform.

The following docker-compose.yml file spins up a single-node Kafka "cluster", but the INTERNAL/EXTERNAL bits enable multi-node clusters to be declared with a small amount of changes.

version: "3"

services:
  zk:
    image: confluentinc/cp-zookeeper:${CP_VERSION:-5.5.3}
    environment:
      JAVA_TOOL_OPTIONS: -Xmx256m
      ZOOKEEPER_CLIENT_PORT: 2181
    volumes:
      - zk:/var/lib/zookeeper/data
  kafka:
    image: confluentinc/cp-kafka:${CP_VERSION:-5.5.3}
    environment:
      JAVA_TOOL_OPTIONS: -Xmx512m
      KAFKA_ZOOKEEPER_CONNECT: zk:2181
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:9092,INTERNAL://kafka:9192
      KAFKA_LISTENERS: EXTERNAL://0.0.0.0:9092,INTERNAL://0.0.0.0:9192
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_RETENTION_HOURS: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      # some loggers are set to TRACE by default...and they're noisy
      KAFKA_LOG4J_LOGGERS: kafka.controller=INFO,state.change.logger=INFO
    volumes:
      - kafka:/var/lib/kafka/data
    ports:
      - 9092:9092
    depends_on:
      - zk

volumes:
  zk:
  kafka:

Docker Compose also makes it easy to exec the usual helper scripts provided by the image, such as listing topics:

docker-compose exec kafka \
  kafka-topics --bootstrap-server localhost:9092 \
  --list

and consuming from a topic:

docker-compose exec kafka \
  kafka-console-consumer --bootstrap-server localhost:9092 \
  --topic your-topic

Confluent has extensive documentation on the images used here.

Also, make sure to update or pass in the CP_VERSION appropriate from the newest tags published.