DEV Community

loading...

Running Kafka with Docker Compose

Geoff Bourne
I am a fullstack developer with a focus on backend in Java with Spring, React, Docker
Updated on ・1 min read

Confluent is the primary sponsor of Apache Kafka and not surprisingly they provide great Docker images for Kafka (and its dependency, ZooKeeper) via their Confluent Platform.

The following docker-compose.yml file spins up a single-node Kafka "cluster", but the INTERNAL/EXTERNAL bits enable multi-node clusters to be declared with a small amount of changes.

version: "3"

services:
  zk:
    image: confluentinc/cp-zookeeper:${CP_VERSION:-5.5.3}
    environment:
      JAVA_TOOL_OPTIONS: -Xmx256m
      ZOOKEEPER_CLIENT_PORT: 2181
    volumes:
      - zk:/var/lib/zookeeper/data
  kafka:
    image: confluentinc/cp-kafka:${CP_VERSION:-5.5.3}
    environment:
      JAVA_TOOL_OPTIONS: -Xmx512m
      KAFKA_ZOOKEEPER_CONNECT: zk:2181
      KAFKA_ADVERTISED_LISTENERS: EXTERNAL://localhost:9092,INTERNAL://kafka:9192
      KAFKA_LISTENERS: EXTERNAL://0.0.0.0:9092,INTERNAL://0.0.0.0:9192
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_RETENTION_HOURS: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      # some loggers are set to TRACE by default...and they're noisy
      KAFKA_LOG4J_LOGGERS: kafka.controller=INFO,state.change.logger=INFO
    volumes:
      - kafka:/var/lib/kafka/data
    ports:
      - 9092:9092
    depends_on:
      - zk

volumes:
  zk:
  kafka:
Enter fullscreen mode Exit fullscreen mode

Docker Compose also makes it easy to exec the usual helper scripts provided by the image, such as listing topics:

docker-compose exec kafka \
  kafka-topics --bootstrap-server localhost:9092 \
  --list
Enter fullscreen mode Exit fullscreen mode

and consuming from a topic:

docker-compose exec kafka \
  kafka-console-consumer --bootstrap-server localhost:9092 \
  --topic your-topic
Enter fullscreen mode Exit fullscreen mode

Confluent has extensive documentation on the images used here.

Also, make sure to update or pass in the CP_VERSION appropriate from the newest tags published.

Discussion (0)