DEV Community

loading...
Cover image for Setting Up Kafka in Docker

Setting Up Kafka in Docker

Constantine
web dev, kendo, yoga
・3 min read

"Here we go, another article on setting up some thing in Docker πŸ˜’", you might think. This time it's Kafka. And I wouldn't write about it if it was that trivial.

Thanks to wurstmeister, we have separate docker images for Kafka and Zookeeper. But setting this up my way without docker compose is a task in and of itself.

Zookeper

We'll need two volumes for this guy. You know how it goes, go to "Volumes" side-menu and smash the "Add volume" button. Name the first one zookeeper_data, and the second one - zookeeper_conf.

Next go to the "Containers" side menu, click the "Add container" button. Set:

  • name: zookeeper
  • image: wurstmeister/zookeeper:latest
  • publish port: 2181:2181
  • add volumes:
    • /opt/zookeeper-3.4.13/data to zookeeper_data
    • /opt/zookeeper-3.4.13/conf to zookeeper_conf

As always, a handy screenshot for you:
Zookeeper create container

Now destroy that "Deploy the container" button and wait a little.

Kafka

This guy will need just one volume. Name it kafka_volume. Now pay attention when creating a container. Set:

  • name: kafka
  • image: wurstmeister/kafka:latest
  • publish port: 9092:9092
  • add volumes:
    • /kafka to kafka_data
    • bind /var/run/docker.sock to /var/run/docker.sock
  • env:
    • KAFKA_ADVERTISED_HOST_NAME: 172.17.0.1
    • KAFKA_ZOOKEEPER_CONNECT: 172.17.0.1:2181
    • ZK: 172.17.0.1:2181
    • HOST: 172.17.0.1:9092
    • KAFKA_ADVERTISED_PORT: 9092

ZK and HOST envs are just for shortening commands later on.

Screenshot for sanity check:
Kafka create container

And deploy.

Communicate

There is already a guide on how to use this stuff, but there is a mistake in the consuming part.

Create topic

But let's start with creating a topic. Go to ">_ Console" of the Kafka container. Or execute command below in your terminal session

docker exec -it kafka /bin/bash
Enter fullscreen mode Exit fullscreen mode

Once inside container create topic just as in above mentioned guide:

kafka-topics.sh --create --topic topic \
--partitions 4 --zookeeper $ZK --replication-factor 1
kafka-topics.sh --describe --topic topic --zookeeper $ZK
Enter fullscreen mode Exit fullscreen mode

You'll see messages about topic creation and it's description:

Created topic topic.

Topic: topic PartitionCount: 4 ReplicationFactor: 1 Configs:
Topic: topic Partition: 0 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: topic Partition: 1 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: topic Partition: 2 Leader: 1002 Replicas: 1002 Isr: 1002
Topic: topic Partition: 3 Leader: 1002 Replicas: 1002 Isr: 1002
Enter fullscreen mode Exit fullscreen mode

Consumer

The guide says to run this:

kafka-console-consumer.sh --topic=topic --zookeeper=$ZK
Enter fullscreen mode Exit fullscreen mode

But if you try to do that you'll get an error about zookeeper is not a recognized option.

So the correct command is this:

kafka-console-consumer.sh --topic=topic --bootstrap-server=$HOST
Enter fullscreen mode Exit fullscreen mode

It didn't freeze, it just waits for messages to come.

Producer

Open another console or terminal session and connect to the kafka container. Execute:

kafka-console-producer.sh --topic=topic \
--broker-list=`broker-list.sh`
Enter fullscreen mode Exit fullscreen mode

Now you can type some messages here and it'll show in another session where we have a consumer πŸš€

Bonus

Inside kafka_app in the project repo, you'll find two short files with producer and consumer. This is just to show how we can use Kafka in Python:

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python consumer.py
Enter fullscreen mode Exit fullscreen mode

In another terminal session:

source venv/bin/activate
python producer.py
Enter fullscreen mode Exit fullscreen mode

Now look at the terminal with our python consumer and you'll see a topic, message key and value.

Also, if you didn't close previous sessions with the console consumer, you'll see the same message there. And publishing from the console producer will deliver a message to both console and python consumers.


Now you have one more tool in your toolbox! Go use this stuff πŸ’ͺ

Discussion (2)

Collapse
hiepxanh profile image
hiepxanh

nice walkthrough, the best tutorial at the moment on the internet about kafka docker portcontainer

Collapse
c_v_ya profile image
Constantine Author

thank you, I'm glad it was useful