Fast apache kafka local deployment

Hey, dev.to!

If you are using apache kafka, you know that local deployment required a lot of things. You need to download zookeeper, apache kafka, edit configs and then run them. And, of cause, it takes time. The detailed description you can find in this article https://dev.to/lehauchicha/spring-boot-application-with-apache-kafka-299n. But is it possible to speed up the local deployment process of apache kafka ?

The answer is yes!
In this article we will deploy apache kafka locally using docker-compose. All you need is download and install docker from official site: https://www.docker.com/get-started

Lets create docker-compose.yml file:

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

As you can see here we specify images for zookeeper and kafka with ports. That's it!

Lets run kafka, using docker-compose:

docker-compose up -d

you will see such result:

Now kafka ready for usage in your applications!

One more thing, for convenient configuring kafka topics, I would like to suggest UI Tool for Kafka - Offset Explorer, which you can find by link https://kafkatool.com/download.html.

For connection to deployed kafka you need install tool and add connection parameters here on properties tab:

and here on advanced tab:

That's it. Thank you for reading!

Comments

[Trym]

Another UI tool which is completely web based.
github.com/obsidiandynamics/kafdrop

[Saurabh Bomble]

This is the best one🙌

[LehaUchicha]

Hey, Trym!
Looks interesting, thank you!

[Mehmet]

Why don't you try upstash? The free version will suffice.

upstash.com/

[LehaUchicha]

Hey, Mehmet!
I just don't heard about upstash, so thank you for mention, I will try)

[vivekEra]

Can we connect kafka connector with any db to read or write the data?(Not via producer
)

[LehaUchicha]

Yes, sure. Here you can read some information about connectors: docs.confluent.io/platform/current....
You can configure source connector(for example jdbs source connector) and sink connector(For example, elastic search sink connector)