Installing Apache Kafka (Confluent edition)

Table of Contents

• Download
• 1. Start zookeeper
• 2. Create a Kafka Broker
• 3. Create a Kafka topic
  • Listing the topics
• 4. Create a Kafka Producer
• 5. Create a Kafka Consumer
• LOGS (DEBUG)

Download

curl -O http://packages.confluent.io/archive/5.4/confluent-community-5.4.0-2.12.zip

1. Start zookeeper

Navigate to the folder where confluent is installed. For me it was /home/divyaksh/Downloads/software/confluent-5.4.0

Linux

bin/zookeeper-server-start etc/kafka/zookeeper.properties

Windows

bin\windows\zookeeper-server-start.bat etc\kafka\zookeeper.properties

Error

Occurs if something is missing. I don't know what do do if this happens in linux.

Classpath is empty. Please build the project first using 'gradlew jarAll'

Solution

• Go to bin/kafka-run-class
• Add some code. Go to a line above this script (do not delete anything)

rem Classpath addition for core

add the below code

rem Classpath addition for LSB style path
if exist %BASE_DIR%\share\java\kafka\* (
    call :concat %BASE_DIR%\share\java\kafka\*
)

2. Create a Kafka Broker

Linux

bin/kafka-server-start etc/kafka/server.properties

Windows

bin\windows\kafka-server-start.bat etc\kafka\server.properties

3. Create a Kafka topic

Linux

bin/kafka-topics --create --topic sampleTopic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092

Windows

bin\windows\kafka-topics.bat --create --topic sampleTopic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092

Warning

WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both

Thus, create topics with names not haveing . or _.

--partition n:

• Number of partitions
  • How much storage is needed (how big my message is)
  • Nature of parallel processing

Here we have a simple workload of a single producer and a single consumer (n = 1)

--replication-factor f : number of copies of each partition in the cluster. You create multiple brokers and a in each you make a copy of the messages. Higher the replication factor, more number of copies. No sense of keeping f=10 when there is only 1 broker. in our example f = 1

--bootstrap-server: it takes 2 arguments ip-address and port-number. In our example we are running the zookeeper on local machine and the default port number of zookeeper is 9092.

Listing the topics

$ bin/kafka-topics --list --bootstrap-server localhost:9092
__confluent.support.metrics
sampleTopic
sample_topic

4. Create a Kafka Producer

Linux

bin/kafka-console-producer --topic sampleTopic --broker-list localhost:9092  < <path to some file>

Windows

bin\windows\kafka-console-producer.bat --topic sampleTopic --broker-list localhost:9092 < <path to some file>

--broker-list: same as --bootstrap-server.

NOTICE

$ bin/kafka-console-producer --help
This tool helps to read data from standard input and publish it to Kafka.

So you have to input the file using < operator before giving <path to some file>

5. Create a Kafka Consumer

Linux

bin/kafka-console-consumer --topic sampleTopic --bootstrap-server localhost:9092 -from-beginning

Windows

bin\windows\kafka-console-consumer.bat --topic sampleTopic --bootstrap-server localhost:9092 -from-beginning

OUTPUT

$ bin/kafka-console-consumer --topic sampleTopic --bootstrap-server localhost:9092 -from-beginning
To: [e-mail]
From: [e-mail]
Subject: This is a test email from msmtp

MSMTP says hello

I had given a test email file as an input to the producer.

LOGS (DEBUG)

You can find the logs at /tmp/kafka-logs/ to debug kafka. Windows C:\tmp\kafka-logs\