I wanted to run kafka on my raspberry pi 4 as i wanted to do a POC on an IoT project. Here are the steps on how to install it. I use SSH to login to my Rpi.
1. Install Java:
- Run the following command to install it:
$ sudo apt update
$ sudo apt install default-jdk
- Check version to test if the installation is successful. It should show you the version installed as shown below.
$ java --version
openjdk 11.0.8 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-post-Raspbian-1deb10u1)
OpenJDK Server VM (build 11.0.8+10-post-Raspbian-1deb10u1, mixed mode)
2. Install Kafka:
- Download kafka. Goto to https://apachemirror.sg.wuchna.com/kafka/ site and select the latest version. At the time that i was writing this post the latest is 2.6.0.
$ wget https://apachemirror.sg.wuchna.com/kafka/2.6.0/kafka_2.12-2.6.0.tgz
- Extract Kafka. Using the tar command. Once extracted, go inside the directory and create /data folder with /kafka and /zookeeper subfolders.
$ tar -xzf kafka_2.12-2.6.0.tgz
$ cd kafka_2.12-2.6.0
~/kafka_2.12-2.6.0 $ mkdir data
~/kafka_2.12-2.6.0/data $ mkdir kafka
~/kafka_2.12-2.6.0/data $ mkdir zookeeper
- Edit config/zookeeper.properties
$ sudo nano kafka_2.12-2.6.0/config/zookeeper.properties
Find and change dataDir value to /home/pi/kafka_2.12-2.6.0/data/zookeeper.
dataDir=/home/pi/kafka_2.12-2.6.0/data/zookeeper
Press Ctrl+X then Y to save it.
- Edit config/server.properties
$ sudo nano kafka_2.12-2.6.0/config/server.properties
Find and Change listeners value to PLAINTEXT://{your_ip_address}:9092. Ensure to replace {your_ip_address} with your device ip address.
listeners=PLAINTEXT://{your_ip_address}:9092
Next is to locate and update log.dirs value to /home/pi/kafka_2.12-2.6.0/data/kafka.
log.dirs=/home/pi/kafka_2.12-2.6.0/data/kafka
3. Test Kafka
- Start Zookeeper
~/kafka_2.12-2.6.0 $ bin/zookeeper-server-start.sh config/zookeeper.properties
- Start Kafka
~/kafka_2.12-2.6.0 $ bin/kafka-server-start.sh config/server.properties
- Create a topic
~/kafka_2.12-2.6.0 $ bin/kafka-topics.sh --create --bootstrap-server {your_ip_address}:9092 --replication-factor 1 --partitions 1 --topic TestTopic
It will show you the message below as reponse for topic created.
Created topic TestTopic.
- Check for the created topic
~/kafka_2.12-2.6.0 $ bin/kafka-topics.sh --list --bootstrap-server {your_ip_address}:9092
This will show you the newly created topic
TestTopic
- Start Kafka Producer
~/kafka_2.12-2.6.0 $ bin/kafka-console-producer.sh --broker-list {your_ip_address}:9092 --topic TestTopic
- Start Kafka Consumer
~/kafka_2.12-2.6.0 $ bin/kafka-console-consumer.sh --bootstrap-server {your_ip_address}:9092 --topic TestTopic
Once both Producer and Consumer is up and running start typing on the producer window. Each message should show in the consumer window.
Top comments (2)
Hello from 2021. I created an account here just to say thanks... but also Start Kafka Consumer should listen to the same topic as the producer, you've got TestTopic and NewTopic. Thanks again, this really helped.
Hey, thanks man, appreciate it. I have updated my post.