DEV Community

Unpublished Post. This URL is public but secret, so share at your own discretion.
Ajeet Singh Raina
Ajeet Singh Raina

Get Started with Kafka

Steps to Get Started with Apache Kafka

Step 1: Get Kafka

Download the latest Kafka release from the official Kafka website.

Extract the downloaded archive:

$ tar -xzf kafka_2.13-3.9.0.tgz
$ cd kafka_2.13-3.9.0
Enter fullscreen mode Exit fullscreen mode

Step 2: Start the Kafka Environment

Prerequisites: Ensure Java 8+ is installed on your system.

Option 1: Kafka with KRaft

  • Generate a Cluster UUID: 
$ KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
Enter fullscreen mode Exit fullscreen mode
  • Format Log Directories: 
$ bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/kraft/reconfig-server.properties
Enter fullscreen mode Exit fullscreen mode
  • Start the Kafka Server: 
$ bin/kafka-server-start.sh config/kraft/reconfig-server.properties
Enter fullscreen mode Exit fullscreen mode

Alternatively, use Docker images for KRaft:

JVM-based Kafka Image:

$ docker pull apache/kafka:3.9.0
$ docker run -p 9092:9092 apache/kafka:3.9.0
Enter fullscreen mode Exit fullscreen mode
  • GraalVM-based Kafka Image: 
$ docker pull apache/kafka-native:3.9.0
$ docker run -p 9092:9092 apache/kafka-native:3.9.0
Enter fullscreen mode Exit fullscreen mode

Option 2: Kafka with ZooKeeper

  • Start ZooKeeper: 
$ bin/zookeeper-server-start.sh config/zookeeper.properties
Enter fullscreen mode Exit fullscreen mode
  • Start Kafka Broker (in another terminal): 
$ bin/kafka-server-start.sh config/server.properties
Enter fullscreen mode Exit fullscreen mode

Step 3: Create a Topic

  • Create a topic to store events: 
$ bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode
  • Verify the topic details: 
$ bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Step 4: Write Events into the Topic

  • Run the producer client: 
$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode
  • Enter events: 
> This is my first event
> This is my second event
Enter fullscreen mode Exit fullscreen mode

Step 5: Read Events from the Topic

  • Run the consumer client (in a new terminal): 
$ bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

You should see the events you added.

Step 6: Import/Export Data with Kafka Connect

Edit the plugin.path property in config/connect-standalone.properties to include the path of connect-file-3.9.0.jar:

$ echo "plugin.path=libs/connect-file-3.9.0.jar" >> config/connect-standalone.properties
Enter fullscreen mode Exit fullscreen mode
  • Create test data: 
$ echo -e "foo\nbar" > test.txt
Enter fullscreen mode Exit fullscreen mode
  • Start connectors: 
$ bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties
Enter fullscreen mode Exit fullscreen mode
  • Verify the output: 
$ more test.sink.txt
Enter fullscreen mode Exit fullscreen mode

Step 7: Process Events with Kafka Streams

Use Kafka Streams to implement real-time applications. Example for WordCount in Java:

KStream<String, String> textLines = builder.stream("quickstart-events");
KTable<String, Long> wordCounts = textLines
    .flatMapValues(line -> Arrays.asList(line.toLowerCase().split(" ")))
    .groupBy((keyIgnored, word) -> word)
    .count();
wordCounts.toStream().to("output-topic", Produced.with(Serdes.String(), Serdes.Long()));
Enter fullscreen mode Exit fullscreen mode

Step 8: Terminate the Kafka Environment

  • Stop the producer and consumer clients using Ctrl+C.
  • Stop the Kafka broker and ZooKeeper (if used) using Ctrl+C.
  • Clean up logs: 
$ rm -rf /tmp/kafka-logs /tmp/zookeeper /tmp/kraft-combined-logs
Enter fullscreen mode Exit fullscreen mode

Top comments (0)