DEV Community

nk sk
nk sk

Posted on

🧑‍💻 Apache Kafka CLI – Detailed Course

#cli #dataengineering #tutorial

1. Introduction

  • Kafka provides CLI tools (usually inside bin/ or bin/windows/) to manage:

    • Brokers (the Kafka servers)
    • Topics (data channels)
    • Producers (apps sending data)
    • Consumers (apps reading data)
    • Consumer groups
    • Cluster configuration & monitoring

📂 Default path:

  • Linux/Mac: kafka/bin/
  • Windows: kafka/bin/windows/

2. Setup & Starting Kafka

Before using CLI, ensure ZooKeeper (if < 2.8) and Kafka broker are running.

# Start Zookeeper (older versions only)
bin/zookeeper-server-start.sh config/zookeeper.properties

# Start Kafka broker
bin/kafka-server-start.sh config/server.properties
Enter fullscreen mode Exit fullscreen mode

👉 In newer versions (Kafka KRaft mode), ZooKeeper is not required.

3. Topic Management

Create a Topic 

bin/kafka-topics.sh --create \
  --topic first-topic \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 1
Enter fullscreen mode Exit fullscreen mode

List Topics 

bin/kafka-topics.sh --list --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Describe a Topic 

bin/kafka-topics.sh --describe \
  --topic first-topic \
  --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Delete a Topic 

bin/kafka-topics.sh --delete \
  --topic first-topic \
  --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

4. Producing Messages

Simple Producer 

bin/kafka-console-producer.sh \
  --broker-list localhost:9092 \
  --topic first-topic
Enter fullscreen mode Exit fullscreen mode
  • Type messages directly → Press Enter → Sent to Kafka.

With Keys 

bin/kafka-console-producer.sh \
  --broker-list localhost:9092 \
  --topic first-topic \
  --property "parse.key=true" \
  --property "key.separator=:"
Enter fullscreen mode Exit fullscreen mode

Then:

user1:hello
user2:hi
Enter fullscreen mode Exit fullscreen mode

5. Consuming Messages

Simple Consumer 

bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic first-topic \
  --from-beginning
Enter fullscreen mode Exit fullscreen mode

Consumer with Group 

bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic first-topic \
  --group my-group
Enter fullscreen mode Exit fullscreen mode

6. Consumer Group Management

List Groups 

bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 --list
Enter fullscreen mode Exit fullscreen mode

Describe a Group 

bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe --group my-group
Enter fullscreen mode Exit fullscreen mode

Reset Offsets 

bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --group my-group \
  --topic first-topic \
  --reset-offsets --to-earliest --execute
Enter fullscreen mode Exit fullscreen mode

7. Cluster & Broker Utilities

Cluster Info 

bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Broker Config 

bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type brokers --entity-name 0 --describe
Enter fullscreen mode Exit fullscreen mode

8. Monitoring & Debugging

Check Lag 

bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --group my-group --describe
Enter fullscreen mode Exit fullscreen mode

List Partitions & Leaders 

bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --topic first-topic --describe
Enter fullscreen mode Exit fullscreen mode

Dump Log Segments (advanced debugging) 

bin/kafka-dump-log.sh --files /tmp/kafka-logs/first-topic-0/00000000000000000000.log
Enter fullscreen mode Exit fullscreen mode

9. Admin CLI (Newer Versions)

Kafka now has kafka-cluster.sh, kafka-storage.sh, kafka-metadata-quorum.sh for KRaft mode:

bin/kafka-storage.sh info -c config/kraft/server.properties
Enter fullscreen mode Exit fullscreen mode

10. Hands-On Practice Plan

✅ Day 1 – Install Kafka, start broker, create/delete/list topics
✅ Day 2 – Use producer & consumer CLI, send & read messages
✅ Day 3 – Experiment with consumer groups, offsets, reset-offsets
✅ Day 4 – Explore configs, broker info, and monitor lag
✅ Day 5 – Debug using dump-log & move to advanced KRaft commands

Top comments (0)