DEV Community

Cover image for Installing Apache Kafka 4.2 on Ubuntu (WSL2): A Complete KRaft Step-by-Step Guide
Sanjay Ghosh
Sanjay Ghosh

Posted on

Installing Apache Kafka 4.2 on Ubuntu (WSL2): A Complete KRaft Step-by-Step Guide

#kafka #java #eventdriven #devops

Installing Apache Kafka 4.2 on Ubuntu 24.04 (WSL2) Using KRaft Mode: A Complete Step-by-Step Guide

Learn how to install Apache Kafka 4.2 in KRaft mode, understand its architecture, create topics, produce and consume messages, and troubleshoot common configuration issues—all without ZooKeeper.

🚀 Introduction

Apache Kafka has become the de facto standard for building event-driven, real-time, and high-throughput applications. Whether you're processing millions of financial transactions, collecting application logs, streaming IoT sensor data, or connecting microservices, Kafka provides a scalable and reliable messaging platform.

Until recently, setting up Kafka required running Apache ZooKeeper alongside Kafka brokers. While powerful, ZooKeeper added operational complexity and introduced another distributed system that administrators had to manage.

Beginning with recent Kafka releases, KRaft (Kafka Raft Metadata mode) removes this dependency by allowing Kafka to manage its own metadata internally. This makes installation simpler, reduces operational overhead, and improves scalability.

In this guide, we'll install Apache Kafka 4.2 on Ubuntu 24.04.4 LTS (WSL2), configure a single-node KRaft cluster, and walk through the complete lifecycle:

  • Installing Kafka
  • Understanding the Kafka architecture
  • Configuring KRaft mode
  • Starting the broker
  • Creating topics
  • Producing and consuming messages
  • Troubleshooting common issues
  • Understanding the purpose of each configuration parameter

Rather than simply listing commands, I'll explain why each step is necessary so that you understand how Kafka works under the hood.

What is Apache Kafka?

Apache Kafka is a distributed event streaming platform designed to move data reliably and efficiently between applications.

Instead of applications communicating directly with each other, they communicate through Kafka.

A producing application writes messages to Kafka.

Kafka stores those messages reliably.

One or more consuming applications read those messages whenever they need them.

This decouples applications from one another, allowing them to scale independently.

For example, imagine an online shopping application.

Customer Places Order
          │
          ▼
      Order Service
          │
          ▼
       Kafka Topic
      /      |      \
     ▼       ▼       ▼
 Inventory  Billing  Shipping
Enter fullscreen mode Exit fullscreen mode

The Order Service publishes a single event.

Inventory updates stock.

Billing processes the payment.

Shipping prepares the shipment.

Each service works independently without knowing about the others.

This is one of Kafka's greatest strengths.

Kafka Architecture

The following diagram illustrates the overall architecture of Apache Kafka running in KRaft mode.

(Insert the cover architecture image here.)

The architecture consists of three primary components:

Producers

Producers publish messages into Kafka.

Examples include:

  • Java applications
  • Spring Boot microservices
  • Python applications
  • IoT devices
  • Web applications
  • Log collection systems

A producer simply sends data—it doesn't need to know who will consume it.

Kafka Cluster

The Kafka cluster is responsible for:

  • Receiving messages
  • Persisting messages to disk
  • Replicating data
  • Managing topics
  • Managing partitions
  • Delivering messages to consumers

In KRaft mode, the cluster also manages its own metadata through the Controller.

Consumers

Consumers subscribe to topics and process messages.

Examples include:

  • Analytics applications
  • Fraud detection services
  • Notification systems
  • Machine Learning pipelines
  • Search indexing services

Multiple consumers can read the same topic independently.

Core Kafka Concepts

Before installing Kafka, it's helpful to understand a few important concepts.

Broker

A Broker is a Kafka server.

It receives messages from producers and stores them on disk.

A Kafka cluster may contain one broker or hundreds of brokers.

For this tutorial, we'll use a single broker.

Topic

A Topic is a logical category used to organize messages.

Examples:

orders

payments

customer-events

application-logs
Enter fullscreen mode Exit fullscreen mode

Applications publish messages to topics.

Consumers subscribe to topics.

Partition

Each topic is divided into one or more Partitions.

Partitions allow Kafka to:

  • Scale horizontally
  • Process data in parallel
  • Increase throughput

For a small development environment, one partition is usually sufficient.

Production systems often contain dozens or even hundreds of partitions.

Producer

A Producer writes messages to a Kafka topic.

For example:

Order Created

Payment Successful

Inventory Updated
Enter fullscreen mode Exit fullscreen mode

Each message is appended sequentially to the topic.

Consumer

A Consumer continuously reads messages from Kafka.

Consumers remember the last message they processed using Offsets, allowing them to resume processing after a restart.

What is KRaft?

Older versions of Kafka required Apache ZooKeeper to maintain cluster metadata.

ZooKeeper stored information such as:

  • Broker registration
  • Controller election
  • Topic metadata
  • Partition assignments
  • Cluster configuration

While reliable, this meant administrators had to deploy and maintain two distributed systems.

Beginning with modern Kafka releases, this dependency has been removed.

Kafka now uses KRaft (Kafka Raft Metadata mode).

Instead of relying on ZooKeeper, Kafka stores metadata internally using the Raft consensus protocol.

This results in:

  • Simpler deployment
  • Fewer moving parts
  • Lower operational overhead
  • Improved scalability
  • Better long-term maintainability

For new Kafka deployments, KRaft is the recommended architecture.

Installing Apache Kafka 4.2

Before installing Kafka, verify that Java 11 or later is installed.

java -version
Enter fullscreen mode Exit fullscreen mode

Download Kafka:

wget https://downloads.apache.org/kafka/4.2.0/kafka_2.13-4.2.0.tgz
Enter fullscreen mode Exit fullscreen mode

Extract the archive:

tar -xzf kafka_2.13-4.2.0.tgz
Enter fullscreen mode Exit fullscreen mode

Navigate into the installation directory:

cd kafka_2.13-4.2.0
Enter fullscreen mode Exit fullscreen mode

At this point, Kafka has been downloaded but is not yet initialized.

The next step is configuring Kafka to run in KRaft mode.

Understanding server.properties

Almost every important Kafka configuration lives inside:

config/server.properties
Enter fullscreen mode Exit fullscreen mode

Although this file contains many configuration options, only a handful are essential for a single-node KRaft installation.

process.roles=broker,controller

node.id=1

controller.quorum.voters=1@localhost:9093

listeners=PLAINTEXT://:9092,CONTROLLER://:9093

advertised.listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093

controller.listener.names=CONTROLLER

log.dirs=/tmp/kraft-combined-logs

delete.topic.enable=true
Enter fullscreen mode Exit fullscreen mode

Let's briefly understand what each property does.

Property Purpose
process.roles Defines whether this node acts as a Broker, Controller, or both.
node.id Unique identifier for the Kafka node.
controller.quorum.voters Identifies which nodes participate in controller elections.
listeners Specifies the network ports Kafka listens on.
advertised.listeners Tells clients how they should connect to Kafka.
controller.listener.names Indicates which listener is reserved for controller communication.
log.dirs Directory used to store Kafka logs and metadata.
delete.topic.enable Enables topic deletion.

Notice that Kafka listens on two ports:

Port Purpose
9092 Producers, Consumers, Kafka Clients
9093 Internal KRaft Controller Communication

Your Java applications, producers, consumers, and administration tools will typically connect to 9092.

The 9093 listener is reserved for Kafka's internal controller operations.

In the next section, we'll initialize Kafka storage, start the broker, create our first topic, and begin publishing and consuming messages.

Initializing Kafka Storage and Starting the Broker

Now that Kafka is installed and server.properties is configured, the next step is to initialize the metadata storage required by KRaft mode.

Unlike older Kafka versions that relied on ZooKeeper to maintain cluster metadata, KRaft stores metadata directly within Kafka. Before the broker can start, this storage must be initialized exactly once.

Step 1 – Generate a Cluster ID

Every Kafka cluster has a unique Cluster ID.

Generate one using:

bin/kafka-storage.sh random-uuid
Enter fullscreen mode Exit fullscreen mode

Example output:

NBPgGxgHTuuFVYzV3U6oyA
Enter fullscreen mode Exit fullscreen mode

Your value will be different.

Save this value because it will be used during the storage formatting step.

Step 2 – Format the Metadata Storage

Initialize the storage using the Cluster ID that was generated in the previous step.

bin/kafka-storage.sh format \
-t NBPgGxgHTuuFVYzV3U6oyA \
-c config/server.properties
Enter fullscreen mode Exit fullscreen mode

If everything is configured correctly, Kafka will display output similar to:

Formatting dynamic metadata voter directory /tmp/kraft-combined-logs
Enter fullscreen mode Exit fullscreen mode

This command creates the metadata required by the KRaft controller.

Important: Formatting should only be performed when creating a new Kafka cluster. Running it again on an existing cluster will erase the existing metadata.

Step 3 – Start Kafka

Start the Kafka broker using:

bin/kafka-server-start.sh config/server.properties
Enter fullscreen mode Exit fullscreen mode

Kafka starts as a foreground process.

You should begin seeing log messages indicating that the broker and controller have started successfully.

Leave this terminal running.

Understanding the Listener Ports

One question that often confuses new Kafka users is:

Why does Kafka use two ports?

Earlier we configured:

listeners=PLAINTEXT://:9092,CONTROLLER://:9093
Enter fullscreen mode Exit fullscreen mode

These listeners serve different purposes.

Port Purpose
9092 Client communication (Producers, Consumers, Admin tools, Java applications)
9093 Internal KRaft Controller communication

Think of 9092 as the public entrance to Kafka.

Applications connect to this port to publish and consume messages.

The 9093 listener is reserved for Kafka itself. It is used internally by the controller to manage cluster metadata and coordinate the broker.

As an application developer, you will almost always connect to 9092.

Creating Your First Topic

A Topic is a logical channel used to organize messages.

Let's create one named my-test-topic.

bin/kafka-topics.sh \
--create \
--topic my-test-topic \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Expected output:

Created topic my-test-topic.
Enter fullscreen mode Exit fullscreen mode

Notice that we connected using localhost:9092.

The --bootstrap-server parameter specifies the Kafka broker that the client should contact.

Listing Existing Topics

To display all available topics:

bin/kafka-topics.sh \
--list \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Example output:

my-test-topic
Enter fullscreen mode Exit fullscreen mode

If your Kafka installation contains multiple topics, each one will be listed.

Describing a Topic

To display detailed information about a topic:

bin/kafka-topics.sh \
--describe \
--topic my-test-topic \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Typical output includes:

  • Topic name
  • Partition count
  • Replication factor
  • Leader
  • Replicas
  • In-Sync Replicas (ISR)

Understanding these values becomes increasingly important as your Kafka cluster grows beyond a single broker.

Producing Messages

Now let's send our first messages.

Open a second terminal and execute:

bin/kafka-console-producer.sh \
--topic my-test-topic \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

You'll see a prompt:

>
Enter fullscreen mode Exit fullscreen mode

Everything you type is immediately published to Kafka.

For example:

Hello Kafka!

Learning Kafka with KRaft mode.

This is my first Kafka message.
Enter fullscreen mode Exit fullscreen mode

Each line becomes a separate Kafka message.

Consuming Messages

Open a third terminal.

Run:

bin/kafka-console-consumer.sh \
--topic my-test-topic \
--from-beginning \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Example output:

Hello Kafka!

Learning Kafka with KRaft mode.

This is my first Kafka message.
Enter fullscreen mode Exit fullscreen mode

The --from-beginning option instructs Kafka to replay all existing messages in the topic.

This is useful when verifying your setup or replaying historical events.

Reading Only New Messages

If you're only interested in messages produced after the consumer starts, simply omit the --from-beginning option.

bin/kafka-console-consumer.sh \
--topic my-test-topic \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Now the consumer waits for new incoming messages.

As soon as a producer publishes another message, it immediately appears in the consumer terminal.

This demonstrates Kafka's real-time event streaming capability.

How Message Flow Works

At this point, you've already built a complete event-driven messaging system.

The overall flow looks like this:

Producer
    │
    │ Publish Message
    ▼
+-------------------+
|       Kafka       |
|    Topic Storage  |
+-------------------+
    │
    │ Read Messages
    ▼
Consumer
Enter fullscreen mode Exit fullscreen mode

The producer and consumer never communicate directly.

Kafka sits between them, storing messages reliably and allowing multiple consumers to process the same events independently.

This decoupling is one of Kafka's biggest architectural advantages and is widely used in microservices and distributed systems.

Verifying That Everything Works

At this stage, you should have successfully:

  • Installed Kafka
  • Configured KRaft mode
  • Started the broker
  • Created a topic
  • Produced messages
  • Consumed messages

You now have a fully functioning single-node Kafka environment ready for development and experimentation.

In the next section, we'll look at common administration commands, deleting topics, troubleshooting configuration issues—including one that prevented my own Kafka broker from starting—and discuss where to go next in your Kafka learning journey.

Kafka Administration

Once your Kafka broker is running successfully, you'll frequently perform administrative tasks such as listing topics, inspecting topic details, and deleting topics during development.

Let's look at some of the most common commands.

Listing All Topics

To display every topic in the Kafka cluster:

bin/kafka-topics.sh \
--list \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Example output:

my-test-topic
orders
payments
application-logs
Enter fullscreen mode Exit fullscreen mode

This command is useful for verifying that topics were created successfully.

Describing a Topic

To view detailed information about a topic:

bin/kafka-topics.sh \
--describe \
--topic my-test-topic \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Typical output contains information similar to:

Topic: my-test-topic

PartitionCount: 1

ReplicationFactor: 1

Leader: 1

Replicas: 1

Isr: 1
Enter fullscreen mode Exit fullscreen mode

Let's briefly understand what these values mean.

Property Description
PartitionCount Number of partitions belonging to the topic
ReplicationFactor Number of replicas maintained for fault tolerance
Leader Broker currently responsible for reads and writes
Replicas Brokers containing copies of the partition
ISR In-Sync Replicas currently synchronized with the leader

In this tutorial we're using a single-node Kafka cluster, so each value is simply 1.

Deleting a Topic

Deleting a topic is straightforward.

bin/kafka-topics.sh \
--bootstrap-server localhost:9092 \
--delete \
--topic my-test-topic
Enter fullscreen mode Exit fullscreen mode

However, topic deletion must be enabled.

Verify that the following property exists inside:

config/server.properties
Enter fullscreen mode Exit fullscreen mode 
delete.topic.enable=true
Enter fullscreen mode Exit fullscreen mode

Without this configuration, Kafka ignores delete requests.

Common Kafka Commands

Create a Topic 

bin/kafka-topics.sh \
--create \
--topic my-test-topic \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

List Topics 

bin/kafka-topics.sh \
--list \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Describe a Topic 

bin/kafka-topics.sh \
--describe \
--topic my-test-topic \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Produce Messages 

bin/kafka-console-producer.sh \
--topic my-test-topic \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Consume All Messages 

bin/kafka-console-consumer.sh \
--topic my-test-topic \
--from-beginning \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Consume Only New Messages 

bin/kafka-console-consumer.sh \
--topic my-test-topic \
--bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Delete a Topic 

bin/kafka-topics.sh \
--bootstrap-server localhost:9092 \
--delete \
--topic my-test-topic
Enter fullscreen mode Exit fullscreen mode

Real-World Issue: Why My Kafka Broker Wouldn't Start

While writing this article, I ran into a problem that took some investigation to resolve.

After formatting the Kafka storage and attempting to start the broker, Kafka continuously displayed messages similar to:

initializeNewPublishers:
the loader is still catching up because we still don't know the high water mark yet.
Enter fullscreen mode Exit fullscreen mode

It also reported:

Unable to register the broker because the RPC got timed out before it could be sent.
Enter fullscreen mode Exit fullscreen mode

Although these messages looked alarming, they were actually symptoms rather than the root cause.

The Real Problem

After reviewing my configuration, I discovered that my server.properties file was missing the following configuration:

controller.quorum.voters=1@localhost:9093
Enter fullscreen mode Exit fullscreen mode

Without this property, Kafka's Controller had no information about which node was responsible for participating in controller elections.

As a result:

  • The Controller never became active.
  • The Broker could not register.
  • Kafka continued waiting indefinitely for metadata initialization.

Adding the missing configuration resolved the issue immediately.

Why This Configuration Matters

In KRaft mode, Kafka no longer relies on ZooKeeper to manage metadata.

Instead, Kafka uses an internal Controller based on the Raft consensus protocol.

The controller.quorum.voters property identifies the nodes participating in that Controller quorum.

For a single-node development environment, the configuration is simply:

controller.quorum.voters=1@localhost:9093
Enter fullscreen mode Exit fullscreen mode

In production clusters, multiple controller nodes would typically be listed.

Lessons Learned

During this installation I learned several useful lessons:

✅ KRaft mode eliminates the need for ZooKeeper.

✅ Every Kafka cluster requires a unique Cluster ID.

✅ Kafka storage must be formatted before the broker starts.

✅ Producers and consumers connect to 9092, while 9093 is reserved for internal Controller communication.

✅ The server.properties file is the heart of Kafka configuration.

✅ A missing controller.quorum.voters property prevents the Controller from becoming active.

Understanding why Kafka behaves this way makes troubleshooting much easier than simply memorizing commands.

What's Next?

Now that Kafka is running successfully, there are many exciting topics to explore.

Some natural next steps include:

  • Building a Java Kafka Producer
  • Building a Java Kafka Consumer
  • Understanding Partitions and Offsets
  • Consumer Groups
  • Kafka Streams
  • Spring Boot with Kafka
  • Event-Driven Microservices
  • Real-time Log Processing
  • Fraud Detection Pipelines
  • Integrating Kafka with AI and RAG applications

Kafka is much more than a messaging system—it is a powerful event streaming platform used by organizations to build highly scalable, real-time applications.

Conclusion

In this guide, we installed Apache Kafka 4.2 on Ubuntu 24.04 (WSL2) using KRaft mode, configured a single-node broker, initialized the metadata storage, created topics, produced and consumed messages, and explored the essential administration commands used in day-to-day development.

Along the way, we also investigated and resolved a real configuration issue involving controller.quorum.voters, highlighting the importance of understanding Kafka's internal architecture rather than simply following installation steps.

Whether you're building microservices, processing application logs, streaming IoT data, or developing real-time analytics, Kafka provides a reliable foundation for event-driven applications.

I hope this guide helps you get started with Kafka and saves you some of the troubleshooting time that I experienced during my own setup. Happy learning!

Top comments (0)