DEV Community

Cover image for How to Install Apache Kafka on Ubuntu 26.04 LTS (KRaft Mode, No ZooKeeper)
Ramansah
Ramansah

Posted on • Originally published at bckinfo.com

How to Install Apache Kafka on Ubuntu 26.04 LTS (KRaft Mode, No ZooKeeper)

#kafka #howto #ubuntu

Table of Contents

  1. What is Apache Kafka and When Should You Use It
  2. KRaft Mode: Why ZooKeeper is No Longer Needed
  3. Prerequisites
  4. Step 1: Install Java 21
  5. Step 2: Create a Dedicated Kafka User
  6. Step 3: Download and Install Apache Kafka 4.2
  7. Step 4: Configure Kafka in KRaft Mode
  8. Step 5: Format the Storage Directory
  9. Step 6: Create a systemd Service
  10. Step 7: Configure UFW Firewall
  11. Step 8: Test with Producer and Consumer
  12. Step 9: Basic Topic Management
  13. Production Configuration Tips
  14. Kafka vs RabbitMQ: When to Use Which
  15. Common Issues and Quick Fixes
  16. Next Steps

Real-time data is everywhere — microservices firing events, IoT sensors streaming telemetry, user actions triggering notifications, logs flowing from dozens of servers simultaneously. Managing all of that reliably, at scale, without building a custom integration between every producer and every consumer, is exactly the problem Apache Kafka was built to solve.

This guide walks through installing Apache Kafka 4.2 on Ubuntu 26.04 LTS (Resolute Raccoon) using KRaft mode — the modern deployment approach that removes the ZooKeeper dependency entirely. If you've seen older Kafka tutorials that tell you to start ZooKeeper first, those are outdated: starting with Kafka 4.0, ZooKeeper support was completely removed.

What is Apache Kafka and When Should You Use It

Apache Kafka is a distributed event streaming platform — a durable, ordered log that producers write to and consumers read from. Unlike a traditional message queue where a message disappears after being consumed, Kafka retains events for a configurable period, allowing multiple independent consumers to read the same stream at their own pace.

Real-world use cases where Kafka fits well:

  • Microservice decoupling — Service A publishes an event; Services B, C, and D each consume it independently, without A knowing or caring who's listening.
  • Log and metrics aggregation — Centralize logs from dozens of services into one stream, then fan out to Elasticsearch, S3, and a monitoring stack simultaneously.
  • Real-time analytics — Process a stream of user events, transactions, or sensor readings as they happen rather than batching overnight.
  • Event sourcing — Store every state change as an immutable event, with the ability to replay history to rebuild application state.
  • Change Data Capture (CDC) — Stream database changes (inserts, updates, deletes) to downstream systems in real time.

Kafka is not the right tool for every messaging need. If you're sending a task to exactly one worker and want it acknowledged once, a simpler queue (Redis Streams, RabbitMQ, or even PostgreSQL LISTEN/NOTIFY) is probably sufficient and far easier to operate. Kafka earns its complexity at scale.

KRaft Mode: Why ZooKeeper is No Longer Needed

Before Kafka 3.x, every Kafka deployment required a separate Apache ZooKeeper cluster to manage broker metadata, leader elections, and cluster state. This meant running and maintaining two distributed systems for every Kafka deployment — doubling the operational complexity.

KRaft (Kafka Raft Metadata mode) replaces ZooKeeper by moving cluster metadata management directly into Kafka itself, using the Raft consensus algorithm. The result:

  • Simpler deployment — one system to install, configure, monitor, and upgrade instead of two.
  • Faster startup and failover — Kafka no longer needs to synchronize with an external ZooKeeper cluster.
  • Better scalability — ZooKeeper had practical limits on the number of partitions it could track; KRaft removes those limits.
  • Kafka 4.0+: ZooKeeper support is completely removed. There is no option to use ZooKeeper with Kafka 4.x — KRaft is the only deployment mode.

Prerequisites

  • Ubuntu 26.04 LTS (Resolute Raccoon) — fresh install or existing server
  • Minimum 4GB RAM (8GB+ recommended for production workloads)
  • At least 2 CPU cores
  • sudo access
  • Internet access to download packages

Step 1: Install Java 21

Kafka 4.x requires Java 17 or higher. Java 21 LTS is the recommended choice for Ubuntu 26.04 — it's the current Long-Term Support release and ships cleanly from Ubuntu's default repository:

sudo apt update
sudo apt install -y openjdk-21-jdk-headless
Enter fullscreen mode Exit fullscreen mode

Verify the installation:

java -version
Enter fullscreen mode Exit fullscreen mode

Expected output:

openjdk version "21.x.x" ...
OpenJDK Runtime Environment (build 21.x.x+...)
Enter fullscreen mode Exit fullscreen mode

Set JAVA_HOME so Kafka can find it:

echo 'export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64' | \
  sudo tee /etc/profile.d/java.sh
source /etc/profile.d/java.sh
Enter fullscreen mode Exit fullscreen mode

Confirm:

echo $JAVA_HOME
# /usr/lib/jvm/java-21-openjdk-amd64
Enter fullscreen mode Exit fullscreen mode

Step 2: Create a Dedicated Kafka User

Running Kafka as root is a security risk. Create a dedicated system user with no login access:

sudo useradd -r -m -U -d /opt/kafka -s /bin/false kafka
Enter fullscreen mode Exit fullscreen mode

This follows the same principle of least privilege covered in our Docker Container Security Best Practices guide — services should run with the minimum access they need, never as root.

Step 3: Download and Install Apache Kafka 4.2

Set the version variables, then download and extract to /opt/kafka:

KAFKA_VERSION="4.2.0"
SCALA_VERSION="2.13"

wget https://downloads.apache.org/kafka/${KAFKA_VERSION}/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz \
  -O /tmp/kafka.tgz

sudo tar -xzf /tmp/kafka.tgz -C /opt/kafka --strip-components=1
sudo chown -R kafka:kafka /opt/kafka
Enter fullscreen mode Exit fullscreen mode

Verify the extraction:

ls /opt/kafka
# bin  config  libs  LICENSE  licenses  NOTICE  site-docs
Enter fullscreen mode Exit fullscreen mode

Create a dedicated data directory for Kafka logs (separate from the application directory — good practice for disk management and backups):

sudo mkdir -p /var/lib/kafka/data
sudo chown -R kafka:kafka /var/lib/kafka
Enter fullscreen mode Exit fullscreen mode

Step 4: Configure Kafka in KRaft Mode

KRaft mode uses a single unified configuration file (server.properties) where the broker also acts as the cluster controller. Edit the KRaft server configuration:

sudo nano /opt/kafka/config/kraft/server.properties
Enter fullscreen mode Exit fullscreen mode

Key settings to review and update:

# The role this node plays in the cluster
# "broker,controller" = combined mode (suitable for single-node and small clusters)
process.roles=broker,controller

# Unique ID for this broker — change if running multiple brokers
node.id=1

# Controller quorum voters — format: node.id@host:port
# For single node: use the same node
controller.quorum.voters=1@localhost:9093

# Listeners — what addresses Kafka binds to
listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093

# The address clients use to reach the broker
# Change 'localhost' to your server's actual IP or hostname if remote clients need access
advertised.listeners=PLAINTEXT://localhost:9092

# Listener used for inter-broker and controller communication
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER

# Log directory — where Kafka stores event data
log.dirs=/var/lib/kafka/data

# Number of partitions for auto-created topics
num.partitions=3

# Log retention (how long to keep events)
log.retention.hours=168        # 7 days
log.retention.bytes=1073741824 # 1GB per partition

# Replication factor for internal topics (keep at 1 for single-node)
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
Enter fullscreen mode Exit fullscreen mode

Remote access note: If clients will connect from other machines, replace localhost in advertised.listeners with your server's actual IP address or DNS hostname. Kafka clients use the advertised address to establish connections — using localhost here will cause remote clients to fail even if they can reach the server on port 9092.

Step 5: Format the Storage Directory

Before first startup, Kafka's storage directory must be initialized with a unique cluster ID. This is a one-time operation:

# Generate a cluster ID
CLUSTER_ID=$(/opt/kafka/bin/kafka-storage.sh random-uuid)
echo "Cluster ID: $CLUSTER_ID"

# Format the storage directory with that cluster ID
sudo -u kafka /opt/kafka/bin/kafka-storage.sh format \
  -t $CLUSTER_ID \
  -c /opt/kafka/config/kraft/server.properties
Enter fullscreen mode Exit fullscreen mode

Expected output:

Formatting /var/lib/kafka/data with metadata.version X.X-IV...
Enter fullscreen mode Exit fullscreen mode

This step replaces the old ZooKeeper initialization — there's no zookeeper-server-start.sh to run first.

Step 6: Create a systemd Service

Create a systemd unit file so Kafka starts automatically on boot and can be managed with systemctl:

sudo nano /etc/systemd/system/kafka.service
Enter fullscreen mode Exit fullscreen mode 
[Unit]
Description=Apache Kafka Server (KRaft Mode)
Documentation=https://kafka.apache.org/documentation/
After=network.target

[Service]
Type=simple
User=kafka
Group=kafka
Environment="JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64"
ExecStart=/opt/kafka/bin/kafka-server-start.sh \
  /opt/kafka/config/kraft/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
StandardOutput=journal
StandardError=journal
SyslogIdentifier=kafka

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable kafka
sudo systemctl start kafka
Enter fullscreen mode Exit fullscreen mode

Check the status:

sudo systemctl status kafka
Enter fullscreen mode Exit fullscreen mode

Expected output:

● kafka.service - Apache Kafka Server (KRaft Mode)
     Loaded: loaded (/etc/systemd/system/kafka.service; enabled)
     Active: active (running) since ...
Enter fullscreen mode Exit fullscreen mode

View live logs:

sudo journalctl -u kafka -f
Enter fullscreen mode Exit fullscreen mode

Step 7: Configure UFW Firewall

If UFW is enabled, open the ports Kafka needs:

# Port 9092 — Kafka broker (clients connect here)
sudo ufw allow 9092/tcp comment "Kafka broker"

# Port 9093 — Kafka controller (internal KRaft communication)
# Only needed if running multi-node — restrict to internal network for single-node
sudo ufw allow from 10.0.0.0/8 to any port 9093 comment "Kafka KRaft controller"

sudo ufw reload
sudo ufw status
Enter fullscreen mode Exit fullscreen mode

Security note: Port 9092 should only be open to trusted clients, not the entire internet. If your Kafka broker is internet-facing, restrict port 9092 to specific IP ranges or put it behind a VPN.

Step 8: Test with Producer and Consumer

Open two separate terminal sessions for this test — one for the producer, one for the consumer.

First: Create a test topic

sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --create \
  --topic test-events \
  --partitions 3 \
  --replication-factor 1 \
  --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Expected output:

Created topic test-events.
Enter fullscreen mode Exit fullscreen mode

Terminal 1: Start a producer and send messages

sudo -u kafka /opt/kafka/bin/kafka-console-producer.sh \
  --topic test-events \
  --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Type a few messages and press Enter after each:

> Hello from Kafka on Ubuntu 26.04
> This is a test message
> Apache Kafka 4.2 KRaft mode works!
Enter fullscreen mode Exit fullscreen mode

Press Ctrl+C to exit.

Terminal 2: Start a consumer and read messages

sudo -u kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --topic test-events \
  --from-beginning \
  --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

You should see the messages you typed appear immediately. Press Ctrl+C to exit.

--from-beginning tells the consumer to read from the earliest available offset — not just new messages. This illustrates one of Kafka's key properties: consumers can replay historical events, not just receive new ones.

Step 9: Basic Topic Management 

# List all topics
sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --list \
  --bootstrap-server localhost:9092

# Describe a topic (partitions, replicas, leader)
sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --describe \
  --topic test-events \
  --bootstrap-server localhost:9092

# Increase partition count (partitions can only be increased, never decreased)
sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --alter \
  --topic test-events \
  --partitions 6 \
  --bootstrap-server localhost:9092

# Delete a topic
sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --delete \
  --topic test-events \
  --bootstrap-server localhost:9092

# List consumer groups
sudo -u kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --list \
  --bootstrap-server localhost:9092

# Check consumer group lag (how far behind a consumer is)
sudo -u kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --describe \
  --group my-consumer-group \
  --bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Consumer group lag (last command) is one of the most important operational metrics in a Kafka deployment — it tells you how many messages a consumer is behind. Integrating this metric into your Prometheus + Grafana monitoring stack via kafka-exporter or JMX exporter gives you real-time visibility into consumer health.

Production Configuration Tips

A single-node Kafka installation on Ubuntu is sufficient for development and low-traffic production workloads. For anything that needs to scale or survive a broker failure, here are the key settings to revisit:

Memory tuning:
Kafka's JVM heap is set to 1GB by default. For production, adjust in /opt/kafka/bin/kafka-server-start.sh:

export KAFKA_HEAP_OPTS="-Xmx4G -Xms4G"
Enter fullscreen mode Exit fullscreen mode

Set both -Xmx and -Xms to the same value to prevent JVM heap resizing pauses.

Log retention by size, not just time:

# Retain logs for 7 days OR until they exceed 10GB per partition, whichever comes first
log.retention.hours=168
log.retention.bytes=10737418240
Enter fullscreen mode Exit fullscreen mode

Auto topic creation in production:

# Disable auto-creation — require topics to be created explicitly
auto.create.topics.enable=false
Enter fullscreen mode Exit fullscreen mode

Auto-created topics use default partition/replication settings, which are rarely correct for specific use cases. Disable this and create topics explicitly with the right parameters.

Replication factor for multi-broker clusters:

# With 3 brokers, use replication factor 3 and min ISR 2
# (data survives loss of any 1 broker, writes require 2 brokers to acknowledge)
default.replication.factor=3
min.insync.replicas=2
Enter fullscreen mode Exit fullscreen mode

Regular backups:
The /var/lib/kafka/data directory contains all event data. Back it up the same way you'd back up any other data volume — a scheduled snapshot with retention policy, tested restore process, stored separately from the host. The scripted approach from our Redis with Docker Compose guide applies directly to the Kafka data directory.

Kafka vs RabbitMQ: When to Use Which

A common question when introducing a message broker for the first time:

Apache Kafka RabbitMQ
Model Distributed log (pull-based) Message queue (push-based)
Message retention Configurable period (days/weeks) Until acknowledged (usually)
Throughput Millions of messages/second Tens to hundreds of thousands/second
Consumer pattern Each consumer group reads the full log independently One consumer per message (competing consumers)
Replay Yes — consumers can re-read past events No — once consumed, message is gone
Ordering guarantee Per partition Per queue
Operational complexity Higher Lower
Best for Event streaming, log aggregation, CDC, high throughput Task queues, RPC, point-to-point messaging, simpler use cases

Choose Kafka when: You need multiple consumers to independently process the same events, you need to replay history, or you need to handle very high throughput (millions of events/second).

Choose RabbitMQ when: You're building a task queue where each task should be processed exactly once by exactly one worker, the message volume is moderate, and you want simpler operations.

Common Issues and Quick Fixes

Symptom Likely Cause Fix
kafka.service fails to start Storage not formatted, or wrong JAVA_HOME Run kafka-storage.sh format first; verify java -version works as kafka user
Connection refused on port 9092 Kafka not running, or listener bound to wrong address Check systemctl status kafka; verify listeners in server.properties
Remote clients can't connect advertised.listeners set to localhost Change advertised.listeners to the server's actual IP or hostname
Leader not available on topic create Kafka still initializing after start Wait 10-15 seconds after Kafka starts before creating topics
Consumer reads no messages Consumer started after producer, no --from-beginning flag Add --from-beginning flag to read historical messages
OutOfMemoryError in logs JVM heap too small for the workload Increase -Xmx and -Xms in kafka-server-start.sh
High consumer group lag Consumer too slow or too few consumer instances Scale consumer instances; check processing logic for bottlenecks

Next Steps

With Kafka running on Ubuntu 26.04, the natural next steps are:

  • Add monitoring — deploy kafka-exporter alongside your Prometheus + Grafana stack to track consumer group lag, message throughput, and broker health in real time.
  • Containerize it — for teams already running Docker Compose stacks, Kafka has an official Docker image and can be added to an existing Compose setup using the same KRaft configuration.
  • Explore Kafka Streams or Apache Flink — for in-stream processing (filtering, aggregating, joining streams) without writing a separate consumer application.
  • Scale to multi-broker — repeat the installation on additional nodes, adjust broker.id and controller.quorum.voters, and distribute your topics' partitions across the cluster for fault tolerance and horizontal throughput scaling.

Top comments (0)