DEV Community

Cover image for Deploying Kafka Distributed Event Streaming on Ubuntu 24.04
Vultr profile image Sanskriti Harmukh
Sanskriti Harmukh for Vultr

Posted on with Aashish Chaurasiya • Originally published at docs.vultr.com

Deploying Kafka Distributed Event Streaming on Ubuntu 24.04

#kafka #docker #devops #streaming

Apache Kafka is an open-source distributed event streaming platform used for real-time pipelines, log aggregation, and event-driven architectures. This guide deploys a single-node Kafka 4.x cluster in KRaft mode (no ZooKeeper) using Docker Compose with persistent storage, then verifies producer/consumer flow with the Kafka CLI tools. By the end, you'll have Kafka accepting produced events and serving them to consumers on your server.

Set Up the Directory Structure

1. Create the project directory:

$ mkdir -p ~/kafka-stack/kafka-data
$ sudo chown 1000:1000 ~/kafka-stack/kafka-data
$ cd ~/kafka-stack
Enter fullscreen mode Exit fullscreen mode

2. Generate a Kafka cluster ID:

$ uuidgen | tr -d '-'
Enter fullscreen mode Exit fullscreen mode

Note the value — it goes into the .env file below.

3. Create the environment file:

$ nano .env
Enter fullscreen mode Exit fullscreen mode 
KAFKA_NODE_ID=1
KAFKA_CLUSTER_ID=YOUR_KAFKA_CLUSTER_ID
KAFKA_PORT=9092
KAFKA_ADVERTISED_HOST=localhost
Enter fullscreen mode Exit fullscreen mode

Deploy with Docker Compose

1. Create the Docker Compose manifest:

$ nano docker-compose.yaml
Enter fullscreen mode Exit fullscreen mode 
services:
  kafka:
    image: apache/kafka:4.1.1
    container_name: kafka
    ports:
      - "${KAFKA_PORT}:9092"
    environment:
      KAFKA_NODE_ID: ${KAFKA_NODE_ID}
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${KAFKA_ADVERTISED_HOST}:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LOG_DIRS: /kafka/data
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_CLUSTER_ID: ${KAFKA_CLUSTER_ID}
    volumes:
      - "./kafka-data:/kafka/data"
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "kafka-broker-api-versions", "--bootstrap-server=localhost:9092"]
      interval: 10s
      timeout: 5s
      retries: 5
Enter fullscreen mode Exit fullscreen mode

2. Start the service:

$ docker compose up -d
Enter fullscreen mode Exit fullscreen mode

3. Verify the service is running:

$ docker compose ps
$ docker compose logs
Enter fullscreen mode Exit fullscreen mode

Install the Kafka CLI Tools

1. Install Java and download the Kafka tools:

$ sudo apt update && sudo apt install -y default-jdk
$ wget https://downloads.apache.org/kafka/4.1.1/kafka_2.13-4.1.1.tgz
$ sudo tar -xzf kafka_2.13-4.1.1.tgz -C /opt
Enter fullscreen mode Exit fullscreen mode

2. Put the Kafka CLI on your PATH:

$ echo 'export PATH=$PATH:/opt/kafka_2.13-4.1.1/bin' | sudo tee /etc/profile.d/kafka.sh
$ source /etc/profile.d/kafka.sh
Enter fullscreen mode Exit fullscreen mode

Test Message Production and Consumption

1. Create a topic:

$ kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test-topic --partitions 3 --replication-factor 1
Enter fullscreen mode Exit fullscreen mode

2. Produce messages:

$ kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-topic
Enter fullscreen mode Exit fullscreen mode

Type a few lines and press Ctrl+D to send.

3. Consume from the beginning:

$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning
Enter fullscreen mode Exit fullscreen mode

4. Describe the topic:

$ kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic test-topic
Enter fullscreen mode Exit fullscreen mode

Next Steps

Kafka is running in KRaft mode with persistent storage. From here you can:

  • Add more brokers and set replication factors > 1 for production durability
  • Enable SASL/SSL listeners to authenticate and encrypt client traffic
  • Wire in Kafka Connect, Schema Registry, or ksqlDB for richer pipelines

For the full guide with additional tips, visit the original article on Vultr Docs.

Top comments (0)