DEV Community

Cover image for 🎯 Apache Kafka Single-Node Practice Guide
Ajinkya Singh
Ajinkya Singh

Posted on

🎯 Apache Kafka Single-Node Practice Guide

#docker #kafka #tutorial #programming

Your Personal Message Delivery System

Welcome to your hands-on Kafka learning journey! In this guide, we'll build a complete Kafka system on a single machine - perfect for learning, testing, and understanding how everything works together.

📦 What We're Building

Think of this as creating your own Digital Post Office on your computer:

┌─────────────────────────────────────────┐
│     YOUR LAPTOP (Single Machine)        │
│                                         │
│  ┌───────────────────────────────┐    │
│  │    KAFKA BROKER (Port 9092)   │    │
│  │         Your Post Office      │    │
│  ├───────────────────────────────┤    │
│  │                               │    │
│  │  📬 Topic: "customer-orders"  │    │
│  │  📬 Topic: "payment-alerts"   │    │
│  │  📬 Topic: "user-activity"    │    │
│  │                               │    │
│  └───────────────────────────────┘    │
│           ↑              ↓             │
│      PRODUCERS      CONSUMERS          │
│   (Send messages) (Receive messages)   │
└─────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Important Note: Single-node setup is ONLY for:

  • ✅ Learning and experimentation
  • ✅ Development and testing
  • ✅ Understanding Kafka concepts
  • ❌ NOT for production use (no fault tolerance!)

🛠 Step 1: Installation & Setup

Option A: Using Docker (Recommended - Easiest!) 

# 1. Generate a unique Cluster ID
CLUSTER_ID=$(docker run --rm apache/kafka:latest /opt/kafka/bin/kafka-storage.sh random-uuid)
echo "Generated Cluster ID: $CLUSTER_ID"

# 2. Create a docker-compose.yml file with the generated ID
cat > docker-compose.yml << EOF
services:
  kafka1:
    image: apache/kafka:latest
    container_name: kafka1
    ports:
      - "9093:9093"
    volumes:
      - ./kafka-data/kafka1:/var/lib/kafka/data
    networks:
      - kafka-bridge
    environment:
      KAFKA_BROKER_ID: 1
      CLUSTER_ID: ${CLUSTER_ID}
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:29092,EXTERNAL://localhost:9093
      KAFKA_LISTENERS: CONTROLLER://:9092,EXTERNAL://0.0.0.0:9093,INTERNAL://:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data

networks:
  kafka-bridge:
    driver: bridge

volumes:
  kafka-data:
EOF

# 3. Start Kafka
docker-compose up -d

# 4. Check if it's running
docker ps

# Expected output:
# CONTAINER ID   IMAGE                  STATUS         PORTS
# abc123def456   apache/kafka:latest    Up 10 seconds  0.0.0.0:9093->9093/tcp
Enter fullscreen mode Exit fullscreen mode

Alternative: Manual Cluster ID Generation

# If you prefer to set it manually, you can generate and copy the ID:
docker run --rm apache/kafka:latest /opt/kafka/bin/kafka-storage.sh random-uuid

# Output example: MkU3OEVBNTcwNTJENDM2Qk
# Then paste this ID into CLUSTER_ID in docker-compose.yml
Enter fullscreen mode Exit fullscreen mode

Understanding the Configuration:

┌─────────────────────────────────────────────────┐
│           YOUR DOCKER KAFKA SETUP               │
├─────────────────────────────────────────────────┤
│                                                 │
│  Container: kafka1                              │
│  ┌───────────────────────────────────────────┐ │
│  │  Port 9093 (External) ← You connect here │ │
│  │  Port 29092 (Internal) ← Container-only  │ │
│  │  Port 9092 (Controller) ← Management     │ │
│  └───────────────────────────────────────────┘ │
│                     ↓                           │
│  Volume: ./kafka-data/kafka1                    │
│  (Your data persists here!)                     │
│                                                 │
└─────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

What Each Port Does:

  • 9093 (EXTERNAL): Use this from your laptop (localhost:9093)
  • 29092 (INTERNAL): Used by containers talking to each other
  • 9092 (CONTROLLER): Kafka's internal coordination

Data Persistence:
Your messages are saved in ./kafka-data/kafka1 directory. Even if you stop the container, your data remains!

🎪 Step 2: Your First Kafka Topic

Let's create our coffee shop's order board!

Create a Topic 

# Using Docker:
docker exec -it kafka1 /opt/kafka/bin/kafka-topics.sh \
  --create \
  --topic coffee-orders \
  --bootstrap-server localhost:9093 \
  --partitions 3 \
  --replication-factor 1
Enter fullscreen mode Exit fullscreen mode

What This Creates:

Topic: "coffee-orders" (3 sections on your order board)

┌─────────────────────────────────────────┐
│         COFFEE ORDERS BOARD             │
├─────────────────────────────────────────┤
│                                         │
│  Section 0    Section 1    Section 2   │
│  (Partition)  (Partition)  (Partition) │
│  ┌────────┐  ┌────────┐  ┌────────┐   │
│  │ Order 1│  │ Order 2│  │ Order 3│   │
│  │ Order 4│  │ Order 5│  │ Order 6│   │
│  │  ...   │  │  ...   │  │  ...   │   │
│  └────────┘  └────────┘  └────────┘   │
└─────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

List All Topics 

docker exec -it kafka1 /opt/kafka/bin/kafka-topics.sh \
  --list \
  --bootstrap-server localhost:9093
Enter fullscreen mode Exit fullscreen mode

View Topic Details 

docker exec -it kafka1 /opt/kafka/bin/kafka-topics.sh \
  --describe \
  --topic coffee-orders \
  --bootstrap-server localhost:9093
Enter fullscreen mode Exit fullscreen mode

Output Explanation:

Topic: coffee-orders
PartitionCount: 3          ← Your order board has 3 sections
ReplicationFactor: 1       ← Only 1 copy (single node)
Partition: 0
  Leader: 1                ← Broker 1 manages this section
  Replicas: 1              ← Only one copy exists
  Isr: 1                   ← In-sync replicas
Enter fullscreen mode Exit fullscreen mode

☕ Step 3: Sending Orders (Producer)

Now let's be the cashier and take some orders!

Start the Console Producer 

docker exec -it kafka1 /opt/kafka/bin/kafka-console-producer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093
Enter fullscreen mode Exit fullscreen mode

Type these orders (press Enter after each):

{"customer":"Alice","drink":"Latte","size":"Medium","price":4.50}
{"customer":"Bob","drink":"Espresso","size":"Small","price":3.00}
{"customer":"Charlie","drink":"Cappuccino","size":"Large","price":5.50}
{"customer":"Diana","drink":"Mocha","size":"Medium","price":4.75}
{"customer":"Eve","drink":"Americano","size":"Large","price":3.50}
Enter fullscreen mode Exit fullscreen mode

What's Happening:

You (Cashier)                  Kafka                    Storage
     │                           │                         │
     │──Order: Alice's Latte────→│                         │
     │                           │──Save to Partition 0───→│
     │                           │                         │
     │──Order: Bob's Espresso───→│                         │
     │                           │──Save to Partition 1───→│
     │                           │                         │
     │──Order: Charlie's Capp.──→│                         │
     │                           │──Save to Partition 2───→│
     │                           │                         │
Enter fullscreen mode Exit fullscreen mode

Press Ctrl+C to stop the producer

🧑‍🍳 Step 4: Processing Orders (Consumer)

Now let's be the barista and read the orders!

Start the Console Consumer (Read from Beginning) 

docker exec -it kafka1 /opt/kafka/bin/kafka-console-consumer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093 \
  --from-beginning
Enter fullscreen mode Exit fullscreen mode

You'll see all the orders scroll by:

{"customer":"Alice","drink":"Latte","size":"Medium","price":4.50}
{"customer":"Bob","drink":"Espresso","size":"Small","price":3.00}
{"customer":"Charlie","drink":"Cappuccino","size":"Large","price":5.50}
...
Enter fullscreen mode Exit fullscreen mode

What's Happening:

Storage                    Kafka                    Barista (You)
   │                         │                          │
   │←──Request all orders────│                          │
   │                         │                          │
   │──Send order 1──────────→│──Alice's Latte─────────→│
   │──Send order 2──────────→│──Bob's Espresso────────→│
   │──Send order 3──────────→│──Charlie's Cappuccino──→│
   │                         │                          │
Enter fullscreen mode Exit fullscreen mode

Start Another Consumer (Real-time Only)

Open a NEW terminal and run:

docker exec -it kafka1 /opt/kafka/bin/kafka-console-consumer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093
Enter fullscreen mode Exit fullscreen mode

This consumer only sees new orders (like a new barista who just arrived).

🎭 Step 5: Multiple Producers & Consumers

Let's simulate a busy coffee shop with multiple cashiers and baristas!

Experiment Setup

Terminal 1 (Cashier A - Producer):

docker exec -it kafka1 /opt/kafka/bin/kafka-console-producer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093
Enter fullscreen mode Exit fullscreen mode

Terminal 2 (Cashier B - Producer):

docker exec -it kafka1 /opt/kafka/bin/kafka-console-producer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093
Enter fullscreen mode Exit fullscreen mode

Terminal 3 (Barista A - Consumer):

docker exec -it kafka1 /opt/kafka/bin/kafka-console-consumer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093 \
  --from-beginning
Enter fullscreen mode Exit fullscreen mode

Terminal 4 (Barista B - Consumer):

docker exec -it kafka1 /opt/kafka/bin/kafka-console-consumer.sh \
  --topic coffee-orders \
  --bootstrap-server localhost:9093 \
  --from-beginning
Enter fullscreen mode Exit fullscreen mode

Now:

  1. Type orders in Terminal 1 and 2 (both cashiers taking orders)
  2. Watch them appear in Terminal 3 and 4 (both baristas see ALL orders)

Visual:

Cashier A ──┐
            ├──→ ORDER BOARD ──┬──→ Barista A (sees all orders)
Cashier B ──┘    (Kafka)       └──→ Barista B (sees all orders)
Enter fullscreen mode Exit fullscreen mode

Top comments (0)