Fixing Kafka Connectivity Issues Between Node.js and Docker Containers

#kafka #node #docker #dns

Introduction

Apache Kafka is a powerful event streaming platform, but setting it up in a Docker environment while ensuring seamless connectivity with an external Node.js service can be challenging. This article explores two key challenges:

Connecting a Local Node.js Server as an External Host Machine with a Kafka Broker in Docker
Understanding and Fixing Docker DNS Resolution Issues for Local Machine Communication

We'll walk through the problems faced, their root causes, and step-by-step solutions.

Setting Up Kafka in a Docker Container

We are using Bitnami's Kafka image, which supports KRaft mode (Kafka's built-in metadata management) without requiring ZooKeeper.

1. Create a Docker Network

To ensure seamless communication between services, create a dedicated network:

docker network create app-tier

2. Run Kafka in KRaft Mode

docker run -d --name kafka-server -p 9092:9092 --hostname kafka-server \  
    --network app-tier \  
    -e KAFKA_CFG_NODE_ID=0 \  
    -e KAFKA_CFG_PROCESS_ROLES=controller,broker \  
    -e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \  
    -e KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \  
    -e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://192.168.1.10:9092 \  
    -e KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka-server:9093 \  
    -e KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER \  
bitnami/kafka:latest

Understanding `advertised.listeners`

listeners=PLAINTEXT://:9092 → Kafka listens on all interfaces inside the container.
advertised.listeners=PLAINTEXT://192.168.1.10:9092 → External clients connect using the host machine's IP.

This prevents Kafka from advertising its internal container IP, which would make it unreachable from the local machine.

Challenge 1: Connecting a Local Node.js Server to Docker Kafka

By default, a Docker container runs in an isolated network. This means localhost:9092 inside the container does not refer to the host machine’s localhost. If a Node.js service runs outside Docker, it cannot connect to Kafka using localhost:9092 unless Kafka explicitly advertises the host machine's IP.

Solution: Use the Host IP

Find your machine's IP address:

ip a | grep inet

Then, update Kafka’s advertised.listeners to point to this IP.

Challenge 2: Docker DNS Resolution and Communication

Even after setting the correct advertised.listeners, issues may arise if:

Kafka and Node.js services are on different networks.
Docker’s DNS resolution prevents local services from reaching Kafka.

Solution: Use Docker’s Built-in DNS

Containers in the same network can resolve each other by name.
Use kafka-server:9092 inside Docker.
Use 192.168.1.10:9092 outside Docker.

Fixing Docker Name Resolution for Node.js

If your Node.js service runs inside a container, ensure it is in the same network:

docker network connect app-tier nodejs-container

Then, use:

const kafka = new Kafka({
    clientId: 'cart.service',
    brokers: ['localhost:9092'],
    retry: { retries: 3 }
});

Final Working Node.js Kafka Integration

Kafka Producer & Consumer Code

import express from "express";
import { Kafka, Partitioners } from "kafkajs";

const app = express();
app.use(express.json());

const kafka = new Kafka({
    clientId: 'cart.service',
    brokers: ['localhost:9092'],
    retry: { retries: 3 }
});

const producer = kafka.producer({ createPartitioner: Partitioners.DefaultPartitioner });
const consumer = kafka.consumer({ groupId: 'cart.service' });

const runConsumer = async () => {
    await consumer.connect();
    await consumer.subscribe({ topic: "cart.item.add" });
    await consumer.run({
        eachMessage: async ({ message }) => {
            console.log("Received message:", message.value?.toString());
        }
    });
};

runConsumer().catch(console.error);

export const kafkaProducer = async (topic: string, message: Record<string, any>) => {
    await producer.connect();
    await producer.send({
        topic,
        messages: [{ value: JSON.stringify(message) }]
    });
};

app.get("/health-check", (_, res) => res.send("OK"));

app.post("/cart", async (req, res) => {
    const { productId, quantity } = req.body;
    await kafkaProducer("cart.item.add", { productId, quantity });
    res.send("OK");
});

app.listen(3000, () => console.log("Cart service is running on port 3000"));

app.on("close", async () => {
    await consumer.disconnect();
    await producer.disconnect();
});

Conclusion

By addressing Kafka’s networking challenges in Docker, we ensured seamless communication between a local Node.js service and a Kafka broker running in a container. Key takeaways:

✅ Use advertised.listeners to expose Kafka correctly
✅ Leverage Docker DNS for intra-container communication
✅ Ensure Kafka and Node.js services are in communication with the docker container kafka broker

Photo by Mélanie THESE on Unsplash