DEV Community: Kneonix

Kafka UI in Action: Monitoring and Managing Kafka Like a Pro

Kneonix — Sun, 29 Jun 2025 23:26:59 +0000

Why You Need to Monitor Your Kafka Broker

Recently, I was working with Apache Kafka as the message broker for a microservice architecture. I quickly realized that it can be difficult to verify if Kafka is running correctly, especially when the rest of your application hasn’t finished connecting. In scenarios like this, a monitoring tool can be incredibly helpful and that’s where Kafka UI shines.

What Can You Do With Kafka UI?

Kafka UI is a powerful, lightweight monitoring tool for Apache Kafka. Here's what it offers out of the box:

Multi-Cluster Management : Monitor and manage multiple Kafka clusters in one place.
Performance Monitoring with Metrics Dashboard : Track essential Kafka metrics through an intuitive dashboard.
View Kafka Brokers : Inspect topic and partition assignments, controller status, and broker metadata.
View Kafka Topics : See partition count, replication status, and custom topic configurations.
View Consumer Groups : Monitor per-partition parked offsets and consumer lag (combined and per partition).
Browse Messages : View Kafka messages in JSON, plain text, or Avro formats.
Dynamic Topic Configuration : Create and configure new topics dynamically.
Configurable Authentication : Secure the UI using GitHub, GitLab, or Google OAuth 2.0.
Custom Serialization/Deserialization Plugins : Use built-in serializers like AWS Glue or Smile, or implement your own.
Role-Based Access Control (RBAC) : Manage fine-grained access permissions to the UI.
Data Masking : Obfuscate sensitive data in topic messages.

How to Set Up a Kafka Broker in Your System

You can run Kafka and Zookeeper in two main ways: Docker and Kubernetes.

Option 1: Docker

Here’s a simple docker-compose.yml setup:

version: "3.8"

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"

Run:

docker-compose up -d

Option 2: Kubernetes

Create two YAML files: zookeeper.depl.yaml and kafka.depl.yaml.

`zookeeper.depl.yaml`

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zookeeper-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
        - name: zookeeper
          image: confluentinc/cp-zookeeper:latest
          ports:
            - containerPort: 2181
          env:
            - name: ZOOKEEPER_CLIENT_PORT
              value: "2181"
            - name: ZOOKEEPER_TICK_TIME
              value: "2000"
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper-srv
spec:
  selector:
    app: zookeeper
  ports:
    - name: zookeeper
      protocol: TCP
      port: 2181
      targetPort: 2181

`kafka.depl.yaml`

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-broker-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-broker
  template:
    metadata:
      labels:
        app: kafka-broker
    spec:
      containers:
        - name: kafka-broker
          image: confluentinc/cp-kafka:latest
          ports:
            - containerPort: 9092
          env:
            - name: KAFKA_BROKER_ID
              value: "1"
            - name: KAFKA_ZOOKEEPER_CONNECT
              value: "zookeeper-srv:2181"
            - name: KAFKA_ADVERTISED_LISTENERS
              value: PLAINTEXT://kafka-broker-srv:9092
            - name: KAFKA_LISTENERS
              value: PLAINTEXT://0.0.0.0:9092
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "1"
            - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
              value: "false"
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-broker-srv
spec:
  selector:
    app: kafka-broker
  ports:
    - name: kafka-broker
      protocol: TCP
      port: 9092
      targetPort: 9092

Apply them by these command:

kubectl apply -f zookeeper.depl.yaml
kubectl apply -f kafka.depl.yaml

How to Set Up Kafka UI

You can deploy Kafka UI via Docker or Kubernetes as well.

Option 1: Docker

Add this to your docker-compose.yml file:

kafka-ui:
  image: provectuslabs/kafka-ui:latest
  container_name: kafka-ui
  ports:
    - "8080:8080"
  depends_on:
    - kafka
  environment:
    KAFKA_CLUSTERS_0_NAME: "local"
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka:9092"

Then run:

docker-compose up -d

Option 2: Kubernetes

Create a kafkaUI.depl.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-ui-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-ui
  template:
    metadata:
      labels:
        app: kafka-ui
    spec:
      containers:
        - name: kafka-ui
          image: provectuslabs/kafka-ui:latest
          ports:
            - containerPort: 8080
          env:
            - name: KAFKA_CLUSTERS_0_NAME
              value: "local"
            - name: KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS
              value: "kafka-broker-srv:9092"
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-ui-srv
spec:
  selector:
    app: kafka-ui
  ports:
    - name: kafka-ui
      protocol: TCP
      port: 8080
      targetPort: 8080

Apply it:

kubectl apply -f kafkaUI.depl.yaml

Viewing the UI

If you’re running this on Kubernetes, forward the port:

kubectl port-forward service/kafka-ui-srv 8080:8080

Then open your browser and go to http://localhost:8080 to monitor and manage your Kafka broker, topics, and consumer groups.

Learn More

GitHub: https://github.com/provectus/kafka-ui
Documentation: https://docs.kafka-ui.provectus.io/

Monitoring using Proxy Design Pattern with Node.js

Kneonix — Fri, 04 Apr 2025 14:35:17 +0000

Introduction

why we need monitoring?
Imagine you have multiple machines (servers) running simultaneously, each located in a different place. To ensure they operate smoothly without issues, we need access to their logs and performance data.

This is where a good monitoring system comes handy.

What Will We Cover?
In this article, we’ll see how to use the Proxy pattern in Node.js to enhance monitoring and performance tracking.

What is Proxy design pattern?

Proxy categories as a structural and object-relational design pattern.
You might known Proxies to move a method calls over the internet.

The Proxy provides a surrogate or placeholder for another object to control access to it.

For example, imagine you have an object A on a server located across the world and you want to access to that. Instead of interacting with A directly, you create another object in your system call it B. Object B then takes responsibility for managing and controlling access to A for you.

Why Use Proxy for Monitoring?

The Proxy design pattern is great choice for monitoring because

Its lets you perform actions before and after send the request, that make your code decouple and maintainable.
Its allows you to logging every request that send to the object service and record them too.
By placing monitoring logic within the proxy, you can centralize monitoring functionality. This avoids scattering monitoring code throughout the client or the service object, making the monitoring implementation easier to manage and maintain.

Implementation

setup our project

First, you need to install Node.js and npm on your device.

After that, create a directory for your service server and name it service.

Inside the directory, run the following command to initial NPM and install the Express package:

npm init -y
npm install express

create service

Next, create file inside the folder and name it service.js. In this file create simple express app with POST /monitoring route.

In the same file, implement class for your data you want monitor. we'll call it ObjectService and create object from it named service.

The ObjectService class should store data related to the machine and include a monitoring method that returns its data.

Our /monitoring route will expect data in req.body, which is in our example is the city we want to monitor. Using this value, the route will call .monitoring(data) and return the result to the client.

const express = require('express');
const app = express();
app.use(express.json());

class ObjectService {
  location = 'Madrid';
  state = 'state one'
  monitoring(data) {
    console.log(`Real Service monitoring data: ${data}`);
    return `the location is ${this.location},
    the state is ${this.state}`
  }
}


const service = new ObjectService();

app.post('/monitoring', (req, res) => {
  const { data } = req.body;
  const result = service.monitoring(data);
  res.send({ result });
});

app.listen(3000, () => {
  console.log('Service running on port 3000');
});

At the end to run our service we use this command:

node service.js

create proxy

Navigate to the main directory and then create folder named monitoring. Inside that create a file named monitoring.js

Javascript has built-in proxy object. in order to use that, we first create class with a property to carrying our proxy instance. We also define a monitoring method to call the proxy’s getter method and log the result.

To create proxy we use new proxy(target, handler)

Target (The Object to Proxy): This is the object that the proxy will wrap. It can be an object, array, or even another proxy.
Handler (The Object with Trap Methods): This is an object that defines traps (interceptor methods) to customize operations on the target.

Some common traps include:
- get – Intercepts property access
- set – Intercepts property assignments
- apply – Intercepts function calls
- construct – Intercepts object instantiation
- deleteProperty – Intercepts property deletion

class MachineMonitor {
    proxy;
    constructor(location) {
        this.proxy = new Proxy({ location }, proxyHandler);
    }
    async monitoring() {
        console.log(await this.proxy.monitoring);
    }
}
const machineMonitor = new MachineMonitor("Madrid");

Now we should create the proxyHandler object which has getter method inside that.

The get method takes two argument:

Target Object: The object passed to the proxy.
Property: The property being accessed. For example, if we call this.proxy.x, the property is "x".

Finally after we add the proxyHandler object to our code it will look like this:

const proxyHandler = {
    async get(target, property) {
        // check the getter property
        if (property === "monitoring") {
            // call the service
            const response = await fetch("http://localhost:5000/monitoring", {
                method: "POST",
                headers: {
                    "Content-Type": "application/json",
                },
                // get location from the target object
                body: JSON.stringify({ data: target.location }),
            });
            const res = await response.json();
            return res;
        }
    },
};

class MachineMonitor {
    proxy;
    constructor(location) {
        this.proxy = new Proxy({ location }, proxyHandler);
    }
    async monitoring() {
        console.log(await this.proxy.monitoring);
    }
}

const machineMonitor = new MachineMonitor("Madrid");
await machineMonitor.monitoring();

Now open up a new terminal then navigate to monitoring folder and run this command

node monitoring.js

Then you will see the request send to our service and in our monitoring terminal we see

{ result: 'the location is Madrid,\n    the state is state one' }

Also in our other terminal which we run the express service, we can see this:

Service running on port 3000
Real Service monitoring data: Madrid

Use Cases and Trade-offs

One of the main use cases of the Proxy pattern with monitoring is when you decide to create a distributed system with multiple servers.

Some trade-offs are:

Performance Overhead
Complexity
Limited Control Over the Target
Decoupled Code
Scalability Concerns

Conclusion

In this article, we've explored how the Proxy design pattern can be used in Node.js for monitoring. By acting as an intermediary between clients and the actual service, the Proxy pattern provides a clean way to manage monitoring tasks, ensuring that your code remains decoupled and modular.

While there are trade-offs, such as potential performance overhead and increased complexity, the benefits of easier management and centralized control make this approach a valuable tool in this type of scenario. Ultimately, choosing to implement the Proxy pattern depends on your specific requirements and the scale of your application.