DEV Community

Vigilmon
Vigilmon

Posted on

How to Monitor Apache Kafka with Vigilmon

#kafka #monitoring #devops #backend

How to Monitor Apache Kafka with Vigilmon

Apache Kafka is the backbone of many event-driven architectures — real-time data pipelines, event sourcing systems, and microservice communication buses. When Kafka goes down or degrades, the effects cascade: consumers stop processing, producers back up, and data pipelines stall silently.

This guide shows you how to monitor Kafka with Vigilmon — from broker health to consumer lag and heartbeat-based worker monitoring.

What Kafka Failure Modes Look Like

Kafka fails in subtle ways:

  • Broker unavailability — a broker crashes and the cluster loses replication quorum
  • Under-replicated partitions — some partitions drop below the required replication factor silently
  • Consumer lag spike — producers write faster than consumers can process; lag builds up without triggering errors
  • Producer connection failures — producers can't connect but log only to local files
  • ZooKeeper/KRaft coordination failures — the metadata layer breaks, causing cluster instability

Step 1: Expose a Kafka Health Endpoint

Kafka doesn't expose an HTTP health endpoint natively. The simplest approach is to add a health check sidecar:

Option A: Node.js Health Probe 

// kafka-health-probe.js
const { Kafka } = require('kafkajs');
const http = require('http');

const kafka = new Kafka({
  clientId: 'health-probe',
  brokers: (process.env.KAFKA_BROKERS || 'localhost:9092').split(','),
});

const admin = kafka.admin();

http.createServer(async (req, res) => {
  if (req.url !== '/health') return res.end();
  try {
    await admin.connect();
    const metadata = await admin.fetchTopicMetadata();
    const brokerCount = metadata.brokers?.length || 0;
    await admin.disconnect();
    res.writeHead(brokerCount > 0 ? 200 : 503);
    res.end(JSON.stringify({ status: brokerCount > 0 ? 'ok' : 'degraded', brokers: brokerCount }));
  } catch (err) {
    res.writeHead(503);
    res.end(JSON.stringify({ status: 'error', error: err.message }));
  }
}).listen(9100);
Enter fullscreen mode Exit fullscreen mode

Run alongside your Kafka setup and monitor http://your-host:9100/health.

Option B: Python Health Probe 

# kafka_health.py
from kafka import KafkaAdminClient
from http.server import HTTPServer, BaseHTTPRequestHandler
import json
import os

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != '/health':
            return
        try:
            client = KafkaAdminClient(
                bootstrap_servers=os.getenv('KAFKA_BROKERS', 'localhost:9092')
            )
            brokers = client.describe_cluster()
            client.close()
            self.send_response(200)
            self.end_headers()
            self.wfile.write(json.dumps({'status': 'ok'}).encode())
        except Exception as e:
            self.send_response(503)
            self.end_headers()
            self.wfile.write(json.dumps({'status': 'error', 'error': str(e)}).encode())
    def log_message(self, *args): pass

HTTPServer(('', 9100), HealthHandler).serve_forever()
Enter fullscreen mode Exit fullscreen mode

Monitor http://your-kafka-host:9100/health with Vigilmon.

Step 2: Monitor Consumer Lag via Heartbeat

Consumer lag doesn't cause HTTP failures — it causes silent data pipeline degradation. Use a heartbeat monitor to track consumer health:

#!/bin/bash
# consumer-health-check.sh — run as a cron every minute

LAG=$(kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group your-consumer-group 2>/dev/null | \
  awk 'NR>1 && $6 ~ /^[0-9]+$/ {sum += $6} END {print sum+0}')

MAX_LAG=${MAX_LAG:-10000}

if [ "$LAG" -lt "$MAX_LAG" ]; then
  curl -sf "https://hb.vigilmon.online/YOUR-HEARTBEAT-ID" > /dev/null
fi
Enter fullscreen mode Exit fullscreen mode

This pings Vigilmon only when consumer lag is below your threshold. If lag exceeds the limit, the heartbeat stops — Vigilmon fires an alert.

Step 3: Add a Kafka JMX Metrics Endpoint (Advanced)

For production Kafka, enable JMX and expose metrics via Prometheus JMX exporter:

# jmx-kafka-config.yaml
startDelaySeconds: 0
ssl: false
lowercaseOutputName: true
rules:
  - pattern: 'kafka.server<type=(.+), name=(.+)><>Value'
    name: kafka_server_$1_$2
    type: GAUGE
  - pattern: 'kafka.controller<type=(.+), name=(.+)><>Value'
    name: kafka_controller_$1_$2
    type: GAUGE
Enter fullscreen mode Exit fullscreen mode

Expose on port 9100 and monitor the endpoint URL with Vigilmon as an HTTP(S) check.

Step 4: Monitor Kafka Connect Workers (If Applicable)

Kafka Connect has a built-in REST API:

# Check Kafka Connect worker health
curl http://localhost:8083/connectors
# Returns list of active connectors

# Check specific connector status
curl http://localhost:8083/connectors/my-connector/status
Enter fullscreen mode Exit fullscreen mode

Monitor http://your-connect-worker:8083/ — it returns 200 if the Connect worker is up.

Kafka Monitoring Coverage Table

Monitor Type Target Alert Condition
HTTP(S) Broker health probe :9100/health Status != 200
Heartbeat Consumer lag cron No ping if lag > threshold
HTTP(S) Kafka Connect REST API Status != 200
Heartbeat Schema Registry probe No ping in > 2 min
HTTP(S) Your Kafka UI (Kowl/Redpanda) Status != 200

Conclusion

Kafka is critical infrastructure, but it doesn't fail loudly. The combination of an HTTP health probe + heartbeat consumer lag monitoring gives you early warning before data pipeline problems cascade.

Start monitoring Kafka infrastructure free at vigilmon.online

Top comments (0)