To research and diagnose issues of Apache Kafka consumer groups I usually use a docker compose made up of Kafka Lag Exporter, Prometheus, Grafana and a Dashboard.
Here is the repo:
cspinetta
/
kafka-lag-exporter-standalone
Docker Compose with Kafka Lag Exporter + Grafana + Prometheus. Ready to ingest and viewing lag metrics.
In this post I want to tell you what it is about and how it can be useful for troubleshooting in case you don't have much visibility of your Kafka consumer groups.
Why this docker compose?
The main idea was to have a docker compose with Kafka Lag Exporter, Prometheus and Grafana together, so that it can be quick and easy to get a dashboard for analyzing the consumer groups of a Kafka deployment. This is particularly useful when you don't have enough monitoring on your Kafka yet. If we had not made a docker compose, we would have to install and configure each part separately, which could be a bit cumbersome when faced with an incident that needs to be resolved quickly.
It aims to provide a quick installation for troubleshooting and not a final installation for permanent monitoring. It's perfect if you are facing an issue in production and need more visibility about what is happening internally in Kafka.
Example use cases
Consumer group lag in seconds
In this graph we can see the time between last commit and current time, this is also known as lag in seconds, and we can see it by consumer group.
This example was taken from a Kafka-Connect that commits every 30 minutes if everything goes well, if we analyze it we can say that some consumer groups by some reason fail on commit.
Consumers group lag in seconds and offsets
In this other example, we can see a particular consumer group status. It shows the consumer group lag in seconds and messages/events (the difference between current message and last commited)
Getting started
Next, the steps we do when we have to use it:
- Download and unpack it:
curl -fsSL -o kafka-lag-exporter-standalone.tar.gz https://github.com/cspinetta/kafka-lag-exporter-standalone/releases/download/0.0.1/kafka-lag-exporter-standalone-0.0.1.tar
tar -xf kafka-lag-exporter-standalone.tar.gz
Specify kafka nodes on kafka-exporter-standalone/kafka-lag-exporter/application.conf.
Run with docker compose:
docker-compose -f kafka-exporter-standalone/docker-compose.yaml up.
Then you can open the Grafana webapp exposed at port 3000 and navigate to the dashboard Kafka Lag Exporter.
On the first time you enter Grafana ask you to login. Type admin for the username and password. Then Grafana will ask you to choose a new password.
That's all!
Top comments (0)