Distributed systems fail quietly. That's not a bug in how they're designed; it's an emergent property of building for resilience. Kafka is no exception. It will absorb punishment, rebalance, re-elect leaders, and keep running while something underneath is slowly going wrong. By the time a consumer starts throwing errors your team can see, the problem has usually been building for a while.
The tricky part isn't that Kafka lacks observability. It's that the signals exist, they're just not wired up in a way that makes the failure obvious. Here's how to actually read what Kafka is telling you before a degraded cluster becomes a production incident.
Consumer Group Lag Is the First Thing to Watch
Consumer group lag is the delta between the latest offset in a partition and the offset your consumer has actually committed. It sounds simple. In practice it's the most important number in your Kafka monitoring setup.
Lag growing consistently over time means your consumers are falling behind. A spike that recovers is usually fine. Lag that grows without recovering means your consumers cannot keep up with the produce rate, and that gap will eventually manifest as delayed processing, stale data, or a backlog so large that recovery becomes painful.
The problem is that lag is silent. Kafka doesn't raise an alert when a consumer group falls behind. The broker doesn't know or care. You have to instrument it yourself.
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--describe --group your-consumer-group
That command gives you per-partition lag. Run it once and it's a snapshot. Run it continuously and pipe it into a metrics system and it becomes a leading indicator. The key is that you need a threshold and an alert on unbounded growth, not just on lag being nonzero.
One pattern that works well: alert when lag has grown monotonically for more than N minutes. A single lag value tells you very little. A trend tells you everything.
Broker Health Is Not the Same as Cluster Health
A common mistake is treating "brokers are up" as equivalent to "Kafka is healthy." Those are different things. Brokers can be reachable and still be in a degraded state that affects your consumers.
Two metrics that matter more than broker uptime:
Under-replicated partitions. When a follower replica falls behind the leader, that partition is under-replicated. Kafka tracks this as UnderReplicatedPartitions at the broker level. Any value above zero deserves attention. A partition that stays under-replicated long enough gets removed from the in-sync replica set, which changes your durability guarantees without any obvious external error.
Active controller count. There should be exactly one active controller in the cluster at any time. If you see zero, something is wrong. If you see more than one, something is very wrong. Leadership elections happen as part of normal operation, but the controller count should resolve quickly. If it doesn't, you have a split-brain scenario or a stalled election.
These aren't hypothetical edge cases. Both can happen in production during rolling restarts, network partitions, or when a broker is under heavy GC pressure, and neither will necessarily produce obvious errors on the producer or consumer side right away.
JMX Is Where Kafka Actually Lives
Kafka exposes its internal metrics through JMX using Yammer Metrics on the broker side and the Kafka Metrics framework on the client side. If you're not scraping JMX, you're flying partially blind.
The broker-side metrics that matter most for cluster health:
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitionskafka.controller:type=KafkaController,name=ActiveControllerCountkafka.network:type=RequestMetrics,name=RequestsPerSeckafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
On the consumer side, records-lag-max from the consumer metrics is the programmatic equivalent of the CLI command above. If you're running Java consumers, this is already being tracked internally. Getting it into your alerting stack is a matter of tooling and configuration.
The gap between "metrics exist" and "metrics are actionable" is where most teams lose time. Scraping JMX into Prometheus via a JMX exporter, or routing through a purpose-built monitoring tool that understands Kafka's metric model, is the work that makes the difference. Tools like Vigilmon are built specifically to bridge that gap, handling the metric collection and surfacing lag and replication health in a way that's easier to act on than raw JMX output.
What to Actually Alert On
Not every metric needs an alert. Alert fatigue is real. Here's a minimal set that covers the most common failure modes:
- Consumer group lag growing for more than 5 minutes without recovering
-
UnderReplicatedPartitionsgreater than zero for more than 2 minutes -
ActiveControllerCountnot equal to 1 - Request handler idle ratio dropping below a threshold (this catches broker overload before it cascades)
The goal is to catch the early signal, not the downstream consequence. By the time consumers are visibly broken, you've already missed the window where intervention is easy.
Kafka's resilience is one of its best features. It's also what makes failures easy to miss. The cluster will keep running, the metrics will keep accumulating, and nothing will look obviously wrong until it suddenly does. The monitoring work isn't glamorous, but it's the difference between catching a problem at lag-is-growing and catching it at where-did-our-data-go.
Top comments (0)