Your Kafka cluster is healthy. But the event is still missing

A recent r/dataengineering question described a familiar setup: many microservices publish JSON events to Kafka, each project has an expected throughput, and the business wants an alert when those events stop arriving.

The natural response is to look for a Kafka monitoring tool. That is useful, but it solves only part of the problem.

A Kafka dashboard can tell you that the brokers are available, partitions have leaders, replicas are in sync, and consumers are not falling behind. Every panel can be green while an upstream service has stopped producing the one event the business needs.

The system is healthy. The data product is not.

That distinction changes what we need to monitor.

Kafka health and business health are different signals

Apache Kafka's monitoring guidance covers the platform well. It recommends watching message and byte rates, request latency, fetch rates, replica state, and consumer lag. Those metrics answer important operational questions:

• Can producers reach the cluster?
• Are brokers accepting requests?
• Are replicas keeping up?
• Are consumers processing records fast enough?
• Is one partition behaving differently from the others?

They do not answer whether the expected invoice.created event arrived for a particular market during the last fifteen minutes.

Consumer lag is a good example. A lag of zero sounds healthy. It can also mean that no records were produced. If the source application failed before publishing, the consumer has nothing to read and therefore nothing to lag behind.

The same problem appears with throughput. An aggregate topic rate may look normal while one tenant, event type, or producer has gone silent. A busy topic can hide a very specific outage.

I would separate monitoring into four layers:

Kafka infrastructure
        ↓
Message flow
        ↓
Event contract
        ↓
Business expectation

Each layer needs different evidence and a different owner.

Layer 1: prove the platform can carry messages

The first layer is normal Kafka operations. Watch broker availability, request errors, under-replicated partitions, ISR changes, disk pressure, network saturation, produce latency, fetch latency, and authentication failures.

This is where JMX exporters, a managed Kafka metrics API, Prometheus, Grafana, or an observability platform fit. The product matters less than collecting the right broker and client signals.

Consumer lag belongs here too, although I would treat it as a flow signal rather than a complete service-level indicator. Kafka exposes records-lag, records-lag-max, fetch rates, commit latency, and time between polls. Those metrics can identify a slow or stuck consumer.

An alert such as this is useful:

consumer lag is increasing
AND
records consumed per second is below the normal range
FOR ten minutes

It tells us that records exist and the consumer is not keeping up. It still says nothing about records that were never published.

Layer 2: prove messages move through the expected path

The next layer measures flow at boundaries. For every important producer and consumer, record a small set of counters and timestamps:

events produced
events accepted by Kafka
events consumed
events processed successfully
events rejected or sent to a dead-letter path
timestamp of the newest event

The labels need care. service, event_type, environment, and perhaps region are usually manageable. Raw customer IDs, transaction IDs, and message IDs are not. Putting high-cardinality business identifiers into a metrics system is an efficient way to turn an observability improvement into a cost incident.

Keep per-message identifiers in logs or traces. Keep metrics dimensions bounded.

OpenTelemetry's messaging conventions provide common attributes for messaging systems, destinations, operations, consumer groups, and message context. The conventions are still marked as development in several areas, so I would pin the version used by the instrumentation instead of assuming the attribute names will never change.

Tracing is helpful when a business operation crosses several services. A correlation or conversation ID can connect the API request, producer span, Kafka operation, consumer span, and downstream write. That gives an engineer a route through the failure instead of a collection of unrelated charts.

Tracing every message may be too expensive at high volume. Sampling is reasonable for diagnostics. Counts and freshness metrics should remain complete because they drive alerts.

Layer 3: prove the event is usable

Arrival is not success.

A producer can publish malformed JSON at the expected rate. A serializer can omit a required field. A schema change can preserve valid syntax while changing the meaning of a value. The throughput chart will look excellent right up to the point where someone opens the downstream report.

Validate the contract where ownership is clearest. That may be in the producer before publish, in a schema registry, or at the consumer boundary. Record at least:

• schema validation failures
• unsupported schema versions
• deserialization failures
• required-field failures
• duplicate or out-of-order events when those conditions matter
• dead-letter volume and age

Do not collapse all of these into processing_error_total. The response to a broken schema is different from the response to a timeout writing to a database.

I would also resist putting the full payload into observability events. It creates a second, poorly governed copy of potentially sensitive data. Record the failure category, event type, schema version, and a safe lookup reference. Keep payload access inside the data platform's normal security boundary.

Layer 4: encode the business expectation

This is the part a Kafka dashboard cannot infer.

"Expected TPS" sounds like a threshold, but a single number is rarely enough. Traffic changes by hour, weekday, region, and business calendar. Some event types are continuous. Others arrive in bursts after a batch closes. A flat minimum can create noise during quiet periods and miss a partial outage during busy ones.

Represent the expectation explicitly:

event_type: invoice.created
producer: billing-service
environment: production
schedule: weekdays 08:00-20:00 Europe/London
minimum_events: 300
window: 5m
maximum_silence: 10m
owner: billing-platform
severity: high
runbook: https://example.internal/runbooks/invoice-events

This is configuration, not dashboard decoration. Put it in version control. Review changes. Give each expectation an owner.

The evaluation process can be simple:

expected event rules
        +
observed event counters and freshness
        ↓
evaluation job
        ↓
pass, warn, fail, or no-data
        ↓
alert with owner and runbook

The no-data state matters. Monitoring systems often treat a missing time series differently from a zero value. Your evaluator must decide what absence means for each rule. Prometheus alerting rules support duration-based conditions through for, which helps avoid paging on a brief gap. The business rule still has to define how long a gap is acceptable.

Reconcile counts instead of trusting one point

For important flows, compare counts across boundaries over the same business window:

producer accepted:  100,000
Kafka observed:      99,998
consumer processed:  99,990
target committed:    99,987

Those numbers do not prove which eight or thirteen records are missing, but they tell you where to investigate.

Use event IDs or business keys for periodic reconciliation when exact completeness matters. Counters are operational signals. Reconciliation is evidence.

The window must account for retries and late arrivals. Comparing two live counters at the current second will generate false differences because each stage observes the event at a different time. Close a window, allow a defined lateness period, then evaluate it. Streaming does not remove the need for accounting.

Alert the team that can act

The Kafka platform team owns broker capacity, replication, cluster access, and platform availability. The producer team owns whether it emits the promised event. The consumer team owns processing and downstream delivery. A data product owner may own the business expectation.

Sending every alert to the Kafka team recreates the old middleware support queue with newer software.

An alert should include:

• what expectation failed
• when the last valid event arrived
• expected and observed counts
• the affected producer, topic, event type, and consumer
• whether Kafka itself is healthy
• the owning team
• the first diagnostic query or dashboard
• the runbook

"Kafka events missing" is not enough. "invoice.created from billing-service has been silent for 12 minutes; brokers are healthy and other producers are active" is actionable.

Start with the important flows

I would not begin by buying a broad data-observability platform or instrumenting every event type. Start with the ten flows whose absence causes money, compliance, or customer impact.

For each one, define:

1. The event contract.
2. The expected schedule and volume.
3. The maximum acceptable silence.
4. The producer and consumer owners.
5. The reconciliation rule.
6. The response when the rule fails.

Then use the monitoring stack already in place. Add a new product only when the existing stack cannot express, evaluate, or route these rules without unreasonable work.

The useful dashboard is not the one with the most Kafka metrics. It is the one that can distinguish a broken broker, a slow consumer, an invalid event, and a producer that stopped doing its job.

Kafka health is necessary. The business still needs proof that the right event arrived.