At twelve minutes past four on a Saturday morning, one partition of our shipments topic stopped moving. We found out at nine, when a depot rang to ask where their morning manifest was. Nine thousand four hundred events behind, four and three quarter hours of no dispatches for every customer whose key hashed to that partition, and not one alert.
A message carried the text N A in a field that holds a weight in kilograms. An upstream team had made that field free text the previous week so an operator could record a manual override, which is a change to a producer that nobody would think to mention to a consumer. Our deserialiser threw. The consumer caught the exception, logged it at warning level, did not commit the offset, waited a second and read the same record again. Then again. It did that for four and a half hours and wrote about fourteen thousand identical warning lines, which is a volume nobody queries and no rule watches.
Ordering is the point of a partition, so nothing behind that offset could be processed while it sat there. The other five partitions were entirely healthy, and our alert read total consumer lag across the topic, where nine thousand stuck events looked like a busy night.
Failures are typed now. A deserialisation or validation error is not transient and never gets retried: the record goes to a dead letter topic with its raw bytes, offset, partition and the exception, the offset commits, and the consumer carries on. Genuinely transient failures, a database timeout or a closed connection, retry five times with backoff and then go the same way. Anything arriving in the dead letter topic pages during working hours and posts the payload, and a small tool replays a record once the producer or our parser has been fixed. Lag is alerted per partition on the oldest uncommitted message, not summed across the topic, because a sum can hide a stopped partition behind five busy ones.
Forty records went to the dead letter topic in the first fortnight, thirty four of them that same free text field.
Retrying forever is not patience. It is a decision to hold up everything behind the one thing you cannot handle, and a consumer with nowhere to put what it cannot process hands a stop button to every producer upstream.
– Sergey Shinder
Top comments (0)