When a return is received at one of our warehouses, the returns service publishes an event to a Kafka topic, and a consumer group called refunds reads it and asks the payment provider to refund the customer. In October we moved to a new payment provider. To stop refunds going to the old one during the cut over, we scaled the refunds consumer to zero on the third. The topic keeps fourteen days of messages, so the plan was simple: start the consumer again when the new provider was live and let it work through the backlog.
The new provider went live on the twelfth. We started the consumer, it reported healthy, its lag showed zero within a minute, and new refunds began to go out. Over the next week customer service took about three hundred calls from people who had returned something between the third and the twelfth and been refunded nothing.
The consumer's first log lines said why. For every partition it had found no committed offset, and it was resetting to the latest one. The broker keeps a group's committed offsets for offsets.retention.minutes after the group becomes empty, and the default is seven days. Ours had been empty for nine. With no offsets to read, the consumer fell back on auto.offset.reset, which we had never set, and its default is latest. Every message from those nine days was still in the topic, and the consumer had started just after the last of them.
Lag did not warn us, because lag is measured from committed offsets, and a group with none has no lag to report. On the twelfth the group showed zero lag because it had only just begun, at the end.
We checked which of those returns had been refunded by hand in the meantime, stopped the consumer, and moved the group back to the morning of the third with kafka-consumer-groups.sh --reset-offsets --to-datetime, which only works while the group has no members. It worked through the backlog that afternoon.
The refunds consumer now sets auto.offset.reset to none, so a missing offset stops it with an error instead of choosing a position for it. Any plan that stops a consumer group for more than a day records the group's offsets first, with kafka-consumer-groups.sh --describe, and the runbook has the command to put them back. We also alert on any group we expect to exist that has no committed offsets.
We had paused a consumer and assumed the broker would keep its place. It kept the messages for fourteen days and the place for seven.
– Sergey Shinder
Top comments (0)