DEV Community

William Rodriguez
William Rodriguez

Posted on

You decide when done is done: Manual offset commits in Kafka.

#opensource

You decide when done is done: Manual offset commits in Kafka.

Day 06 of the WKafka Open-Source Engineering Series.

If your consumer crashes right after reading a message with auto_commit=True, that message is gone forever. Manual commit puts you in control.

The Pain Points We Faced

  • Auto-commit marking messages as 'processed' before database writes finish
  • Silent data loss when consumer crashes halfway through business logic
  • Ghost accounting discrepancies caused by premature offset commits

The Implementation 

from wkafka import WKafka

kafka = WKafka(auto_commit=False)

@kafka.consumer(topic="financial_ops", format="json")
def on_financial(msg):
    save_to_database(msg.value)   # 1. Database commit succeeds
    msg.commit()                  # 2. Only now is offset committed!
Enter fullscreen mode Exit fullscreen mode

Why This Architecture Wins

  • Zero Data Loss: auto_commit=False prevents premature offset advancement.
  • msg.commit(): Commit only after successful side-effects and database writes.
  • Clean Replay: Uncommitted messages are reprocessed immediately upon recovery.

Verification & Status

Tested and verified with Apache Kafka against real broker clusters. Compatible with Python 3.9 through 3.14 with strict typing.

Top comments (1)