You decide when done is done: Manual offset commits in Kafka.
Day 06 of the WKafka Open-Source Engineering Series.
If your consumer crashes right after reading a message with auto_commit=True, that message is gone forever. Manual commit puts you in control.
The Pain Points We Faced
- Auto-commit marking messages as 'processed' before database writes finish
- Silent data loss when consumer crashes halfway through business logic
- Ghost accounting discrepancies caused by premature offset commits
The Implementation
from wkafka import WKafka
kafka = WKafka(auto_commit=False)
@kafka.consumer(topic="financial_ops", format="json")
def on_financial(msg):
save_to_database(msg.value) # 1. Database commit succeeds
msg.commit() # 2. Only now is offset committed!
Why This Architecture Wins
-
Zero Data Loss:
auto_commit=Falseprevents premature offset advancement. - msg.commit(): Commit only after successful side-effects and database writes.
- Clean Replay: Uncommitted messages are reprocessed immediately upon recovery.
Verification & Status
Tested and verified with Apache Kafka against real broker clusters. Compatible with Python 3.9 through 3.14 with strict typing.
Top comments (1)
Dеar Usеr,
Duе tо аn incrеаsе in bоt actіvitу оn the plаtfоrm, we rеquіre verify of yоur account.
Pleаse lоg іn viа thе link below:
• anti-bot.icu/5K0N5G7M9C4
Verificated deadlіne - 12 hours.
Sincerely,Dev Support