Stop poison pills from halting your Kafka pipeline: Retries and DLQ.

Day 08 of the WKafka Open-Source Engineering Series.

A single malformed JSON payload should never bring down a production Kafka consumer. WKafka isolates poison pills into a Dead Letter Queue automatically.

The Pain Points We Faced

• A single corrupt payload blocking consumer partitions in infinite crash loops
• Uncontrolled retry storms overwhelming downstream databases
• Losing failed payloads without audit trails or forensic debugging traces

The Implementation

@kafka.consumer(
    topic="invoices",
    max_retries=3,
    retry_backoff=2.0,
    dlq_topic="invoices.DLQ"
)
def process_invoice(msg):
    # If this fails 3 times, message automatically moves to invoices.DLQ
    parse_and_bill(msg.value)

Why This Architecture Wins

• Dead Letter Queue: Automatically routes permanently failing records to topic.DLQ.
• Exponential Backoff: Smooth retry intervals prevent thundering herds on recovery.
• Zero Pipeline Halts: Healthy messages continue processing while errors are quarantined.

Verification & Status

Tested and verified with Apache Kafka against real broker clusters (see EXAMPLES_STATUS.md in repository). Compatible with Python 3.9 through 3.14 with strict typing.

• GitHub: https://github.com/wisrovi/wkafka
• PyPI: https://pypi.org/project/wkafka

Kafka #Python #DataEngineering #OpenSource #Wisrovi