The request-response pattern that Kafka doesn't give you for free.

Day 05 of the WKafka Open-Source Engineering Series.

Building reliable distributed systems means moving from synchronous HTTP blocking to asynchronous event workers. WKafka makes this pattern painless.

The Pain Points We Faced

• HTTP 504 gateway timeouts on long-running compute jobs
• Brittle tight coupling between front-facing APIs and background workers
• Lost task completion notifications when worker nodes crash mid-execution

The Implementation

from wkafka import WKafka

kafka = WKafka(servers=["localhost:9092"], client_id="task_worker_node")

# Worker Node consuming tasks and responding with results
@kafka.consumer(topic="tasks", format="json")
def on_task(msg):
    result = compute_heavy_work(msg.value)
    kafka.produce(topic="results", key=msg.key, value=result)

Why This Architecture Wins

• Decoupled Flow: Client produces task; worker consumes, computes, and responds.
• Correlation IDs: Match task requests with results via native message keys.
• Infinite Scale: Add worker pods on demand without changing client architecture.

Verification & Status

Tested and verified with Apache Kafka against real broker clusters. Compatible with Python 3.9 through 3.14 with strict typing.

• GitHub: https://github.com/wisrovi/wkafka
• PyPI: https://pypi.org/project/wkafka

Kafka #Python #DataEngineering #OpenSource #Wisrovi