DEV Community

William Rodriguez
William Rodriguez

Posted on

The request-response pattern that Kafka doesn't give you for free.

Day 05 of the WKafka Open-Source Engineering Series.

Building reliable distributed systems means moving from synchronous HTTP blocking to asynchronous event workers. WKafka makes this pattern painless.

The Pain Points We Faced

  • HTTP 504 gateway timeouts on long-running compute jobs
  • Brittle tight coupling between front-facing APIs and background workers
  • Lost task completion notifications when worker nodes crash mid-execution

The Implementation

from wkafka import WKafka

kafka = WKafka(servers=["localhost:9092"], client_id="task_worker_node")

# Worker Node consuming tasks and responding with results
@kafka.consumer(topic="tasks", format="json")
def on_task(msg):
    result = compute_heavy_work(msg.value)
    kafka.produce(topic="results", key=msg.key, value=result)
Enter fullscreen mode Exit fullscreen mode

Why This Architecture Wins

  • Decoupled Flow: Client produces task; worker consumes, computes, and responds.
  • Correlation IDs: Match task requests with results via native message keys.
  • Infinite Scale: Add worker pods on demand without changing client architecture.

Verification & Status

Tested and verified with Apache Kafka against real broker clusters. Compatible with Python 3.9 through 3.14 with strict typing.

Kafka #Python #DataEngineering #OpenSource #Wisrovi

Top comments (1)

Collapse
 
william_rodriguez_65a5898 profile image
William Rodriguez •

In event-driven Python microservices, maintaining explicit control over consumer rebalance timeouts while executing decoupled handlers is essential. Declarative subscription decorators streamline message routing while keeping offset commit contracts strictly deterministic.

Do your architectures favor centralized orchestrator sinks or localized idempotent consumer handlers for event pipelines?