Subscribe to dozens of Kafka topics with a single regex pattern.
Day 09 of the WKafka Open-Source Engineering Series.
In IoT and multi-tenant architectures, topics spawn dynamically. Hardcoding topic lists is an antipattern. Use WKafka regex pattern subscriptions.
The Pain Points We Faced
- Redeploying consumers every time a new device or tenant topic is created
- Duplicating identical consumer code across multiple topic subscribers
- Manual topic list synchronization leading to configuration drift
The Implementation
@kafka.consumer(topic_pattern=r"sensor_.*", format="json")
def on_telemetry(msg):
# Handles events from sensor_temp, sensor_humidity, sensor_pressure...
print(f"[{msg.topic}] Reading: {msg.value}")
Why This Architecture Wins
- Regex Discovery: topic_pattern='sensor_.*' auto-subscribes to matching topics.
- Dynamic Rebalance: Discovers newly created broker topics without restarting.
- Single Handler: Centralize telemetry logic across hundreds of streams.
Verification & Status
Tested and verified with Apache Kafka against real broker clusters (see EXAMPLES_STATUS.md in repository). Compatible with Python 3.9 through 3.14 with strict typing.
Top comments (1)
In event-driven Python microservices, maintaining explicit control over consumer rebalance timeouts while executing decoupled handlers is essential. Declarative subscription decorators streamline message routing while keeping offset commit contracts strictly deterministic.
Do your architectures favor centralized orchestrator sinks or localized idempotent consumer handlers for event pipelines?