Subscribe to dozens of Kafka topics with a single regex pattern.

Day 09 of the WKafka Open-Source Engineering Series.

In IoT and multi-tenant architectures, topics spawn dynamically. Hardcoding topic lists is an antipattern. Use WKafka regex pattern subscriptions.

The Pain Points We Faced

• Redeploying consumers every time a new device or tenant topic is created
• Duplicating identical consumer code across multiple topic subscribers
• Manual topic list synchronization leading to configuration drift

The Implementation

@kafka.consumer(topic_pattern=r"sensor_.*", format="json")
def on_telemetry(msg):
    # Handles events from sensor_temp, sensor_humidity, sensor_pressure...
    print(f"[{msg.topic}] Reading: {msg.value}")

Why This Architecture Wins

• Regex Discovery: topic_pattern='sensor_.*' auto-subscribes to matching topics.
• Dynamic Rebalance: Discovers newly created broker topics without restarting.
• Single Handler: Centralize telemetry logic across hundreds of streams.

Verification & Status

Tested and verified with Apache Kafka against real broker clusters (see EXAMPLES_STATUS.md in repository). Compatible with Python 3.9 through 3.14 with strict typing.

• GitHub: https://github.com/wisrovi/wkafka
• PyPI: https://pypi.org/project/wkafka

Kafka #Python #DataEngineering #OpenSource #Wisrovi