Decoupled Distributed Queues & Pub/Sub in Python with WRedis

Building asynchronous message workers and real-time pub/sub channels in Python usually forces teams to adopt heavy, complex brokers before they actually need them.

WRedis v1.0.0 LTS provides declarative, lightweight abstractions for Redis Pub/Sub and Queues (RPUSH / BRPOP) with automatic JSON serialization, managed thread pools, and graceful SIGINT shutdown handling.

This is Day 02 of the WRedis Open-Source Architecture Series.

---

1. Declarative Pub/Sub with @on_message

Broadcast events across distributed microservices with zero subscription boilerplate:

from wredis.pubsub import RedisPubSubManager

pubsub = RedisPubSubManager(host="localhost", port=6379)

# Register listener callback
@pubsub.on_message("order_notifications")
def handle_notification(event):
    print(f"Received notification: {event['order_id']} -> {event['status']}")

# Publish structured payload
pubsub.publish_message(
    channel="order_notifications",
    message={"order_id": 9921, "status": "shipped"}
)

---

2. High-Throughput Queue Processing

Deploy durable FIFO task workers with automatic reconnection and concurrency governance:

from wredis.queue import RedisQueueManager

queue_mgr = RedisQueueManager(host="localhost", poll_interval=1, max_retries=3)

# Register worker
@queue_mgr.on_message("render_tasks")
def process_render(task):
    print(f"Rendering frame: {task['frame_idx']}")

# Push task to queue
queue_mgr.publish(
    queue_name="render_tasks",
    data={"frame_idx": 104, "resolution": "4K"},
    ttl=3600
)

# Start multi-threaded workers with clean shutdown
queue_mgr.start()
queue_mgr.wait()

---

Why WRedis Architecture Wins

• Thread-Managed Workers: Starts parallel consumption threads and joins them cleanly without orphan connections.
• Auto JSON Serialization: Send and receive Python dictionaries natively without manual json.dumps() or json.loads().
• 95%+ Test Coverage: Built and tested against real Redis clusters under stress workloads.

---

Install & Explore

pip install wredis

• GitHub: https://github.com/wisrovi/wredis
• PyPI: https://pypi.org/project/wredis/

Author: William Steve Rodríguez Villamizar (Wisrovi)