DEV Community

Geetansh Garg
Geetansh Garg

Posted on

🎯 Using Redis Streams Over Pub/Sub: A Practical Guide for Backend Engineers

Hey DEV Community! 👋

I'm Geetansh Garg, a backend engineer who loves building real-time, scalable systems. In my previous post, I shared how we built a real-time notification service using FastAPI and Redis Streams.

Today, I want to go deeper into why we moved from Redis Pub/Sub to Redis Streams — and why this shift made our real-time infrastructure much more reliable and production-ready.

❓ Why Redis Pub/Sub Didn't Work for Us

We initially chose Redis Pub/Sub because it was simple to implement and worked well in early testing. But as the system grew, we hit several limitations:

  • ❌ Message loss: If a user wasn't connected when the message was published, it was gone forever.
  • ❌ No persistence: Pub/Sub doesn’t store message history.
  • ❌ No retries or acknowledgment: We couldn’t confirm message delivery.
  • ❌ Hard to scale: Horizontal scaling of consumers was error-prone and lacked coordination.

In short, Pub/Sub is great for fire-and-forget messages, but not suitable when reliability matters.

✅ Why Redis Streams Was a Better Fit

We explored alternatives and landed on Redis Streams, which offered exactly what we needed:

  • ✅ Durability: Messages are stored until consumed.
  • ✅ Consumer Groups: Support for multiple parallel consumers with coordinated delivery.
  • ✅ Acknowledgement: Messages are marked delivered only after explicit acknowledgment.
  • ✅ Replayability: Consumers can resume from the last processed ID.

These features helped us build a fault-tolerant, reliable real-time pipeline for notifications, especially for users who might be temporarily offline.

⚙️ Sample Code: Redis Streams in Action

Here’s how we use Redis Streams for our notification service:

1. Pushing a notification to the stream:

redis.xadd("notifications_stream", {
    "user_id": "123",
    "message": "You’ve got a new alert",
    "lang": "en",
    "priority": "high"
})
Enter fullscreen mode Exit fullscreen mode

2. Reading from the stream as a consumer:

redis.xreadgroup(
    groupname="notifier_group",
    consumername="worker-1",
    streams={"notifications_stream": ">"}
)
Enter fullscreen mode Exit fullscreen mode

3. Acknowledging processed messages:

redis.xack("notifications_stream", "notifier_group", message_id)
Enter fullscreen mode Exit fullscreen mode

We also set up logic to retry unacknowledged messages using XPENDING and XCLAIM.

🧠 Lessons Learned

Redis Streams was a game-changer for our use case, allowing us to scale safely and avoid message loss.

  • We added a background task to periodically trim the stream using XTRIM to keep memory usage in check.
  • Creating a resumable consumer logic gave us high availability even during service restarts.
  • Implementing a WebSocket layer on top helped us push these events live to users with JWT-secured sessions.

🛠️ When NOT to Use Redis Streams

Redis Streams isn’t the perfect fit for all cases. You might not want to use it when:

  • ⚡ You need ultra-low latency fire-and-forget messaging (Pub/Sub is simpler here).
  • 🌀 Your system has complex message routing or massive scale — in that case, Kafka might be a better tool.
  • 🔍 You don’t need history or delivery guarantees.

🔚 Final Takeaway

Redis Streams gave us exactly what we needed — a scalable, reliable, and production-friendly way to deliver real-time notifications with FastAPI and WebSockets.

If you're building something similar, I highly recommend giving Redis Streams a serious look — especially if you're struggling with Pub/Sub limitations.

💬 Let’s Talk!

Have you worked with Redis Streams or built similar systems?

Would love to hear how you tackled durability, replay, or scale challenges!

Drop a comment or DM — always happy to nerd out on system design 😄

Top comments (0)