🎯 Using Redis Streams Over Pub/Sub: A Practical Guide for Backend Engineers

Geetansh Garg — Fri, 20 Jun 2025 11:09:50 +0000

Hey DEV Community! 👋

I'm Geetansh Garg, a backend engineer who loves building real-time, scalable systems. In my previous post, I shared how we built a real-time notification service using FastAPI and Redis Streams.

Today, I want to go deeper into why we moved from Redis Pub/Sub to Redis Streams — and why this shift made our real-time infrastructure much more reliable and production-ready.

❓ Why Redis Pub/Sub Didn't Work for Us

We initially chose Redis Pub/Sub because it was simple to implement and worked well in early testing. But as the system grew, we hit several limitations:

❌ Message loss: If a user wasn't connected when the message was published, it was gone forever.
❌ No persistence: Pub/Sub doesn’t store message history.
❌ No retries or acknowledgment: We couldn’t confirm message delivery.
❌ Hard to scale: Horizontal scaling of consumers was error-prone and lacked coordination.

In short, Pub/Sub is great for fire-and-forget messages, but not suitable when reliability matters.

✅ Why Redis Streams Was a Better Fit

We explored alternatives and landed on Redis Streams, which offered exactly what we needed:

✅ Durability: Messages are stored until consumed.
✅ Consumer Groups: Support for multiple parallel consumers with coordinated delivery.
✅ Acknowledgement: Messages are marked delivered only after explicit acknowledgment.
✅ Replayability: Consumers can resume from the last processed ID.

These features helped us build a fault-tolerant, reliable real-time pipeline for notifications, especially for users who might be temporarily offline.

⚙️ Sample Code: Redis Streams in Action

Here’s how we use Redis Streams for our notification service:

1. Pushing a notification to the stream:

redis.xadd("notifications_stream", {
    "user_id": "123",
    "message": "You’ve got a new alert",
    "lang": "en",
    "priority": "high"
})

2. Reading from the stream as a consumer:

redis.xreadgroup(
    groupname="notifier_group",
    consumername="worker-1",
    streams={"notifications_stream": ">"}
)

3. Acknowledging processed messages:

redis.xack("notifications_stream", "notifier_group", message_id)

We also set up logic to retry unacknowledged messages using XPENDING and XCLAIM.

🧠 Lessons Learned

Redis Streams was a game-changer for our use case, allowing us to scale safely and avoid message loss.

We added a background task to periodically trim the stream using XTRIM to keep memory usage in check.
Creating a resumable consumer logic gave us high availability even during service restarts.
Implementing a WebSocket layer on top helped us push these events live to users with JWT-secured sessions.

🛠️ When NOT to Use Redis Streams

Redis Streams isn’t the perfect fit for all cases. You might not want to use it when:

⚡ You need ultra-low latency fire-and-forget messaging (Pub/Sub is simpler here).
🌀 Your system has complex message routing or massive scale — in that case, Kafka might be a better tool.
🔍 You don’t need history or delivery guarantees.

🔚 Final Takeaway

Redis Streams gave us exactly what we needed — a scalable, reliable, and production-friendly way to deliver real-time notifications with FastAPI and WebSockets.

If you're building something similar, I highly recommend giving Redis Streams a serious look — especially if you're struggling with Pub/Sub limitations.

💬 Let’s Talk!

Have you worked with Redis Streams or built similar systems?

Would love to hear how you tackled durability, replay, or scale challenges!

Drop a comment or DM — always happy to nerd out on system design 😄

🚀 Building a Real-Time Notification Service with FastAPI, Redis Streams, and WebSockets

Geetansh Garg — Fri, 20 Jun 2025 10:47:28 +0000

Hi DEV Community! 👋

I'm Geetansh Garg, a Senior Software Engineer passionate about scalable backend systems, clean architecture, and real-world problem solving. This is my first post here, and I wanted to kick things off by sharing one of the most exciting problems I’ve been working on lately.

🧩 The Problem

We needed to deliver real-time notifications to users in a FastAPI-based product. But it wasn’t just about sending messages — we had to handle:

Scalable delivery (thousands of users)
Persistence (so no message is lost)
Unread tracking
Internationalization (i18n)
Prioritization & retries

The traditional Redis Pub/Sub model just didn’t cut it — messages would be lost if a user was offline. So we decided to re-architect using Redis Streams for reliable and replayable messaging.

🛠️ The Stack

FastAPI: Async Python framework for APIs and WebSockets
Redis Streams: Durable, ordered messaging system
PostgreSQL: To store notification history and unread status
WebSockets: For pushing updates in real-time to the user
Pydantic & SQLAlchemy: For models and data validation
Docker + AWS Lambda (future): For production-ready deployment

🔄 How It Works

A notification is created (user action, cron job, webhook, etc.)
It's pushed into a Redis Stream with metadata like user_id, priority, lang, etc.
A Redis Stream consumer reads it, stores it in PostgreSQL, and pushes to the WebSocket server.
If the user is connected — great, they get it instantly!
If not, it’s stored, marked as unread, and delivered when they come back online.

✨ Key Highlights

✅ Message durability with Redis Streams
🔄 Replay logic in consumer to handle restarts
🌍 Multi-language templates per user
🔒 Secure WebSocket authentication using JWT and x-api-key
📬 Push fallback logic coming soon (Slack/Email integration)

📈 What's Next?

I’m looking to open-source this soon and write detailed breakdowns of:

Redis Streams vs Pub/Sub for real-time systems
Notification retry and backoff strategies
WebSocket gateway with reconnect handling

💬 Let’s Connect!

Have you built similar real-time systems? Or explored Redis Streams for production workloads?

Let’s connect, geek out, and share learnings!

Also, I’m open to feedback — especially since this is my first post 😄

DEV Community: Geetansh Garg