Implementing a Reliable Event-Driven System with Dead Letter Queues in Golang and Redis

Event-driven architectures are powerful for handling asynchronous communication between services. However, failures are inevitable—messages might fail due to consumer crashes, processing errors, or unexpected system behavior. A Dead Letter Queue (DLQ) helps ensure fault tolerance by capturing failed messages for later review and reprocessing.

In this article, we'll extend our previous Golang + Redis Streams implementation by adding a Dead Letter Queue to handle failed events gracefully.

Why Do We Need a Dead Letter Queue?

A DLQ is essential when dealing with critical event-driven applications, such as:

Alert monitoring systems (missed alerts could have severe consequences)

Financial transactions (failed payments must be retried)

Workflow automation (ensuring no task is permanently lost)

By implementing a DLQ, we can:

✅ Capture failed events for debugging and reprocessing.
✅ Prevent failed messages from blocking other events.
✅ Improve observability and system reliability.

Architecture Overview

Our system consists of the following components:

Event Producer: Publishes messages to Redis Streams.

Event Consumer: Reads messages, processes them, and acknowledges successful processing.

Dead Letter Queue: Stores failed messages for later analysis and retrying.

Implementing the System in Golang

1. Setting Up Redis

Make sure Redis is running:

redis-server

2. Install Redis Client for Golang

We'll use the github.com/redis/go-redis/v9 package:

go get github.com/redis/go-redis/v9

3. Event Producer: Publishing Messages

This producer pushes events to the alerts stream.

package main

import (
    "context"
    "fmt"
    "log"
    "github.com/redis/go-redis/v9"
)

var ctx = context.Background()

func main() {
    client := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    })
    eventData := map[string]interface{}{
        "message": "Critical alert! Server down.",
    }
    _, err := client.XAdd(ctx, &redis.XAddArgs{
        Stream: "alerts",
        Values: eventData,
    }).Result()
    if err != nil {
        log.Fatalf("Failed to publish event: %v", err)
    }
    fmt.Println("Event published successfully")
}

4. Event Consumer with Dead Letter Queue Handling

The consumer reads events and processes them. If an error occurs, it moves the failed event to the dead-letter-queue.

package main

import (
    "context"
    "fmt"
    "log"
    "math/rand"
    "time"
    "github.com/redis/go-redis/v9"
)

var ctx = context.Background()

func processMessage(message map[string]interface{}) error {
    // Simulate random failures
    if rand.Float32() < 0.3 {
        return fmt.Errorf("Simulated processing failure")
    }
    fmt.Printf("Processed event: %v\n", message)
    return nil
}

func moveToDLQ(client *redis.Client, message map[string]interface{}) {
    _, err := client.XAdd(ctx, &redis.XAddArgs{
        Stream: "dead-letter-queue",
        Values: message,
    }).Result()
    if err != nil {
        log.Printf("Failed to move message to DLQ: %v", err)
    }
    fmt.Println("Message moved to DLQ")
}

func main() {
    rand.Seed(time.Now().UnixNano())
    client := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    })

    for {
        res, err := client.XRead(ctx, &redis.XReadArgs{
            Streams: []string{"alerts", "$"},
            Count:   1,
            Block:   0,
        }).Result()
        if err != nil {
            log.Fatalf("Failed to read event: %v", err)
        }

        for _, stream := range res {
            for _, message := range stream.Messages {
                err := processMessage(message.Values)
                if err != nil {
                    fmt.Println("Processing failed, moving to DLQ...")
                    moveToDLQ(client, message.Values)
                }
            }
        }
    }
}

5. Monitoring and Reprocessing Failed Events

To manually check messages in the DLQ:

redis-cli XREAD STREAMS dead-letter-queue 0

To retry processing failed messages:

func retryFailedMessages(client *redis.Client) {
    res, err := client.XRead(ctx, &redis.XReadArgs{
        Streams: []string{"dead-letter-queue", "0"},
        Count:   10,
        Block:   0,
    }).Result()
    if err != nil {
        log.Fatalf("Failed to read DLQ: %v", err)
    }

    for _, stream := range res {
        for _, message := range stream.Messages {
            fmt.Printf("Retrying event: %v\n", message.Values)
            err := processMessage(message.Values)
            if err == nil {
                fmt.Println("Successfully reprocessed event")
            }
        }
    }
}

Enhancements for Production

For a production system, consider:

Automated retries with exponential backoff.

DLQ monitoring with alerts on message buildup.

Retention policies to avoid infinite storage growth.

Conclusion

Adding a Dead Letter Queue to an event-driven system significantly improves reliability by preventing failed events from being lost. Redis Streams makes it easy to implement a DLQ, allowing us to capture, inspect, and retry failed messages efficiently.

Have you implemented DLQs in your systems? Let me know your thoughts! 🚀