When Your Redis Consumer Group Goes Silent

The queue was growing. The processor container was running. Nothing in the logs suggested a problem. This is the specific kind of system failure that makes you question your own perception, because every visible indicator says "fine" while the actual work just stops.

Close to midnight, weeks into building an email automation pipeline, I was looking at a Redis stream with a climbing length and a Postgres table that hadn't updated in three hours. My instinct was to look for an error. Errors have timestamps. Errors have context. You can work with an error. But there was nothing. No panics, no stack traces, not even a weird log line to pull on. The processor looked healthy. The stream looked healthy. The dashboard was green. The Postgres table was just... stopped.

I refreshed it about six times before I admitted something was structurally wrong.

That's the tax silent failures levy on you. Lost hours, and underneath them, the slow erosion of confidence in your own mental model. Maybe I was looking at the wrong environment. Maybe I had the wrong table name. I started auditing things I knew were right because the alternative, that something invisible was broken in a way I didn't have vocabulary for yet, was harder to sit with.

It took longer than I'd like to admit. When I finally traced it, the diagnostic path I'd stumbled through was worth writing down. So here it is.

When the dashboard lies

Most debugging starts with an error. Errors are unpleasant but workable: you have a stack trace, a timestamp, somewhere to start. A stuck consumer group is different. The error you're looking for doesn't exist. The system isn't wrong from its own perspective. It's doing exactly what it's configured to do. It's just not doing what you think it's configured to do.

My processor service was alive. Passing health checks. Consuming a normal amount of CPU and memory. The Redis stream was accepting new messages and its length was incrementing on schedule. The only signal that anything had failed was downstream: no new rows in Postgres. If you weren't watching that specific table, you'd miss this for hours. I almost did.

I came out of this with one opinion that keeps holding up: health monitoring for event-driven systems and health monitoring for request-response services are two different things, and most teams don't treat them that way. Process health is not pipeline health. A green container doesn't mean work is moving. In synchronous systems, a stalled handler produces timeouts, you feel the failure, often immediately. In an event-driven system, the queue is infinitely patient. It'll hold your stuck messages forever while your dashboards stay green.

Stuck consumer groups don't crash. The failure mode is a silent stall. The process doesn't die. It doesn't throw. It simply stops advancing state, and the queue absorbs the growing backlog without complaint.

What a consumer group actually is

Redis Streams documentation explains consumer groups clearly enough in the happy path. The mental model most developers walk away with: a consumer group distributes messages across multiple consumers, tracks delivery, and once a consumer acknowledges a message, it's done. Clean. Sensible.

What that optimistic model glosses over is what happens in the gap between "delivered" and "acknowledged." That gap has a name, the pending state, and it's where everything interesting happens.

When a consumer reads a message from a stream, Redis doesn't delete it. It moves it into the Pending Entries List, tagged with the consumer's name and a delivery timestamp. The expectation is that the consumer will do the work and then acknowledge to close the loop. Redis removes the entry from pending, the pipeline advances, everyone's happy.

But if the consumer reads the message and then fails, or stalls, or silently decides not to process it, the message sits in the pending list indefinitely. Other consumers in the same group don't see it. Redis already delivered it. From the stream's perspective, that work is "in progress." It will never complete unless something explicitly intervenes.

The consumer group is, functionally, an optimistic ledger. It assumes consumers finish what they start. When they don't, you end up with a pending list full of work that nothing is actively doing, and nothing that announces this loudly.

My take: the design is reasonable, but the tooling around observing the pending state has been historically underinvested. You have to know to look. Most developers don't know to look until they've had a night like mine.

The four diagnostics you run first

When you suspect a stuck consumer group, the sequence of your diagnostic commands matters. Running them in the wrong order gives you answers that don't connect. Here's the order that consistently gets me to root cause fastest.

Start with the stream itself. You want the high-level picture: total length, the ID of the last entry, timestamps. The question you're answering is simple: are messages still arriving, and how many are in the stream? A climbing stream length with no downstream progress is your first hard confirmation that work is stalling somewhere.

Then ask about the groups. Look at every consumer group on the stream and specifically at the pending count per group, the number of messages sitting in each group's Pending Entries List. If your consumer group is supposed to be processing work and that pending count is climbing rather than hovering near zero, the consumption loop is broken.

Third, get granular with the pending entries themselves. Look at individual entries: message IDs, consumer names, how long since each was delivered (the idle time), and how many times each has been delivered. This is where the diagnostic branches open. The idle times and delivery counts are your evidence.

Finally, cross-reference with the raw stream length. Comparing total stream length against pending count tells you whether the consumer has stopped picking up new messages entirely, or whether it's picking them up and then stalling internally. These are different problems with different fixes.

Four data points, under two minutes. Most of the time, the shape of the problem becomes obvious as soon as you have all four together.

Three branches, three different root causes

The pending data points you down one of three paths, and each requires a different response.

Pending entries with no live consumer. The consumer process died, restarted under a different name, or was replaced without cleaning up its old registration. The messages are tagged to a consumer that no longer exists in any operational sense. Reassign those entries to a live consumer and the pipeline can drain.

Pending entries with an idle consumer. The consumer exists, it shows up in the group, but the idle time on its pending entries is enormous, hours or days instead of seconds or milliseconds. The consumer is alive but has stopped processing. This was the surface symptom I was seeing. The actual cause turned out to be the third branch.

Dedup-key hits silently dropping work. This was my actual problem, and it was the most frustrating to find because the consumer wasn't stuck in any traditional sense. It was running. It was reading messages. It was actively deciding not to process them.

The dedup key in my pipeline was constructed incorrectly. The consumer was reading every message, checking the key, concluding it had already processed everything, and skipping, without error, without log noise, without any downstream effect. My processor wasn't broken. It was functioning exactly as designed, and the design was wrong.

The compounding issue: the code path that handled the skip didn't issue an acknowledgment. So every "already done" message stayed in the pending list indefinitely. The pending count kept climbing. Nothing was alarming about any individual component. Everything together was silent and broken.

When I finally figured this out, I sat back and laughed at myself for a moment. The processor had been working perfectly, right up to the part where it was supposed to do anything useful.

The fix sequence and what to actually watch

Don't immediately destroy the consumer group. The pending list is evidence. Fix the consumer logic first, the dedup key construction, the processing path, whatever caused the stall, then let the consumer drain the pending list naturally. Reassign stuck entries to a live consumer where needed, then watch the pending count decrease toward zero as acknowledgments flow through.

That's your verification: the pending count actually moving. Container status can fool you. So can the health endpoint. Everything else can lie. The pending count doesn't.

What to add so this never goes dark again

Expose pending entry count per consumer group as a metric. Alert when it climbs beyond a threshold tied to your expected processing latency. Expose maximum idle time across pending entries as a separate signal. Log consumer name registration explicitly. Instrument the dedup skip path so every skip decision appears somewhere observable.

These aren't sophisticated solutions. They're just closing the gap between what the system looks like and what it's actually doing. That gap is where the 11pm nights live.

The deeper thing

The stuck consumer group is an instance of a broader pattern: the component that appears healthy while failing to advance state. The queue is infinitely patient. The messages aren't going anywhere. And the system will report green indefinitely while the backlog grows.

Progress monitoring and process monitoring are not the same thing. This distinction compounds the deeper you go into event-driven systems, because the failure modes are quieter and the feedback loops are longer.

Silent failures accumulate. They let the distance grow between what the system appears to be doing and what it's actually doing, until that gap becomes undeniable.

The work of making these systems production-worthy is largely the work of closing that gap before it opens. The pending list is where the truth lives. Everything else is performance.