One Long Job Took Down Our Whole Redis Streams Worker Pool

We run a small fleet of Python workers on top of Redis Streams. One night, a single long-running job quietly took the entire interactive pool offline for 14 hours. Nothing crashed, nothing logged an error, and every worker was "busy". This post is about why, and the two-line idea that fixed it.

The setup

At PPCEye we collect Google search ads from Google's public ad archive and let people look up which advertisers run ads for a given domain. When a logged-in user opens a domain page with stale data, we enqueue a fetch job. Workers pick jobs up from Redis Streams with consumer groups:

messages = r.xreadgroup(GROUP, consumer, {STREAM: '>'}, count=1, block=4000)
# ... run the handler ...
r.xack(STREAM, GROUP, message_id)

Workers die sometimes (deploys, OOM, a bad proxy). So, like most Redis Streams tutorials recommend, every worker also periodically reclaims messages that have been pending for too long:

r.xautoclaim(STREAM, GROUP, consumer, min_idle_time=RUNNING_TIMEOUT_MS, start_id='0-0', count=10)

RUNNING_TIMEOUT was 15 minutes. If a worker took a message and never acked it within 15 minutes, we assumed it was dead and let someone else take over.

What happened

Most fetch jobs take seconds. Then a user opened a page for a very large domain, one where the archive window held almost 2 million ads. That fetch was going to run for hours.

Here is what the logs showed, one line per worker picking up the same message:

23:50:17  worker-4  claims fetch:<big-domain>
00:05:27  worker-6  claims fetch:<big-domain>
00:20:41  worker-3  claims fetch:<big-domain>
00:36:08  worker-1  claims fetch:<big-domain>
00:51:35  worker-5  claims fetch:<big-domain>
01:07:09  worker-2  claims fetch:<big-domain>

Exactly 15 minutes apart. XPENDING showed times_delivered=6 for that message. All six interactive workers were now running the same multi-hour job in parallel, and nothing else in the queue moved until morning.

Why: idle time only resets on delivery

The idle time of a pending message in Redis Streams is measured from the last time it was delivered. It does not care that your handler is still running. Redis has no way to know your worker is alive; it only knows nobody has touched the message for 15 minutes.

So "reclaim anything idle for more than N minutes" really means "re-run any job that takes longer than N minutes". With N = 15 and a job that runs for hours, every worker in the pool eventually joins in.

The tempting fix is to raise the timeout to 2 hours, or 12. That only moves the cliff. The next job that runs longer than your new number does the same thing, and in the meantime, genuinely dead workers keep their messages for hours.

The fix: lease renewal with XCLAIM

XCLAIM with min_idle_time=0 on a message you already own resets its idle time to zero. With JUSTID it does not re-deliver the payload or bump the delivery counter. In effect, it is a heartbeat: "I still have this, I'm still working."

def heartbeat(r, stream, group, consumer, message_id):
    try:
        r.xclaim(stream, group, consumer,
                 min_idle_time=0, message_ids=[message_id], justid=True)
        return True
    except Exception:
        return False

We run it from a background thread for as long as the handler runs:

import threading

HEARTBEAT_INTERVAL = 120  # seconds, well under the 15-minute reclaim timeout

class TaskHeartbeat:
    def __init__(self, r, stream, group, consumer, message_id):
        self.args = (r, stream, group, consumer, message_id)
        self._stop = threading.Event()

    def __enter__(self):
        threading.Thread(target=self._run, daemon=True).start()
        return self

    def _run(self):
        while not self._stop.wait(HEARTBEAT_INTERVAL):
            if not heartbeat(*self.args):
                log.warning("lease renewal failed for %s", self.args[-1])

    def __exit__(self, *exc):
        self._stop.set()
        return False

with TaskHeartbeat(r, STREAM, GROUP, consumer, message_id):
    handle(task)
r.xack(STREAM, GROUP, message_id)

Two details matter:

• The thread is a daemon. If the worker process really dies, the heartbeat dies with it, the idle time starts growing again, and XAUTOCLAIM takes over exactly as before. We did not trade away crash recovery to get this.
• The interval is far below the timeout. 2 minutes against 15 means a few missed heartbeats (a Redis blip, a GC pause) still don't trigger a reclaim.

A second safety net

We also keep job state in a database table (queued / running / done, plus updated_at). When a worker reclaims a stale message, it now checks that row first. If the job is still running and was updated recently, the original worker is probably alive, so it skips the job without acking it. The message now belongs to the new consumer with a fresh idle time, so it won't come back for another 15 minutes, and when the original worker finishes, its XACK still works because acks don't check ownership.

With the heartbeat in place this branch almost never fires. It is there in case the heartbeat thread itself breaks.

A related trap: "self-healing" that amplifies itself

While cleaning up we found a second, similar bug. Another job type had a healer: if a job had been queued for more than 2 minutes, it re-enqueued it, bypassing the dedupe check, in case the message had been lost.

When the queue got deep, every job waited more than 2 minutes, so every job got re-enqueued, which made the queue deeper. We found 20,669 messages representing 103 real jobs, one of them re-enqueued 230 times in 10 hours.

The fix was to stop judging queued by the clock. Before re-enqueuing, check whether the stream still has unconsumed messages (XLEN > 0 or the group has pending entries). If it does, workers are alive and the job is just waiting its turn. Only when the queue has fully drained is a missing job actually lost. On any error, the check answers "don't re-enqueue". Doing one job too few is cheap; adding one more copy is what caused the problem.

Takeaways

• In Redis Streams, pending idle time means "time since delivery", not "time since the worker last showed signs of life".
• A reclaim timeout without lease renewal is really a maximum job duration. Anything longer gets run again by every worker in turn.
• Renew leases with XCLAIM min_idle_time=0 JUSTID from a daemon thread. Keep crash recovery, lose the duplicates.
• Never decide a queued job is lost just because it has waited a while. Deep queues are normal; check whether the queue is actually moving.
• "The scheduler says the queue is full" and "the queue is actually moving" are different facts. Ours reported a full queue for 14 hours because the stuck jobs never finished.

If you're curious what the workers actually fetch, it's Google search ad data. You can look up any domain on ppceye.com to see which advertisers send ads to it.