Celery loses 8% of your tasks by default. Here's the reliability layer I built to fix that.

Kolade Fajimi — Tue, 02 Jun 2026 00:47:11 +0000

Celery is one of the most widely deployed task queue systems in Python. It is also, by default, a system that silently loses approximately 8% of your tasks the moment a worker crashes.

This is not a bug. It is the designed default behaviour. And most teams shipping Celery in production either do not know about it or have accepted it as a cost of doing business.

I built Relier because I was not willing to accept it.

How Celery loses tasks

When a Celery worker picks up a task from the broker, it sends an acknowledgement (ACK) immediately, before the task runs. From the broker's perspective, the task is done. The worker owns it now.

If the worker is killed (OOM, SIGKILL, kernel memory pressure, deploy) while the task is executing, the broker has already marked that task as delivered. The task is gone. No retry, no trace, no record it was ever picked up.

This is task_acks_late=False, Celery's default.

At 10M tasks per day, 8% loss is 800,000 silently dropped jobs. Every. Single. Day.

Why flipping `task_acks_late=True` is not enough

The standard advice for this problem is task_acks_late=True. It helps. In our benchmarks, it takes delivery from 92.0% to 96.0%, recovering about half the lost tasks.

But it does not solve the problem, for a specific reason.

When a worker dies with task_acks_late=True, the broker keeps the unacknowledged message in an unacked set. Redelivery is gated by visibility_timeout, the time the broker waits before assuming the worker is gone and requeuing the message. On the Redis broker, this defaults to approximately one hour.

So a task killed at 2:00 PM sits waiting for redelivery until 3:00 PM. In most production systems, the SLA for that task is measured in seconds or minutes, not hours.

The deeper problem: you have traded silent loss for hour-long redelivery latency, without knowing which tasks are stuck in that limbo.

Our bench ran 500 tasks through 5 SIGKILL cycles:

	Delivery rate
Vanilla Celery (default)	92.0% (460/500)
Vanilla + `task_acks_late=True`	96.0% (480/500)
Relier	100% (500/500)

The Phoenix Pattern

Relier implements what I call the Phoenix Pattern. The design is straightforward in principle and non-trivial to get right in practice.

Every @rl_task registers a heartbeat in Redis when it starts executing, a key with a configurable TTL (default 10 seconds). The task refreshes that heartbeat on a background loop while it runs. Every worker embeds a resurrection scanner that watches for expired heartbeats every few seconds, so the surviving workers recover a dead worker's tasks on their own, distributed locks keep concurrent scanners from replaying the same task twice. (You can also run a standalone rl run-resurrector process as belt-and-suspenders for the case where every worker dies at once.)

When a worker dies mid-task, its heartbeat stops refreshing. After one TTL window, the resurrector detects the expired heartbeat and atomically re-queues the orphaned task onto a special re-queue queue. A healthy worker picks it up. The original task arguments are preserved exactly.

In our benchmarks, OOM recovery averaged 7.1 seconds with a p99 of 8.9 seconds not 35 seconds, not an hour. Seconds.

Worker dies at t=0
Heartbeat expires at t=10s (heartbeat_ttl)
Resurrector detects at t=12s (next scan)
Task re-queued at t=12s (atomic)
Healthy worker picks up at t=12–14s
Task completes

This is why Relier achieves 100% delivery: it does not rely on the broker's visibility timeout. It has its own independent detection mechanism with a TTL you control.

The hard part: fence tokens and zombie workers

The description above makes Phoenix sound simple. The part that took the most work to get right is the zombie worker problem.

Consider this scenario:

Worker A picks up Task X. Heartbeat registered.
Worker A has a long GC pause. Its heartbeat expires.
The resurrector detects the expired heartbeat and re-queues Task X.
Worker B picks up Task X and completes it. Result committed to Redis.
Worker A wakes up from its GC pause and tries to commit its result.

Without any protection, step 5 causes silent data corruption. Worker A commits a stale result, overwriting Worker B's correct result. The task has now effectively executed twice, with the wrong result stored.

Relier prevents this with fence tokens. When Phoenix re-queues a task, it generates a new fence token, a monotonically increasing integer associated with the task's execution slot. The completion protocol is an atomic Lua script: "commit this result only if the current fence token matches the token this worker was given when it claimed the task."

Worker A was given fence token v1. After resurrection, the slot is now at v2. When Worker A tries to commit, the Lua script sees the mismatch and rejects the write. No data corruption. No duplicate result.

This is the correctness guarantee that makes "exactly-once execution" mean something.

Everything else Relier adds

Beyond Phoenix, a production-grade task system needs several more things. Relier ships them as part of the same @rl_task decorator:

Idempotency. @rl_task(idempotent=True) adds an atomic Redis Lua check before task execution. If the same task has already been submitted for the same logical key (which you can set explicitly or let Relier derive from the arguments), the second submission returns immediately without spawning work. In our benchmark: 50 submissions of the same task, 1 execution. Vanilla Celery: 50 executions.

Two-tier timeouts. soft_timeout=8, hard_timeout=10 gives you a cleanup hook that fires at 8 seconds (save state, close connections, emit structured logs) and a hard cancellation at 10 seconds via asyncio.CancelledError. Zombie tasks that would block a worker forever are quarantined instead.

Graceful shutdown. On SIGTERM, the worker drains in-flight tasks rather than dropping them. Tasks that cannot complete before shutdown hands them off to Phoenix on the re-queue queue. In our benchmark: 3 cycles of 20 tasks each, Relier 100% survival, vanilla Celery 0%.

Dead Letter Queue. Tasks that exhaust their max_resurrections allowance are quarantined to the DLQ with their full payload, stack trace, and complete resurrection history. The rl dlq inspect CLI shows everything. rl dlq release <id> re-dispatches a specific failed task. Nothing disappears silently.

Admission control. An atomic Lua fixed-window rate limiter on every apush() call. If the cluster is saturated, you get an AdmissionRejectedError with a Retry-After header, not a flooded queue and a cascade failure. In our benchmark: p99 0.559ms, well under the 1ms claim.

Rolling deploy protection. Every payload is wrapped in a versioned envelope with a SHA-256 checksum. Register a migration function, bump CURRENT_VERSION, and v2 workers silently upgrade v1 payloads mid-deploy. Old and new workers can run simultaneously without payload schema mismatches. Checksums catch broker-side corruption before your code ever runs.

Benchmarks

All numbers below from the built-in bench suite running against live Redis on Linux (Docker, python:3.11-slim, prefork=4 workers), synthetic 0.5s tasks:

Metric	Relier v0.1.6	Vanilla	Vanilla + acks_late
Task delivery (500 tasks, 5 kills)	100%	92.0%	96.0%
OOM recovery avg / p99	7.1s / 8.9s	∞ lost	∞ (visibility_timeout)
Idempotent recovery (delayed restart)	re-ran 4.8s	∞ lost	∞ (visibility_timeout)
Graceful shutdown (3 cycles)	100%	0%	0%
Duplicate prevention (50 submissions)	1/50 ran	50/50 ran	50/50 ran
Admission control p99	0.559ms	n/a	n/a
Dispatch overhead (net)	+1.87ms	baseline	n/a

The 1.87ms dispatch overhead covers the admission Lua script + SHA-256 envelope wrap + heartbeat registration. On any task doing real work (a database query, an HTTP call, an AI inference), this cost is invisible.

Getting started

pip install relier

from relier import rl_task

@rl_task(
    queue="default",
    idempotent=True,
    soft_timeout=25,
    hard_timeout=30,
)
async def send_invoice(invoice_id: str) -> dict:
    await charge_card(invoice_id)
    await send_email(invoice_id)
    return {"invoice_id": invoice_id}

# From FastAPI:
await send_invoice.apush(invoice_id)

# From Flask / Django:
send_invoice.push(invoice_id)

Three processes to run:

celery -A relier.tasks.app worker -l info -Q high_priority,default,re-queue --include=tasks
rl run-resurrector
uvicorn main:app

Or the full stack (Redis, workers, resurrector, Prometheus, Grafana) with:

docker compose -f docker-compose.bench.yml up --build

Requirements: Python 3.11+, Redis 7+ with AOF persistence and maxmemory-policy noeviction. Relier preflight-checks both on startup and refuses to run if either is wrong.

What I learned building this

The failure modes that are hardest to reason about are not the obvious ones. A worker dying is obvious, you see the process disappear. A GC pause that makes a healthy process look dead to an external observer, then have it wake up and try to write stale state, that is the case that breaks naive implementations.

Rolling deploys without schema versioning are a silent data loss vector that almost nobody talks about. The checksum + migration system exists because I watched a TypeError on a renamed argument silently DLQ a week's worth of invoice tasks with no alert.

Fence tokens are not a novel idea. The pattern comes from Martin Kleppmann's writing on distributed locking. But seeing the exact failure mode in a test, instrumenting it, and then watching the Lua script atomically reject the zombie commit, that was the moment Relier went from "probably correct" to "verifiably correct."

The chaos suite in the repo exists for this reason. Five scenarios: worker-kill, network-partition, load-spike, task-corrupt, slow-task. Run them against your own cluster, your own Redis, your own task code. You should not have to trust my benchmarks. Prove it yourself.

GitHub: github.com/getrelier/relier

Docs: getrelier.github.io/relier

Install: pip install relier

DEV Community: Kolade Fajimi