Celery’s defaults were chosen for tasks that take milliseconds. Every one of them is wrong for a task that spends a minute waiting on a model, and the failure they produce is not an error — it is a worker that looks idle while holding forty jobs it has not started.
What runs where
Four Kubernetes objects. Redis as a StatefulSet with a PersistentVolumeClaim and a headless Service. A Deployment of Celery workers. A Secret holding the broker URL and the provider API key. And, if you want results back through Celery rather than through your own table, a result backend — which can be the same Redis, though pointing both at one instance means a broker outage is also a results outage.
The producer, whatever it is, does not need to be in this diagram. It imports the same task module, calls .delay() or .apply_async(), and gets back an id. Everything in this page is about the consumer side.
# tasks.py
from celery import Celery
import os
app = Celery("inference", broker=os.environ["CELERY_BROKER_URL"])
app.conf.update(
task_acks_late=True,
task_reject_on_worker_lost=True,
worker_prefetch_multiplier=1,
task_soft_time_limit=180,
task_time_limit=210,
broker_transport_options={"visibility_timeout": 900},
)
@app.task(bind=True, max_retries=3)
def infer(self, job_id: str, prompt: str) -> None:
if already_done(job_id):
return
write_result(job_id, call_model(prompt))
Three Celery defaults that are wrong here
Prefetch. Celery documents worker_prefetch_multiplier as defaulting to 4, meaning each worker process reserves four messages at a time. With ten concurrent processes that is forty jobs claimed by one pod, most of them sitting in memory doing nothing. On a millisecond task this is a throughput win. On a ninety-second task it is a queue that appears drained to anything watching depth, which breaks autoscaling — see scaling Celery workers by queue length. Set it to 1.
Early acknowledgement. Celery documents task_acks_late as disabled by default, which means a task is acknowledged when it is handed to a worker, not when it finishes. If the pod is evicted mid-call the job is simply gone. Turning it on flips the failure mode from lost work to duplicated work, which for a model call is the trade you want only once the task is idempotent — so make it idempotent, then turn it on, and pair it with task_reject_on_worker_lost so a killed worker requeues rather than silently dropping.
No time limit. Celery documents both task_time_limit and task_soft_time_limit as having no limit by default. A hung HTTPS connection to a provider therefore occupies a worker slot indefinitely. The soft limit raises SoftTimeLimitExceeded inside your task so you can record the failure; the hard limit kills the process. Set the soft one below your provider client’s own read timeout so you get the informative exception rather than the abrupt one.
What Redis as a broker does and does not give you
Redis is not a message broker; Kombu implements one on top of it. The consequence you have to know about is the transport’s visibility_timeout, which Kombu documents as defaulting to 3600 seconds. A task still unacknowledged after that window is redelivered to another worker, exactly as SQS would.
An hour is long enough that most people never see it and short enough that a genuinely stuck job eventually doubles. The number that matters is not the model call, though — it is the whole time between delivery and ack, which with late acks includes any retry backoff Celery applies inside the task. If your task retries three times with exponential backoff, budget for the total, not for one attempt.
The 3600-second default is from the Kombu Redis transport reference, read August 2026. It is a library default rather than a service quota, so it can change with a dependency bump rather than with a vendor announcement. Kombu: Redis transport
Redis also gives you no durable acknowledgement semantics and no queue mirroring. What Kombu calls an acknowledgement is a delete from an in-flight structure, which is why the emulated visibility timeout exists at all, and why a Redis failover can lose or duplicate work in ways a broker with real publisher confirms will not. If durability matters more than operational simplicity, the alternative is a real broker; the equivalent setup is in RabbitMQ for queued model inference on Kubernetes.
The practical consequence for this workload is that you cannot treat the queue as the record of what has to happen. Write the pending row in your own database at intake and let the queue be a work signal rather than the source of truth. Then a lost message is a row that never left PENDING, which a sweeper can requeue, instead of a request that silently evaporated between a user pressing a button and nothing happening.
The worker Deployment, and shutting it down properly
The default Kubernetes grace period is 30 seconds. A Celery worker receiving SIGTERM performs a warm shutdown: it stops taking new tasks and waits for running ones. A ninety-second model call plus a thirty-second grace period means SIGKILL lands in the middle of it every time you deploy. Set terminationGracePeriodSeconds above your hard task time limit.
apiVersion: apps/v1
kind: Deployment
metadata:
name: celery-inference-worker
spec:
replicas: 2
selector:
matchLabels: { app: celery-inference-worker }
template:
metadata:
labels: { app: celery-inference-worker }
spec:
terminationGracePeriodSeconds: 240
containers:
- name: worker
image: registry.example.com/inference-worker:1.4.0
args: ["celery", "-A", "tasks", "worker",
"--loglevel=INFO", "--concurrency=4", "--prefetch-multiplier=1"]
envFrom:
- secretRef: { name: inference-secrets }
resources:
requests: { cpu: "200m", memory: "512Mi" }
limits: { memory: "1Gi" }
Note the absence of a CPU limit and the presence of a low CPU request. A worker that is waiting on a socket uses almost no CPU, and a CPU limit on this workload buys throttling you do not need. Memory is the resource worth capping, because a large response accumulated in memory is the realistic way this pod dies.
When the pod is OOM-killed instead
Everything above assumes SIGTERM. An out-of-memory kill is SIGKILL, and none of it applies: no warm shutdown, no grace period, no chance to acknowledge or requeue. The worker is gone between one instruction and the next, holding however many unacknowledged deliveries its prefetch had claimed.
What happens next depends entirely on the two settings from earlier. With task_acks_late disabled — the Celery default — those tasks were acknowledged when they were handed over, so they are simply gone, and the only trace is a pending row that never becomes done. With late acks on, they are unacknowledged, and Kombu’s Redis transport returns them after its visibility_timeout. That is the default 3600 seconds unless you changed it, so the honest description of the recovery time for an OOM-killed worker on a stock configuration is “up to an hour later”. On a user-facing pipeline that is indistinguishable from a lost request.
Which is the argument for setting visibility_timeout to a value derived from your work rather than leaving it: long enough to cover the worst-case task with its retries, short enough that a killed pod is a delay a person will tolerate. Somewhere in the region of a few minutes past your hard time limit, not an hour.
Reducing the frequency of the kill is the other half. A model response held as a string, then parsed, then re-serialised, is three copies of the same payload, and a worker with a concurrency of eight is doing that eight times at once. Two mitigations are cheap: keep concurrency proportionate to the memory limit rather than to the CPU request, and set --max-tasks-per-child so worker processes are recycled periodically, which bounds the accumulation of anything that leaks. Watch for pods whose last state shows OOMKilled with exit code 137 — a Deployment can look perfectly healthy while quietly killing and restarting a worker every few minutes, and the only symptom on the outside is that a small fraction of jobs take an hour.
Building it
- Deploy Redis as a StatefulSet with a PVC and a headless Service. Put the resulting
CELERY_BROKER_URLand your provider key in one Secret. - Write
tasks.pywith the four configuration overrides above, and make the task body start with a check that the job is not already done. - Build the image with the worker command as its args, not baked into an entrypoint script, so you can change concurrency without a rebuild.
- Apply the Deployment with
terminationGracePeriodSecondsabovetask_time_limit. - Enqueue one task, then
kubectl delete podthe worker mid-call and confirm the job is redelivered rather than lost — that is the test that late acks are actually on. - Watch queue depth while the worker runs. With the prefetch multiplier at 1 it should track real backlog, which is the precondition for autoscaling on it.
Top comments (0)