Scaling Celery Workers by Queue Length on Kubernetes

#ai #machinelearning #tutorial #llm

A Celery worker calling a model API is almost entirely idle. It holds a socket open and waits. Scale it on CPU and it will never scale, because the resource it is short of is not one the horizontal pod autoscaler can see.

Why CPU is the wrong signal

The standard HPA scales on resource utilisation. For a worker whose job is waiting on a network call, CPU utilisation sits near zero whether the queue holds three messages or thirty thousand. Memory is no better. The only number that reflects demand is the depth of the queue, and that number lives in the broker, not in the Kubernetes metrics pipeline.

KEDA bridges that. It reads an external source on a schedule and drives a standard HPA underneath, which means you keep normal HPA behaviour and gain the ability to scale to zero — genuinely useful here, because an idle worker pool that exists only to hold API sockets is pure waste.

The ScaledObject and its defaults

KEDA’s ScaledObject is on keda.sh/v1alpha1. Its documented defaults are a pollingInterval of 30 seconds, a cooldownPeriod of 300 seconds, a minReplicaCount of 0 and a maxReplicaCount of 100.

Two of those deserve thought on this workload. Polling every 30 seconds means up to half a minute of latency between a burst arriving and KEDA noticing, which is fine when your tasks take a minute and poor when they take two seconds. And the 300-second cooldown is the period after the last trigger activation before scaling back to zero — it is what stops a pod being killed the moment the queue empties, and it interacts with your task duration, because a worker terminated mid-call is a lost or duplicated model call.

KEDA field names and defaults here are from the KEDA 2.17 documentation, read August 2026. Scaler metadata fields have been renamed between minor versions before; check the docs for the version you actually run. KEDA: ScaledObject specification

The RabbitMQ trigger

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: celery-inference-worker
spec:
  scaleTargetRef:
    name: celery-inference-worker
  pollingInterval: 15
  cooldownPeriod: 300
  minReplicaCount: 0
  maxReplicaCount: 30
  triggers:
    - type: rabbitmq
      metadata:
        protocol: http
        queueName: celery
        mode: QueueLength
        value: "5"
        activationValue: "1"
        excludeUnacknowledged: "true"
      authenticationRef:
        name: keda-trigger-auth-rabbitmq-conn

The trigger type is rabbitmq. mode takes QueueLength or MessageRate, and value is the target per replica — with a value of 5 and 50 messages waiting, KEDA asks for 10 replicas. activationValue is the separate threshold that decides whether to leave zero at all, which is why it is not the same field as value.

Use protocol: http rather than amqp when you want MessageRate or queue-name regexes, since those go through the management API. That also means the management plugin has to be reachable, which it is on the operator-created Service described in RabbitMQ for queued model inference on Kubernetes.

The Redis trigger, and what it counts

With Redis as the Celery broker, the relevant KEDA scaler is redis, using the Redis Lists variant: Celery stores queued tasks in a Redis list named after the queue, which by default is celery. The metadata fields are address, listName, listLength as the average target value, activationListLength, databaseIndex (default 0) and enableTLS.

The important difference from RabbitMQ is what the number means. The Redis list length counts tasks that have not yet been handed to a worker. It has no concept of unacknowledged deliveries, because Redis has no acknowledgements — Kombu emulates them. So a task that has been delivered but not finished has already left the list, and the scaler cannot see it. Your metric is strictly “work not yet started”, which is usually what you want for scaling but is not the same as backlog.

If you route tasks to several named queues, remember each is a separate list and needs its own trigger. A ScaledObject may carry several triggers, and KEDA takes the maximum of the replica counts they request.

The prefetch trap, and flapping

Here is the failure that wastes an afternoon. Celery documents worker_prefetch_multiplier as defaulting to 4. With a concurrency of 8, each worker reserves 32 messages the moment they exist. Two workers drain a 64-message backlog out of the visible queue instantly and then spend an hour working through it. The scaler sees an empty queue, scales to zero or to the minimum, and the backlog is invisible for the entire time it takes to clear.

The fix has two halves. Set worker_prefetch_multiplier to 1 so workers claim what they are working on and no more. And on RabbitMQ, set excludeUnacknowledged deliberately: with it false, the scaler counts messages that are delivered-but-unacked as backlog, which overstates demand and can drive the replica count up while the existing workers are perfectly busy. Which setting is right depends on whether your prefetch is 1; with prefetch at 1 and long tasks, counting only ready messages is the honest signal.

Flapping is the other thing to design against, and it is worse here than usual because scaling down kills a worker holding a paid-for in-flight call. Three settings mitigate it: a terminationGracePeriodSeconds on the Deployment longer than your hard task time limit, a cooldownPeriod that is not shorter than a typical task, and an HPA scale-down stabilisation window configured through the ScaledObject’s advanced HPA behaviour section. Without those, a queue that oscillates around the target produces a worker pool that oscillates with it, and each oscillation costs a generation.

Building it

Install KEDA into its own namespace and confirm the scaledobjects.keda.sh CRD is present.
Set worker_prefetch_multiplier to 1 and task_acks_late to true in the Celery app, and redeploy the workers, before adding any scaler.
Create a TriggerAuthentication referencing the broker credentials Secret, rather than putting a connection string in the ScaledObject metadata.
Apply the ScaledObject with a conservative value and a maxReplicaCount you have checked against your provider’s concurrency allowance — the scaler will otherwise happily scale you into a wall of 429s.
Set terminationGracePeriodSeconds above the hard task time limit on the worker Deployment.
Enqueue a known backlog and watch replica count against queue depth for one full cycle up and back to zero. If depth drops to zero instantly while work continues, your prefetch change did not take.