Two workers, two runtimes, one broker
I was adding video rendering to a pipeline that had never rendered video. Rendering needs Node and a headless Chromium; nothing else in the system needs either, and neither belongs on a host that also holds the production database. So the renderer became its own container image with its own Celery worker on its own queue, which is the ordinary way to run heterogeneous work under one broker.
The backend sends a job by task name. The render worker picks it up, produces a file on a shared volume, and returns the path. Two images, two runtimes, one Redis between them. Clean.
Ready, healthy, deaf
I deployed it, triggered an episode, and watched the logs. The render worker said everything you want to see:
[INFO/MainProcess] mingle: searching for neighbors
[INFO/MainProcess] mingle: all alone
[INFO/MainProcess] render@8d1fd465fd70 ready.
And then nothing. No task received. Meanwhile the sending task sat blocked, waiting on a result that was never going to arrive, and the container health check kept returning success on a sixty-second interval.
The cause was two characters. My worker had a sensible-looking default:
BROKER = os.environ.get("REDIS_URL", "redis://redis:6379/0")
The rest of the system publishes on database 1 and keeps results on database 2. The worker was listening on database 0, an empty room in the same building. The episode was sitting in a queue it never looked at.
The health check agreed with the worker
A misconfigured worker is a five-minute bug. What made this worth writing down is that I had a health check specifically designed to catch a dead worker, it ran every minute, and it passed the entire time.
celery -A worker:app inspect ping -d render@$HOSTNAME --timeout 10
That command is a good health check. It does not just look at the process table: it round-trips a message through the broker and back, so it fails if the worker is wedged, if the event loop is blocked, or if the broker is unreachable. I had chosen it deliberately over a naive process check.
But inspect ping round-trips through the worker's own broker connection. It read the same environment variable, connected to the same database 0, sent a ping into that empty room, and the worker answered it. Both ends of the check were on the wrong side of the misconfiguration, so the check confirmed, correctly and uselessly, that a worker nobody could reach was alive and well.
This is the same class of error as a backup timer that reports "active" while the backup has not completed in three days, or a video file that every metadata tool calls valid because they all read the same intact header. The instrument and the fault share an assumption.
Delete the default
The repair was not a better health check. It was removing the thing that made a wrong configuration possible:
BROKER = os.environ["CELERY_BROKER_URL"]
RESULT = os.environ["CELERY_RESULT_BACKEND"]
Both required, no fallback, both passed through from the same environment file the backend reads rather than written out a second time in the compose file. Missing configuration now stops the container instead of producing a healthy-looking one, and there is no second copy of a broker URL to drift.
That is the trade a default hides. A fallback value turns a loud startup failure into a quiet runtime one, and it does it precisely in the situation where you most need the noise: the first deploy, when nothing is known to work yet.
Check the channel, not your end of it
Three questions I now ask of any health check before trusting it:
- Can this pass while the thing it monitors is completely broken? If yes, it is checking a process, not a result. Rewrite it.
- Does it share configuration with its subject? A check that reads the same environment variable inherits the same mistake. The useful check comes from the other end of the wire, or from the artifact that should exist.
- What would this look like if the answer were "nobody is listening"? Usually: exactly like success. That is the whole problem.
For queues specifically, the check that would have caught this is boring and effective: the sender enqueues a trivial job and asserts a result comes back within N seconds. It crosses the same boundary the real work crosses, so it cannot be fooled by an assumption only one side holds. It costs one queue message a minute.
The worker was fixed in five minutes. The health check took longer, because the first version was not wrong, it was simply looking in the wrong direction, and those are harder to notice than a check that is missing.
Originally published at neuragrowth.co. I run a one-person digital-products studio and write up what breaks in production.
If you write CLAUDE.md files, I keep a set of working templates here: neuragrowth.co/free/claude-md-templates.
Top comments (0)