The exec inside a pipeline detail is easy to miss, and your test that checks the real script rather than trusting the script is a strong idea. For the long running generations that still die on deploy, would you rather drain the queue before a deploy or make those jobs resumable, given that they are paid model calls?
Head of Backend Engineering at GoodBarber. My team owns the APIs, services and infra that turn no-code projects into real published apps. I write about no-code at scale — and where AI fits in.
Thanks. Today it's neither: the deploy kills the generation, the user runs it again, and the calls the first run made are paid for nothing.
For deploys, I'd rather drain: a call that finishes is the only call we pay for once. Since the fix, the pool already stops taking jobs on SIGTERM and waits for the one it's running. The catch is our deploy script, which takes everything down before it brings anything up: a grace period longer than our slowest generation would keep the whole host down for up to eight minutes. Resuming I'd keep for the deaths we don't schedule, and a model call cut halfway leaves nothing to resume from.
Draining makes sense when a finished call is paid for only once, since a killed generation wastes the first attempt entirely. How do you decide the cutoff when a slow generation is close to finishing, given that a longer grace period could keep the whole host down for up to eight minutes?
Head of Backend Engineering at GoodBarber. My team owns the APIs, services and infra that turn no-code projects into real published apps. I write about no-code at scale — and where AI fits in.
Today nothing decides it. The cutoff is a flat 15 seconds on every worker pool, and Docker doesn't know how far along a job is. A generation a minute from done dies like one that just started, and it's the most expensive one to lose: by then it has paid for almost everything it needed.
So I wouldn't try to pick a better cutoff. I'd change the order of the deploy instead: start the new workers first, then stop the old ones with a grace as long as the slowest generation. They finish what they're running while the new ones take the next jobs, and nothing else waits for them.
The catch is that for a few minutes both versions share the queue, so each has to read the other's jobs. Our pickle to JSON switch couldn't, which is why that deploy drained the queues first.
Starting the new workers first makes sense, since it avoids killing a nearly finished generation just to meet a fixed cutoff. During the overlap, how do you keep the two versions from picking up jobs they cannot handle, or would you tag jobs by version until the old workers stop?
Head of Backend Engineering at GoodBarber. My team owns the APIs, services and infra that turn no-code projects into real published apps. I write about no-code at scale — and where AI fits in.
I wouldn't tag them. If the old workers get SIGTERM before the new code enqueues anything, they never see a new job: from SIGTERM on, they take nothing and only finish the one they hold. Jobs would only cross one way, old ones read by new workers, so the rule becomes that each release reads the previous one's jobs. A change that can't, like our pickle to JSON switch, gets a drained deploy.
Custom software & AI development company. We build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results. lycore.com
The cat /proc/1/comm check is a great habit, we've been bitten by the same thing with shell entrypoints that wrap the real process. One thing we'd look at after this fix is the relationship between stop_grace_period and your longest job. With RQ's warm shutdown the worker finishes its current job, so a 15 second grace still kills anything that runs longer. Do you requeue interrupted jobs on startup, or rely on the job registry to surface them?
Head of Backend Engineering at GoodBarber. My team owns the APIs, services and infra that turn no-code projects into real published apps. I write about no-code at scale — and where AI fits in.
Thanks, you're pointing at exactly the part we haven't fixed.
To your question: today nothing requeues them. Since the fix, RQ's own cleanup fails an abandoned job the way it should and runs its failure callback, so the started registry no longer fills up with jobs nobody runs. That keeps RQ's books straight, but it isn't what the user sees. Each running generation holds a Redis key with a 90-second expiry, and the worker keeps renewing it. When the page checks on a generation, if that key is gone and the generation is more than 180 seconds old, the generation is marked as failed. So a generation killed by a deploy shows up as failed within about three minutes while the page is open, and the user runs it again.
How do you handle yours: do you requeue on startup? Our jobs are paid model calls, so the part I'd want to be sure of is never running one twice.
Head of Backend Engineering at GoodBarber. My team owns the APIs, services and infra that turn no-code projects into real published apps. I write about no-code at scale — and where AI fits in.
Thanks! I'll be honest, this was the first time I'd run into this problem. I knew exec was there to hand the signals to the real process, and I had never seen it quietly do nothing inside a pipeline. Working out why turned out to be the most interesting part of the whole fix.
One follow-up if you run the check after a fix like ours: with init: true, it doesn't print python anymore, it prints docker-init, the tini that Docker bundles. The question then moves one level down: is the child of PID 1 your process, or a shell? docker top <container> shows that from the host, even in an image without ps.
Good point on docker-init. If /proc/1/comm prints that instead of python, the next check is what sits under it, and docker top shows that from the host.
We're a place where coders share, stay up-to-date and grow their careers.
The exec inside a pipeline detail is easy to miss, and your test that checks the real script rather than trusting the script is a strong idea. For the long running generations that still die on deploy, would you rather drain the queue before a deploy or make those jobs resumable, given that they are paid model calls?
Thanks. Today it's neither: the deploy kills the generation, the user runs it again, and the calls the first run made are paid for nothing.
For deploys, I'd rather drain: a call that finishes is the only call we pay for once. Since the fix, the pool already stops taking jobs on SIGTERM and waits for the one it's running. The catch is our deploy script, which takes everything down before it brings anything up: a grace period longer than our slowest generation would keep the whole host down for up to eight minutes. Resuming I'd keep for the deaths we don't schedule, and a model call cut halfway leaves nothing to resume from.
Draining makes sense when a finished call is paid for only once, since a killed generation wastes the first attempt entirely. How do you decide the cutoff when a slow generation is close to finishing, given that a longer grace period could keep the whole host down for up to eight minutes?
Today nothing decides it. The cutoff is a flat 15 seconds on every worker pool, and Docker doesn't know how far along a job is. A generation a minute from done dies like one that just started, and it's the most expensive one to lose: by then it has paid for almost everything it needed.
So I wouldn't try to pick a better cutoff. I'd change the order of the deploy instead: start the new workers first, then stop the old ones with a grace as long as the slowest generation. They finish what they're running while the new ones take the next jobs, and nothing else waits for them.
The catch is that for a few minutes both versions share the queue, so each has to read the other's jobs. Our pickle to JSON switch couldn't, which is why that deploy drained the queues first.
Starting the new workers first makes sense, since it avoids killing a nearly finished generation just to meet a fixed cutoff. During the overlap, how do you keep the two versions from picking up jobs they cannot handle, or would you tag jobs by version until the old workers stop?
I wouldn't tag them. If the old workers get SIGTERM before the new code enqueues anything, they never see a new job: from SIGTERM on, they take nothing and only finish the one they hold. Jobs would only cross one way, old ones read by new workers, so the rule becomes that each release reads the previous one's jobs. A change that can't, like our pickle to JSON switch, gets a drained deploy.
The
cat /proc/1/commcheck is a great habit, we've been bitten by the same thing with shell entrypoints that wrap the real process. One thing we'd look at after this fix is the relationship between stop_grace_period and your longest job. With RQ's warm shutdown the worker finishes its current job, so a 15 second grace still kills anything that runs longer. Do you requeue interrupted jobs on startup, or rely on the job registry to surface them?Thanks, you're pointing at exactly the part we haven't fixed.
To your question: today nothing requeues them. Since the fix, RQ's own cleanup fails an abandoned job the way it should and runs its failure callback, so the started registry no longer fills up with jobs nobody runs. That keeps RQ's books straight, but it isn't what the user sees. Each running generation holds a Redis key with a 90-second expiry, and the worker keeps renewing it. When the page checks on a generation, if that key is gone and the generation is more than 180 seconds old, the generation is marked as failed. So a generation killed by a deploy shows up as failed within about three minutes while the page is open, and the user runs it again.
How do you handle yours: do you requeue on startup? Our jobs are paid model calls, so the part I'd want to be sure of is never running one twice.
The /proc/1/comm check is the best part. If it prints bash, the rest of the story writes itself.
Thanks! I'll be honest, this was the first time I'd run into this problem. I knew
execwas there to hand the signals to the real process, and I had never seen it quietly do nothing inside a pipeline. Working out why turned out to be the most interesting part of the whole fix.One follow-up if you run the check after a fix like ours: with
init: true, it doesn't printpythonanymore, it printsdocker-init, the tini that Docker bundles. The question then moves one level down: is the child of PID 1 your process, or a shell?docker top <container>shows that from the host, even in an image withoutps.Good point on docker-init. If /proc/1/comm prints that instead of python, the next check is what sits under it, and docker top shows that from the host.