Had an interesting realization about job queues this week that I figured I would share since it came up in a code review.
The setup
You are running a scheduled job. It is configured to run every N hours. Most of the time it finishes in time, but sometimes it does not — maybe it hits a rate limit, maybe the data volume is higher than usual, whatever.
What happens when your job is still running when the scheduler tries to start it again?
The semantics you probably have not thought about
Turns out, job queue implementations typically give you a few options when this happens:
- Prefer New: Cancel the running job, start the new one
- Prefer Old: Let the running job finish, skip the new trigger
- Wait: Queue the new trigger, run it after the current one finishes
- Parallel: Run both concurrently (if you have concurrency > 1)
Most people, including me apparently, assume Prefer New is the sensible default. Newer runs should use fresher data, right?
Where that assumption breaks down
Here is the scenario that made me rethink this:
You are running a job that takes 7 hours on weekends but only 2 hours on weekdays (less data to process). You set the interval to 3 hours thinking the 2-hour job will finish well before the next trigger.
On the weekend, your 7-hour job starts. At the 3-hour mark, a new trigger fires. With Prefer New, you cancel the running job and start fresh. It gets canceled again at the 6-hour mark. And again. You will run the job 16 times over a weekend and none of them will ever finish.
With Prefer Old, the running job just continues. You might cancel a few queued triggers, but your job actually completes.
The practical takeaway
When you are configuring scheduled jobs, think about what should happen when the job outlives its interval. Prefer Old feels wrong intuitively, but in situations where:
- Your job takes longer than expected due to external factors
- Stale results are better than no results
- You want to avoid wasted compute on canceled runs
...it might actually be the right choice.
Check what semantics your job queue exposes. Celery has task_acks_late and task_reject_on_worker_lost. Sidekiq has lock options. Bull, Kubernetes CronJobs — they all handle this differently.
The defaults might not match what you actually need.
Top comments (1)
The choice gets easier if you ask what the run writes rather than how long it takes. Cancellation is only safe when the output is atomic. If the job writes incrementally, Prefer New does not give you a clean restart, it gives you a partially written result with no error attached to it. Row count is nonzero, nothing raised, monitoring stays green, and downstream reads it as a complete refresh. Stale output is at least internally consistent. Half output is the one that fools people. So the rule I ended up with is that a run may only be cancelled if it publishes through a staging table or a fresh partition that gets promoted at the very end. Everything else gets Prefer Old, regardless of how appealing the fresher data sounds.
The second thing your weekend example hints at but does not say out loud is that overlap is self amplifying whenever the runs share an external budget. Two copies of the same collector against the same host do not each get their own rate limit. They split it. So the second run does not merely double the compute, it slows the first one down, which pushes it further past the interval, which invites a third. Across roughly 2190 production runs on 32 collectors, with one target accounting for 962 of them, every runaway schedule I have had to unpick had that shape. The 7 hour weekend job is often not 7 hours because there is more data. It is 7 hours because the previous trigger is still in flight, competing with it for the same 429 budget.
The part I have not solved. Log the skip. Prefer Old skips quietly, and from downstream a skipped trigger and a scheduler that never fired look identical, both are just absent data. And I still cannot choose Prefer Old versus Prefer New automatically. It depends on how much staleness the consumer can absorb, and I have never managed to encode that anywhere, so a human still sets it per collector.