The spreadsheet arrived on a Tuesday in August, from a supply chain analyst at a regional pharmacy chain, and it was mostly gray. Ninety rows, one for each night of the previous quarter. Forty-one of them were highlighted in yellow.
She had spent six weeks reconciling those nights by hand, and she had narrowed the complaint down to one sentence: "It's not that the numbers are wrong. It's that they're late by exactly one day."
I wrote that replenishment sync two years ago, so I opened the job log expecting an error. There wasn't one. There never is.
Every night at 00:30 the platform started an automation that called an ERP export, recalculated reorder points and wrote them back. On the good nights it finished in nineteen minutes. On the yellow nights the export took longer, and something did not happen. No failure, no retry, no record of a decision. On forty-one nights out of ninety, work that was supposed to happen simply didn't, and the only witness was a human being with a calculator.
The interval is not a schedule. It is a hope.
Here is the sentence I had read years earlier and not understood, from the platform's own scheduler documentation: the precision of a scheduled task is one minute, so every execution carries zero to sixty seconds of delay — and at any given moment, only one execution of a scheduled task is running. Which means that if a task configured to run every five minutes takes six minutes, the execution planned for the fifth minute is cancelled.
Cancelled. Not queued, not deferred, not marked as skipped. It is removed from the future as though it had never been scheduled, and the calendar simply continues from the new position.
Read against a demo, that sounds reasonable — obviously you do not want two copies of the same job running at once. Read against a business, it is something else. The schedule we configured was "every night at 00:30". The schedule that actually ran was "every night at 00:30, unless the previous night was slow, in which case sometimes not". The interval between two runs was not a decision anyone made. It was a remainder: whatever was left over after the work took as long as it took.
Then subtract the other half of the promise. A "one minute" scheduler with up to sixty seconds of drift means "every five minutes" is really "every five to six minutes, chosen by a scheduler you do not control". Nobody is harmed by that on a good night. But the analyst's complaint was about nights, and her nights were being silently merged and silently dropped while the configuration screen kept displaying the tidy version of the schedule.
There is no state called "skipped". So there was nothing to show her.
The only queue you have is a number in a file.
Six weeks later the same shape appeared as a support ticket from a mid-size manufacturer in Dongguan. Their statement and reminder runs had stopped entirely for nine days during their busiest quarter. No failure notifications, no red rows. The jobs had not errored. They had not run.
The documentation's troubleshooting page for "scheduled tasks stopped running" is unusually candid, and it lists the causes: the scheduling thread pool is too small to pick up the work, a task's next-run time fails to update, archived or disabled state filters tasks out, or a misfire path interrupts subsequent scheduling. The remedy is to raise the scheduling pool size in a server configuration file, restart, and then hunt for the stalled work with a SQL query — the one that finds scheduled rows whose status is still Pending and whose next-run time is already in the past.
That query is the most honest line in the entire manual, because it reveals what the platform's model of a backlog actually is. It is a database table.
There is no queue view. No depth gauge. No "twelve waiting, three running, one given up on". The word the scheduler uses for a job that has not run yet is the same word it uses for a job that should have run twenty minutes ago. Pending is doing two very different jobs, and only one of them is visible in the interface.
So the throughput ceiling of that customer's nightly operations was a default number in a file they had never opened, discoverable only by reading a troubleshooting page about tasks that do not run, and raisable only by someone with shell access. We sell a platform where a business analyst can build an entire application without ever seeing a terminal. And then the arithmetic of how much work that application can absorb per minute lives in YAML.
Every misfire policy is a bet placed before you know the odds.
The platform does offer one piece of genuine queue thinking: a misfire policy, for tasks that miss their window. There are two choices. Run everything that was missed, as soon as possible. Or discard everything that was missed and wait for the next occurrence.
Both are defensible. Neither is a strategy.
Catch-up means that after a maintenance window or a long weekend you get every overdue execution arriving at once, competing for the same small pool — precisely the condition that caused the misses. Discard-all means the platform quietly forgets work a business already promised someone. And the resulting hole is never labeled a hole; it looks exactly like a night on which there was simply nothing to do.
Worse, the choice has to be made per task, at design time, before you know which failure mode will hurt. You cannot know whether the missed run was a redundant refresh or the only transaction in the month that mattered. That is not a setting. That is a guess, written into the metadata as though it were a fact.
Notice also what the scheduler forbids: a scheduled execution may not contain interactive steps. That rule is correct — nobody should be waiting for a human at 00:30 — but it means every judgement call inside a batch job becomes either a guess or an abort. Both arrive as data. Neither arrives as an alert.
The retry mechanism is a person.
So what did both customers do? They rebuilt the missing queue out of the only materials available: their own calendars and their own attention. The pharmacy analyst wrote a manual rebuild procedure for every yellow night. The manufacturer's IT team ran the missed statements by hand each morning until someone finally asked why they were doing it.
That is the real cost — not the lost execution, but the fact that the platform has no way to represent unfinished work, so unfinished work is stored in human memory and in the habit of checking. A job is either running or it is nothing. Between those two states lies a whole territory — waiting, deferred, failed, retried, abandoned — and every square inch of it has been outsourced to the person who depends on the outcome.
It is also why batch jobs on platforms like ours get written defensively, with "have I already processed this window" checks buried in the script. Idempotency should be something the runtime gives you; here it is the tax you pay for the scheduler's silence.
The uncomfortable conclusion
We call it scheduling because a calendar is the metaphor business people already understand, and the metaphor is the entire problem. A calendar answers exactly one question — when does this happen next — and there is a place on a screen where you can draw that answer. What an operations team actually needs is a queue: work items that persist, that carry a state you can see, that can be claimed, retried, paused, given up on, and counted.
Every queue in the world can answer "how much is waiting?". Ours can only answer "when will it next run?", which is the wrong question on the morning the numbers are wrong. The pharmacy did not need a better cron expression from me. She needed to open a screen and see that forty-one nights had been cancelled.
The uncomfortable part is that we did the arithmetic and chose the calendar anyway. A calendar is a row of times. A queue is a list of numbers that grow — and numbers that grow invite questions we would have had to answer. How deep. Since when. For whom. Which ones gave up.
So we shipped the clock and left the bookkeeping to whoever reconciles on Tuesday.
The calendar was the demo. The queue was the product.
We shipped the demo.
Top comments (0)