Reordering an overflowing job queue does not shrink it. Splitting it does.

Background

I run about 20 small sites and update them automatically. A batch runs four times a day, decides which sites have work due, and then does only that work.

The decision itself lives in one script, whats-due.js. Its only inputs are tasks.json (task definitions), state.json (run state), and today's date. No network calls, no randomness, so the same inputs always produce the same answer. That determinism is what makes the whole thing safe to re-run after a crash.

But a correct decision and a workable decision are two different things. This post is about the second one: what to do when the number of due tasks keeps exceeding what a single run can finish.

How it works

The shape of the decision is this.

tasks.json + state.json + today(JST) + slot
        ──> whats-due.js ──>
{ anyDue, priorityOrder, staleWarnings, configWarnings, tasks: [ { id, site, due, reason, ... } ] }

Every task carries a quota.kind, and cadenceOf() folds those kinds into four cadences.

• per-run (research-append / ranked-append / new-critical-cves) — runs every batch, because freshness is the point
• per-day (min-articles-per-day and friends) — a daily quota; once met, the task reports done-today
• periodic (periodic-days) — every N days
• on-demand — never due in the daily run; the definition exists only so other tooling can look up the repo path and deploy command

state.json holds very little per site: lastRun, lastRunArticlesAdded, knownSlugs. Quota checks are just a comparison between lastRun and today.

That part was fine. The trouble started downstream.

Implementation

What was actually happening

On the morning of 2026-08-15, the run had 19 due tasks and finished 12 of them. Seven spilled over. The next morning, the same thing.

My first fix was a priorityOrder that sorted by starvation, longest-neglected first. It did nothing. Sorting changes which tasks get dropped; it does not change how many. With a per-run capacity of roughly five or six items and 19 items due, no ordering saves you. The problem was the count, not the order.

Fix 1: split the count into slots

I divided the day into four slots (morning 10:00, noon 14:00, evening 18:00, night 22:00) and made each task declare its slots in tasks.json. whats-due.js runs the normal quota logic first, then drops anything not assigned to the current slot:

for (const r of result) {
  const t = tasksDoc.tasks.find((x) => x.id === r.id);
  r.slots = (t && Array.isArray(t.slots) && t.slots.length ? t.slots : null) || SLOT_NAMES;
  r.slot = slot;
  if (r.due && !taskRunsInSlot(t, slot)) {
    r.due = false;
    r.slotDeferredReason = r.reason;
    r.reason = "other-slot";
    r.nextSlots = r.slots;
  }
}

The detail that matters is slotDeferredReason. The original reason for being due is preserved rather than overwritten, so the output alone explains why a task is absent from this run. Drop that field and, a day later, you cannot tell from the logs whether a task is broken or simply scheduled elsewhere.

Fix 2: count misses per run, not per day

The detection side had a matching hole. The old counter, consecutiveMisses, is derived from lastRun, which is a date. A per-run task touched in the morning sets lastRun to today, so dropping it that same night leaves the counter at zero. A task that is handled every morning and skipped every night is structurally invisible at day granularity.

So state.run.ledger now keeps a per-run record, and this counts how many recent runs in a row a task was skipped:

function missedRuns(state, taskId, opts = {}) {
  const runs = recentRuns(state, { excludeCurrent: true, ...opts });
  if (!runs.length) return null;
  let n = 0;
  for (const r of runs) {
    const t = r && r.tasks && r.tasks[taskId];
    if (t && ATTEMPTED.has(t.status)) break;
    n++;
  }
  return n;
}

Returning null rather than 0 for an empty ledger is deliberate. 0 reads as "never dropped", which quietly conflates "no data" with "healthy".

Ledger entries use four statuses: done, noop (looked, found nothing new), failed, and deferred. Collapsing noop into done makes every legitimate empty check show up as a warning until nobody reads the warnings any more. Pushing noop toward "not attempted" collides with the rule that a run must never end with zero output. Four values, then.

With that in place, the first sort key of priorityOrder moved from days to missedRuns:

const priorityOrder = result
  .filter((r) => r.due)
  .map((r, i) => ({ r, i }))
  .sort((a, b) => {
    const ma = a.r.neverRun ? Infinity : a.r.missedRuns || 0;
    const mb = b.r.neverRun ? Infinity : b.r.missedRuns || 0;
    if (mb !== ma) return mb - ma;
    const da = a.r.neverRun ? Infinity : a.r.daysSinceLastRun || 0;
    const db = b.r.neverRun ? Infinity : b.r.daysSinceLastRun || 0;
    if (db !== da) return db - da;
    return a.i - b.i;
  })
  .map((x) => x.r.id);

Anything dropped at night surfaces the next morning with missedRuns=1 and lands at the front, so the same set cannot be dropped twice in a row. Environments without a ledger get missedRuns=null, are treated as 0, and fall back to the old day-based ordering.

What the output looks like

The decision is reproducible with no side effects. --no-mark skips the start marker, so you can run it as often as you like without touching state.json.

node automation/scripts/whats-due.js --no-mark --date 2026-08-20 --batch noon

Here is one task from that output:

{
  "id": "menrui-daily",
  "site": "menrui",
  "due": false,
  "reason": "other-slot",
  "cadence": "per-run",
  "slots": ["evening"],
  "slot": "noon",
  "slotDeferredReason": "research-and-append",
  "nextSlots": ["evening"],
  "lastRun": "2026-08-20",
  "neverRun": false,
  "daysSinceLastRun": 0,
  "consecutiveMisses": 0,
  "missedRuns": 3,
  "attemptRate": { "attempted": 1, "of": 6 }
}

lastRun is today, so consecutiveMisses is 0 and the day-based view calls this healthy. The run-based view disagrees: three consecutive runs skipped, and only one attempt in the last six. That is exactly the shape day granularity cannot see.

Running the same command per slot and tallying due against other-slot shows what the split bought:

morning  due=6  other-slot=14
noon     due=5  other-slot=15
evening  due=5  other-slot=15
night    due=6  other-slot=14

Before slots, all 20 of those (the sum of both columns) were due in every run. Now each run gets five or six. Nothing was made less frequent: the daily total is preserved by the slots assignments, and sites that are supposed to update twice a day are assigned to two slots.

Gotchas

1. Configuration mistakes need their own exit, separate from the due check

Change a quota.kind from research-append to research-apend and the task falls into the unknown-kind branch, which safely sets due=false. So far so good. The problem was that staleWarnings filtered on r.due, so I measured a task sitting unrun for 207 consecutive days while both the warning list and the starvation list stayed empty and the process exited 0. One typo, and a task disappears without a sound.

Unknown kinds now always land in configWarnings:

const configWarnings = result
  .filter((r) => r.reason === "unknown-quota-kind")
  .map((r) => ({
    id: r.id,
    site: r.site,
    problem: "unknown-quota-kind",
    quotaKind: r.quotaKind,
    lastRun: r.lastRun,
    daysSinceLastRun: r.daysSinceLastRun,
    hint: "...",
  }));

Missing slots (silently means "every slot", so the count never drops) and unknown slot names (silently means "never due") go to the same place, for the same reason: neither is fixable until somebody sees it.

2. "Not part of the daily run" deserves a declared kind, not just enabled:false

Some sites are measured but carry no daily production quota. Excluding them with enabled:false alone means that re-enabling the entry silently adds daily work. A dedicated quota.kind: "on-demand" states the exclusion explicitly, doubles up with enabled, and keeps an intentional opt-out distinguishable from a typo in the output.

3. Make the start marker a side effect, not a step

"Record the start of the run" was step zero of the runbook. It was skipped on essentially every run, which meant the silent-failure detector never had anything to compare against. It now happens inside whats-due.js via markStart(). The decision always runs first, so making it a side effect of the decision makes it unskippable.

4. Read order matters around that marker

The state used for the ledger has to be re-read after markStart(). markStart() rewrites run.lastStart, so reusing the state object read at the top of the script makes "the current run" point at the previous one, which excludes the most recent run and undercounts misses by one.

5. Two machines need two sets of markers

Once runs alternate between two machines, a single pair of start/complete markers breaks: the next machine's start overwrites the previous machine's record. Either an interruption vanishes entirely, or a late completion marker gets matched against a different run and reports a false positive. Markers now live under state.run.hosts[<host>] and are compared within a host. The shared lastStart / lastComplete fields are still written for existing readers.

The result

One of the sites this keeps updated daily: https://manga.autoarticles.net

Wrap-up

• An overflowing queue does not shrink when you reorder it. Split the count to match per-run capacity.
• Keep the reason you deferred something (slotDeferredReason) in the output, so a smaller queue stays observable.
• Count misses per run, not per day. Day granularity cannot see a task that is handled in one run and dropped in another.
• Absent data is not healthy data. An empty ledger returns null, not 0.
• Give configuration errors their own exit (configWarnings). Failing safe is not enough if failing safe is also failing silently.

Making the decision deterministic is only the starting point; the automation settles down only once the decision's output is sized to what one run can actually finish.

---

This article is about my own side project. It was written with AI assistance.