We upgraded our job platform at 10:22 and deleted it at 13:34

On 16 September the Munchable repo has four commits about its background job platform. At 10:07 the curation cadence was changed. At 10:18 the CLI version was pinned because a mismatch aborts the deploy. At 10:22 the packages were upgraded to the latest release. At 13:34 the platform was removed, along with 2,500 lines that existed to serve it.

This is not a post about that platform, which did what it said. It is about noticing that the work no longer needed a platform at all, and what replaced it.

What the schedule was actually doing

Why nothing is scheduled. Curation used to sit on `*/30 * * * *`: with a drained
backlog that was 96 runs a day reading ten thousand rows each to decide there
was nothing to do. Making it requested instead of scheduled fixed the bill but
left a whole deployment target, a second set of environment variables and a
second place for a run to die, all to carry work that fits in `after()`.

The curation pipeline names ingredients scraped off labels. Most of it is deterministic layers with a model call at the end, and over the summer the deterministic part moved into the request path: a scan names what it can inline and writes the answer straight into the overlay. What was left for a background job was the tail of one label, one overlay read and at most one model call, and that fits inside Next's after() with no runner at all.

The lesson that had to be learned first, from the deleted dispatch code:

// Handing work to the hosted runner is itself an HTTP call, and a bare
// floating promise is not work the platform knows about: the response is
// sent, the invocation is frozen or recycled, and the hand-off never arrives
// (nor does the `.catch`, so nothing is logged). Under `after()` the call is
// part of the invocation and runs to completion.

The remaining work is the bulk pass: drain a backlog of forty thousand words, review the commonest ingredients no condition speaks for. The doc's position on that is one sentence: "a bulk pass wants a person watching it."

The replacement is a command with a loop in it

pnpm job taxonomy-curate --max 2000            # deterministic pass + model
pnpm job taxonomy-curate --no-model            # deterministic pass only
pnpm job taxonomy-curate --until-done          # keep going until it is empty
pnpm job rules-review --allow-general --top 500

A run stops at its row cap and reports capped: true. Forty thousand rows at the default cap is a hundred invocations, and a boolean you have to read and act on is a loop with a person standing in it. So the loop is in the script, under one lock acquisition:

async function drain(): Promise<unknown> {
  if (!flag('--until-done')) return body();
  let last: Record<string, any> = {};
  for (let pass = 1; ; pass++) {
    last = (await body()) as Record<string, any>;
    const progressed = (last.accepted ?? 0) + (last.deterministic?.resolved ?? 0) > 0;
    console.log(`[pass ${pass}] ${JSON.stringify(last)}`);
    if (!last.capped) return last;
    if (!progressed) {
      console.error(`[pass ${pass}] still capped but nothing was written; stopping rather than looping`);
      return last;
    }
  }
}

Progress is the guard. A run that writes nothing has hit something it cannot get past, and running it again would ask the same rows the same question forever. One unproductive pass ends the loop and its summary is what gets printed, so the reason is on screen rather than buried in the runs before it.

There is no queue table for this. Resumption is the backlog row's own status plus a marker on each row recording which version of the deterministic layers last looked at it, so a row is not re-read on every pass until the layers learn something new. In the state this replaced, the same 10,586 rows were re-read four times an hour.

The step clock has to clear the client's worst case

The batch runner puts several model calls in flight and handles their results strictly in order, because each handler validates against a merge state the previous batch just added to. It also has a per-step clock so a hung call is reported rather than waited on forever. That clock had a bug that is easy to ship:

/**
 * `createOpenAIClient` gives the curation jobs a 90 s timeout and one retry, so
 * a call that IS retried legitimately runs for up to 180 s. At 120 s the step
 * clock fired first and reported a healthy retry as a stall; three in a row end
 * the run, which is how a 36,000-row drain stopped after 320 rows. Any step
 * clock over a model call has to clear the client's own worst case or it is
 * measuring the wrong thing.
 */
export const MODEL_BATCH_STEP_MS = 210_000;

Concurrency makes this likelier rather than causing it: the provider queues the calls, every one slows down as the others pile up, and the ones that then retry are the ones that used to trip the clock. The review job now derives its step clock from its own client timeout and retry count in code, so the two cannot drift apart again.

The second thing deleted: the scoring queue

The same commit removed a second pair of jobs entirely. They mined ingredients the engine could name but not score, queued them, and asked a model to write clearances. They drove a number no user sees, and the clearances they wrote were a liability rather than data. What replaced them is not a job:

• a script that measures which ingredients on real labels a condition speaks for, ranked by occurrence rather than by distinct id, because deciding en:flour settles every flour under it;
• a one-pass review of the silent ones that writes positive triggers only;
• a script that runs the engine over random real products and prints what a user would be told, because "coverage percentages are a proxy; this is the thing itself."

The trade is stated in the docs rather than hidden: a word that survives the lexicon, the free layers and one model call stays in the backlog until somebody runs the command. That is affordable because the engine no longer penalises a label for carrying an unnamed word, and because such a word is nearly always a brand, an OCR misread or half a sentence rather than an ingredient any condition cares about.

What you can see from outside

Scan a product whose label has a word the engine has never met, and the name is resolved in the request path, not by a job. There is no queue to wait on. Sign in and scan something obscure, or photograph a label the catalogue does not have. The answer pages at munchable.app/answers are the output of the curated data those commands maintain.

Two earlier posts set this up: 60% of my LLM backlog was never a question for an LLM is why the deterministic layers exist, and the lock whose TTL was the job's worst case is the outage two days earlier that made a scheduled runner with no one watching it look less attractive.