Munchable is a barcode scanner for people with digestive conditions. Behind the app there are scheduled jobs, and two different things run them: Vercel Cron hitting a route, and a CLI a person runs in a terminal for the bulk data passes. One of those jobs announces changed pages like the recipe index to search engines every hour; another settles contributor rewards once a month.
They share one guard, withCronLock(name, fn), which is a Redis key under cron:<name> taken with NX and a TTL. Nothing surprising. The interesting part is what that TTL used to be.
The reasonable version that was wrong
It used to be the job's whole worst-case runtime: three hours for the scheduled runner, six from the CLI. The reasoning writes itself. The lock has to outlive the run, otherwise a second runner starts while the first is still going.
That reasoning is correct. The cost just lands somewhere else:
/**
* It used to be the job's whole worst-case runtime, three hours on the
* scheduled runner and six from the CLI, on the reasoning that the lock must
* outlive the run. It does, but the cost landed somewhere else: a run that is
* cancelled or killed never reaches its `finally`, so the key sat there for
* the full three hours while the fifteen-minute schedule kept firing, and
* every one of those runs skipped with `{ skipped: true, reason: 'locked' }`,
* which the platform records as a SUCCESSFUL run. The dashboard showed green
* every quarter hour while nothing was curated for three hours. Observed
* exactly that on 2026-09-14, after two runs were cancelled by hand.
*/
Two things combined there, and both are worth separating out.
A finally block is not a guarantee. It runs when the function returns or throws. It does not run when the process is killed, when a platform cancels an invocation, or when somebody hits the stop button in a dashboard. Any cleanup that only exists in a finally is cleanup that is skipped exactly when something has already gone wrong.
And a skip that returns 200 is invisible. Every quarter hour the schedule fired, found the lock, returned "skipped, reason: locked", and the platform ticked it green. The monitoring was not broken; it was faithfully reporting that the HTTP request succeeded. Nobody had told it that a successful request and a completed job are different events.
A lease, not a timeout on the work
The fix is to stop treating the TTL as a statement about how long the job takes, and make it a statement about how recently the job was alive:
export const LOCK_LEASE_S = 300;
/** Renewed comfortably inside the lease, so one failed renewal is survivable. */
export const RENEW_EVERY_MS = 90_000;
Five minutes, renewed every ninety seconds, which means the lease is three ticks long and a single failed renewal costs nothing. A live job holds the lock for as long as it likes because it keeps saying so. A dead job stops saying so and loses it within five minutes instead of three hours.
const renew = setInterval(() => {
void client.expire(key, lease).catch(() => {
// A blip is survivable: the next tick renews, and the lease is three
// ticks long. Losing the lock entirely only risks an overlap, which is
// the lesser failure against curation stopping for hours.
});
}, RENEW_EVERY_MS);
renew.unref?.();
The unref matters in the CLI case: a pending renewal timer must never be the reason a process refuses to exit after its work is done.
The release path still deletes the key, and the comment on its catch block is now a much smaller promise:
} catch {
// The lease releases it, now within five minutes rather than three hours.
}
That is the shape of the whole change. The explicit cleanup is still there and is still the normal path. It simply stopped being the only thing standing between a crash and a stalled pipeline.
The signature invited the bug
My favourite part of the rewrite is not the lease. It is this:
/**
* `leaseSeconds` is exposed for tests. Callers should not pass it: a caller
* that thinks it is declaring "how long my job may run" is the bug this
* signature used to invite.
*/
The old API took a TTL per call site, so every caller had to form an opinion about its own worst case, and that opinion was wrong in the same way every time. The parameter survives for tests, with a comment naming the misunderstanding rather than trusting that it has been designed out.
Which failure to prefer
There is a real trade here, stated in the code rather than left implicit. If the lock lapses under a running job, two runners can overlap. If the lock is stuck, work stops for hours and looks fine. The second is worse for this product, so the lease is short and a lost renewal is tolerated.
The opposite decision is a few lines away. When Redis is unavailable entirely, the lock cannot be taken and the job does not run at all:
const client = redis;
if (!client) return { ran: false, reason: 'lock_unavailable' };
These jobs spend money on model calls and write curated data, so an unlocked run is worse than no run. The rate limiter in the same codebase makes the opposite call and falls back to an in-memory limiter, because a few unmetered requests beat a sign-up page that 500s. Same outage, opposite answer, and the deciding question is only ever what a mistake costs.
The auth guard follows the same rule. A missing CRON_SECRET is a 503, not a free pass:
if (!secret || secret.length < 16) return 'unset';
A mismatch is a 401, the comparison is timingSafeEqual, and the decision is a pure function of a header and an environment variable so it can be tested without a request.
Giving "I did nothing" a status code
The last piece is the one I would do first next time. The scheduled routes no longer answer 200 when they skip:
if (!run.ran) {
return NextResponse.json(
{ error: run.reason, period },
{ status: run.reason === 'locked' ? 409 : 503, headers: noStore },
);
}
409 means another runner holds the lock. 503 means there was no lock to take. Both are visible in exactly the place that was showing green ticks, with no dashboards to build and no alerts to configure.
The CLI got the same treatment from the other direction. Its long drain runs many passes under one lock acquisition rather than a hundred, stops when a pass stops making progress, and prints the reason it did not run at all:
not run: locked (another run holds the lock, or Redis is unavailable; --no-lock overrides)
The product side of all this is meant to be boring: pages like the ingredient answers index and the recipe pages stay current, and the hourly job that tells search engines so either ran or said out loud that it did not. The failure I actually want to never repeat is not a crash. It is three hours of green ticks over nothing.
Top comments (1)
The skipped-as-success point is the one to carry into other jobs: a skip needs its own status or metric, otherwise the dashboard cannot tell skipped from done. A cheap guard is an alert on time since the last run that actually did work, not on run failures, so three hours of green skips trips it. On the lease itself, one detail worth checking: if a renewal fires after the lock expired and another runner took it, a bare EXPIRE will extend the new owner's key. Renewing only when the stored value matches a per-run token (a small compare-and-expire script) keeps a stale runner from holding the lock it no longer owns.