DEV Community

Daniel Pertu
Daniel Pertu

Posted on

Three ways this codebase refuses to do a thing twice, and the TTL is never the runtime

Munchable has three kinds of work that must not happen twice: webhook deliveries, scheduled jobs, and a monthly settlement pass that touches subscriptions. All three are "exactly once" problems. None of them use the same mechanism, and the reason is that they fail differently.

The common mistake in all three, which I made in two of them, is the same: setting the guard's lifetime to how long the work takes.

1. Webhooks: a short claim, promoted on success

Stripe and Resend both retry on a non-2xx and both occasionally deliver a duplicate. The handlers have side effects, so each event id is claimed before anything happens:

const key = `webhook:${provider}:${eventId}`;
const claimed = await redis.set(key, Date.now().toString(), { nx: true, ex: ttlSeconds });
if (claimed === null) return { isFirstDelivery: false, degraded: false };
Enter fullscreen mode Exit fullscreen mode

SET NX plus a TTL, rather than a table, because the record only has to outlive the provider's retry window. Svix's retries taper off well inside 24 hours, so 24 hours is the dedupe horizon.

Except the first claim is not written for 24 hours. It is written for 120 seconds:

export const PROCESSING_TTL_SECONDS = 120;
Enter fullscreen mode Exit fullscreen mode

and only promoted to the full horizon after the handler has succeeded. On a caught failure, the claim is deleted outright so the provider's retry is not mistaken for a duplicate.

The 24 hour claim up front is the version that looks right and is wrong. It is correct until the handler dies in a way that reaches no cleanup code: a hard timeout, an instance killed mid-request, an out-of-memory. Then the key sits there for a day, the provider's retries all get "already processed", and an email is never forwarded or a subscription never updated. Nothing in any log says anything is wrong, because from the handler's point of view those retries were duplicates.

120 seconds is "longer than the handler can possibly take, shorter than I am willing to lose an event for". A crash expires on its own, the retry gets a real second attempt, and the only cost is that handlers have to be idempotent anyway.

And it fails open:

if (!redis) {
  console.error('[webhook-idempotency] Redis not configured, cannot de-duplicate.');
  return { isFirstDelivery: true, degraded: true };
}
Enter fullscreen mode Exit fullscreen mode

Processing a duplicate is better than dropping a real event, so when the de-duplicator is unavailable the event is processed and the result is marked degraded. Same reasoning as the rate limiters on the read paths, opposite answer to the ones that spend money.

2. Scheduled jobs: a lease, not a lock

Two jobs run on a schedule, plus a long-running curation job that can also be started from a CLI. Overlap is the failure mode: a slow run still going when the next tick fires, or a manual trigger on top of a scheduled one.

acquired = await client.set(`cron:${name}`, String(Date.now()), { nx: true, ex: lease });
if (acquired !== 'OK') return { ran: false, reason: 'locked' };
Enter fullscreen mode Exit fullscreen mode

The lease is 300 seconds and it is renewed every 90 while the work runs:

export const LOCK_LEASE_S = 300;
export const RENEW_EVERY_MS = 90_000;
Enter fullscreen mode Exit fullscreen mode

It used to be the job's whole worst-case runtime: three hours on the scheduled runner, six from the CLI. The reasoning was that the lock must outlive the run, which is true, and it is still the wrong design.

Here is what it cost, and it is my favourite kind of bug because nothing anywhere reported an error. A run was cancelled by hand. A cancelled run never reaches its finally, so the key stayed for the full three hours. The schedule kept firing every fifteen minutes, and every one of those runs did exactly what it was told to do: saw the lock, skipped, and returned { skipped: true, reason: 'locked' } with a 200. The platform records a 200 as a successful run.

So the dashboard showed a green tick every quarter of an hour, for three hours, while nothing was curated at all. I watched it happen on 14 September after cancelling two runs.

A lease fixes it by separating two things the single number was conflating. How long may the work run? As long as it likes. How long may a dead holder keep the lock? One lease, 300 seconds. The renewal interval is a third of the lease, so one failed renewal is survivable, and the renewal timer is unref'd so a pending renewal can never be the reason a worker process refuses to exit.

The signature carries the lesson as a comment, because the API itself invited the original bug:

`leaseSeconds` is exposed for tests. Callers should not pass it: a caller that
thinks it is declaring "how long my job may run" is the bug this signature
used to invite.
Enter fullscreen mode Exit fullscreen mode

There is also a deliberate preference buried in the renewal's error handler. If a renewal blip loses the lock entirely, the risk is an overlap. An overlap is a worse outcome than a single run, and a far better outcome than curation silently stopping for hours. When a guard has two ways to fail, you have to pick which one, out loud.

Both of those jobs have the same two guards in front of them: a bearer token compared in constant time, refusing with 503 when the secret is not configured rather than running unauthenticated, and then the lease. Unlike the webhook claim, the lease fails closed: no Redis, no run. These jobs spend money and write curated data, so a run that cannot be coordinated does not happen.

One of them is the hourly search-engine announcement I wrote about in 516 fingerprints in one Redis hash, which keeps the 516 pages at munchable.app/sitemap.xml in front of the engines within the hour. Its normal run submits nothing, which is exactly why a silent skip would have gone unnoticed there too.

3. A monthly settlement: the claim is a row

The third job settles contributor rewards once a month. Two passes, both of which must be safe to run again: one records what each contributor earned for the previous calendar month, one applies those to subscriptions.

This one cannot use a Redis key, because the thing being guarded is not an invocation. It is a row. A reward that is halfway applied when the instance dies has to be retryable without being applied twice, and that state lives in the database.

So the claim is the UPDATE itself: rows in a claimable state are moved to applying by a single statement with RETURNING, which means the database decides who owns each row and a second runner finds nothing to do. Rows stuck in applying for more than fifteen minutes, from a run that died between the claim and the result, are treated as claimable again.

Three details make it work:

  • the earn pass writes at most one row per contributor per period, and leaves an existing row alone,
  • settlement runs on a lag rather than at the end of the month, so other people have time to report a bad contribution before anything moves,
  • the apply pass asks the payment provider whether the reward is already on the subscription, because the strongest idempotency guarantee available is the one from the system that actually holds the state.

That last point is the general version of all three sections. A claim in Redis protects against a retry; a claim in a row protects against two runners; only the provider can tell you whether the money already moved.

The sentence I wish I had written first

Every one of these guards has a duration in it, and the bug in two of the three was reading that duration as "how long the work takes". It is not. It is "how long I am prepared to be wrong if the holder disappears".

Webhook claim: 120 seconds, then let the retry through. Job lease: 300 seconds, then let the next tick in. Stale applying row: 15 minutes, then let another runner take it. None of those numbers is the runtime of anything.

The product underneath all this plumbing is at munchable.app, and the part the curation job feeds is the 461 question pages indexed at munchable.app/answers. The jobs are boring when they work, which took a surprising amount of code to arrange.

Top comments (1)

Collapse
 
krusenas profile image
Karolis •

Try replaying the same delivery with the processed state deleted or expired, then check the handler doesn't repeat the side effect. That's the path that usually goes untested. fwiw I built Webhook Relay. It stores every request and lets you replay past ones.