Use Outbox Leases for OTP Email Workers

OTP email pipelines often look fine until retries land from two directions at once: the API retries a request, and the worker retries an unfinished send. That is when one login challenge can fan out into two or three messages, each with slightly different timing and logs. The auth flow still kind of works, but the system gets harder to trust.

The pattern I keep coming back to is simple: store a receipt for the logical send, then let workers claim the outbox row through a short lease in PostgreSQL. That keeps the REST API deterministic, gives workers a safe retry boundary, and makes incident review much less messy. It also pairs nicely with related lessons from session-bound email change flows and from broader immutable retry handling.

Why OTP email workers duplicate messages

The duplicate-send bug usually comes from a race between request identity and worker identity:

1. POST /otp/email creates a challenge and writes an outbox event.
2. The client never sees the response, so it retries.
3. A worker picks the first event but crashes after calling the mail provider.
4. Another worker picks the same event again because nothing recorded ownership clearly.

At that point you have two questions that matter more than the status code:

• Did the API create one logical send or two?
• Did one worker own the send attempt, or did multiple workers race it?

If you cannot answer both from database state, debugging turns into log archaeology. That is usuallly where teams start adding ad hoc cooldowns instead of fixing the contract.

The receipt and lease model in PostgreSQL

I split the problem into two durable records:

• A receipt row proves the API accepted one logical send for one user and one idempotency key.
• A lease on the outbox row proves which worker may send it right now.

The outbox table can look like this:

create table otp_email_outbox (
  id bigserial primary key,
  user_id bigint not null,
  challenge_id bigint not null,
  lease_token uuid,
  lease_expires_at timestamptz,
  sent_at timestamptz,
  created_at timestamptz not null default now()
);

The important bit is that the worker does not "just read pending rows". It atomically claims one row by setting lease_token and lease_expires_at. If the process dies, the lease expires and another worker can safely retry. If it lives, other workers leave that row alone.

For OTP flows, a short lease is enough. Thirty to sixty seconds usually covers provider latency without making recovery feel slow. I would rather make that window explicit than hope the queue semantics save me.

A deterministic claim query for workers

The claim step should be one statement, not a read-then-write dance:

with next_job as (
  select id
  from otp_email_outbox
  where sent_at is null
    and (lease_expires_at is null or lease_expires_at < now())
  order by created_at
  limit 1
  for update skip locked
)
update otp_email_outbox o
set lease_token = gen_random_uuid(),
    lease_expires_at = now() + interval '45 seconds'
from next_job
where o.id = next_job.id
returning o.id, o.user_id, o.challenge_id, o.lease_token;

This is the whole trick. for update skip locked keeps workers from piling onto the same row, and the lease expiry gives you a clean recovery path after crashes. PostgreSQL is doing exactly the job it is good at: one committed truth with boring concurrency rules.

Node.js API and worker example

My API side still uses a receipt so repeated requests stay stable:

const result = await db.tx(async (trx) => {
  const existing = await trx.oneOrNone(
    `select challenge_id, outbox_event_id
       from otp_send_receipts
      where user_id = $1 and idempotency_key = $2`,
    [userId, idempotencyKey]
  );

  if (existing) return { reused: true, ...existing };

  const challengeId = await createOtpChallenge(trx, userId);
  const outboxEventId = await insertOtpOutbox(trx, userId, challengeId);
  await insertOtpReceipt(trx, userId, idempotencyKey, challengeId, outboxEventId);

  return { reused: false, challengeId, outboxEventId };
});

Then the worker sends only if it still owns the lease:

const job = await claimOtpEmailJob(db);
if (!job) return;

await mailer.sendOtp(job.user_id, job.challenge_id);

await db.none(
  `update otp_email_outbox
      set sent_at = now()
    where id = $1 and lease_token = $2`,
  [job.id, job.lease_token]
);

That last predicate matters more than it looks. If the lease expired and another worker reclaimed the row, the old worker cannot mark it sent by accident. Small detail, big difference.

Where temporary inboxes still fit

I still use a temp mailbox in automated tests, but only as delivery evidence after the backend contract is sound. It is handy for isolating recipients, especially when a shared QA inbox gets noisy or when someone still has old notes mentioning tamp mail com in setup docs. But the inbox should confirm one accepted send, not decide whether the send was unique.

That distinction saves time in test failures. If the receipt and lease data say one logical send happened, I debug provider delivery or rendering. If the database says two sends were possible, I stay in backend land and fix the contract first.

Q&A

Why not rely only on the queue system?

Because most queues help with delivery, not with application-level identity. The backend still needs a durable answer for "was this logical OTP request already accepted?"

Should I clear the lease after sending?

You can, but I usually just set sent_at and keep the lease metadata for debugging. It costs little and helps a lot when you are tracing a weird retry path later.

Does this work only for OTP emails?

No. Password reset, verification, invite, and receipt-like email flows all benefit from the same shape. OTP just makes the race more obvious because users feel duplicate messages imediately.