Email Verification Workers Need a Delivery Lease

Email verification is often implemented as a single request: create a token, send a message, and return success. That model breaks when the mail provider times out after accepting the request, a worker crashes halfway through delivery, or two workers pick the same job.

The useful boundary is a delivery lease. The API records the verification intent in PostgreSQL, and a worker temporarily claims the resulting outbox row. A crash then leaves work available for another worker, while the lease prevents two active workers from sending the same message at the same time.

This is the pattern I reach for when an authentication flow must be boring under failure. It also keeps test data separate from real customer data; a disposable email address or free temporary email inbox can be useful for a controlled staging check, but neither replaces a server-side delivery contract.

Why a verification worker needs a lease

A queue acknowledgement is not the same as email delivery. Imagine a worker calls the provider and the network connection drops before the provider's response reaches your service. The worker cannot know whether the message was accepted. Retrying may send a duplicate, while refusing to retry may lose the verification email.

There is no universal way to remove that ambiguity across an arbitrary email provider. The practical goal is to make the duplicate safe and the state observable. Store one logical delivery per verification request, assign it a bounded retry budget, and record provider message IDs when they are available.

The lease handles a different problem: concurrent ownership. It should have an owner, an expiry timestamp, and a version or attempt number. A worker may process a row only while its lease is current. If the process dies, the expiry makes the row claimable again.

Define the outbox and lease states

Keep the state machine small. A typical row in an email_outbox table contains:

• id and verification_id, with a unique constraint on the logical verification delivery
• status, such as pending, sending, sent, or failed
• lease_owner and lease_until
• attempts, next_attempt_at, and last_error
• provider_message_id and timestamps for creation and completion

The verification record should own the token and its expiration. The outbox row owns delivery attempts. Mixing those concerns makes it easy for a mail retry to accidentally create a new token, or for a token refresh to reset a delivery counter.

I also keep the token value out of ordinary logs. Log a verification ID and a short correlation ID instead. A temp mailid appearing in a debug line is not harmless if the log retention and access policy were never designed for it.

Claim work safely in PostgreSQL

Workers can claim a batch with a short transaction. FOR UPDATE SKIP LOCKED is a good fit when several workers poll the same table:

WITH candidate AS (
  SELECT id
  FROM email_outbox
  WHERE status = 'pending'
    AND next_attempt_at <= now()
    AND (lease_until IS NULL OR lease_until < now())
  ORDER BY next_attempt_at, id
  FOR UPDATE SKIP LOCKED
  LIMIT 25
)
UPDATE email_outbox AS e
SET status = 'sending',
    lease_owner = $1,
    lease_until = now() + interval '2 minutes',
    attempts = attempts + 1
FROM candidate
WHERE e.id = candidate.id
RETURNING e.*;

The transaction should commit before the worker performs the network call. Holding row locks while waiting on a provider creates unnecessary contention. The two-minute value is only an example; choose it from measured provider latency and worker shutdown behavior, then renew it for genuinely long sends.

When updating the result, include both id and lease_owner in the WHERE clause. A stale worker must not mark a row as sent after another worker has reclaimed it. This small condition prevents a late response from corrupting the current state.

Make retries and expiration explicit

Use exponential backoff with a ceiling, for example one minute, five minutes, and fifteen minutes, instead of retrying in a tight loop. Classify provider responses into transient failures, permanent failures, and unknown outcomes. Unknown outcomes deserve a retry policy plus an alert when the same logical delivery keeps oscillating.

The token expiration and delivery retry deadline should be related but not identical. A token can expire while its final message is still being attempted. In that case, stop sending it and let the API issue a fresh verification request. Otherwise users receive links that were already unusable when the message arrived.

At the API layer, accept an idempotency key for the create-verification operation. Repeating the same request should return the existing verification result, not create another outbox row. A different email payload with an existing key should be rejected as a conflict. This gives clients a stable REST API even when their timeout handling is imperfect.

For end-to-end coverage, combine database assertions with a mailbox check. A browser-level email check can validate the user-visible link, while a privacy review for email fixtures helps keep test addresses and message content inside the right boundary.

Operational checklist

Before shipping this worker, I verify the following:

1. The logical verification delivery has a unique database constraint.
2. Claims use short transactions and skip rows already leased.
3. Result updates reject expired or foreign leases.
4. Retry counts, lease age, provider latency, and permanent failures are measurable.
5. Expired tokens cannot be sent, even if an old job is retried.
6. Shutdown extends or releases leases in a predictable way.
7. Staging uses isolated mailboxes and redacted logs.

The important design choice is not the queue product. It is separating verification state from delivery state, then giving every worker attempt a bounded ownership period. Once that contract is in the database, provider timeouts and process crashes become recoverable cases instead of mysterious authentication bugs.