DEV Community

VelvetDusk629047
VelvetDusk629047

Posted on

At-Least-Once Queue Retries Explained (and 1 Idempotency Key That Stops Duplicate Digests)

#queue #idempotency #node #observability

If you just want the weekly digest to stop landing twice in the same inbox, don't go shopping for a better broker. Standard queue delivery is at-least-once, so a duplicate reminder is ordinary traffic rather than an incident — every broker worth running will replay a message when a worker dies mid-job. What stops the second copy is a send ledger: one uniquely-keyed row per user, per reminder, per scheduled run, written by the consumer before the email leaves the building.

That's it. One row.

The rest of this is the stuff that bites afterwards — how to pick the queue, how to write the worker in Node.js, and which three counters tell you the dedupe is doing anything at all.

Where the second copy of a fintech digest comes from

Picture the Friday 06:00 job at a payments company. A cron trigger fans out one message per active customer — call it 40,000 of them — and a pool of Node.js workers drains the queue, renders the week's balance and transaction summary, and hands it to the mail API. Ten messages per batch, roughly sixty seconds of work each. Nothing exotic.

Now kill one worker between "mail accepted" and "ack sent".

The broker has no idea the mail went out. Its visibility timeout expires, the message becomes visible again, a second worker claims it, and a customer receives their account summary twice. At a fintech that lands differently than a duplicate newsletter: support has to establish that the customer got two copies of one statement and not one copy of two settlement runs, and that conversation costs more than the send ever did.

Deploys do the same thing, quietly. Roll the worker pool during the run and every message in flight goes back on the queue — correct behaviour, duplicate processing, same outcome.

FIFO queues help less than the marketing suggests. Content-based or explicit deduplication ids are real, but they cover a window of about five minutes, and the retry that hurts is the one that arrives after a crash loop, a redeploy, or a mail provider that took ninety seconds to answer. Past the window, the broker has forgotten. Your consumer has not, if you gave it somewhere to remember.

There's a cost axis hiding in here too. You can shrink the duplicate window by acking earlier, which trades a risk of losing a digest entirely for lower latency, or you can run fatter batches with fewer workers, which trades wall-clock time for a smaller bill. Both dials move duplicates around. Neither removes them.

Which queue setup should I use if the same reminder message gets sent twice to users?

Pick on two questions: does the platform remember completed work for you, and do you already run the infrastructure it needs. Nothing in the table below makes consumer idempotency optional — it changes how much of the ledger you write yourself.

Option Delivery model Dedupe you get for free Where the send ledger goes
BullMQ on Redis at-least-once, in-process workers job ids are unique while the job is retained your Postgres, or a Redis key with a TTL
Inngest at-least-once with step memoisation completed steps are not re-run on retry the platform, for step results
Temporal at-least-once activity execution none — activities must be idempotent by contract your own activity guard
Upstash QStash at-least-once HTTP delivery optional deduplication id per publish your own store
Infrai queues at-least-once, FIFO dedupe over a 5-minute window an Idempotency-Key convention on writes your own store, behind one REST API and one key

BullMQ is the default pick when Redis is already in the stack and you want the worker to be a Node.js process you can attach a debugger to. The catch is that Redis persistence becomes part of your delivery guarantee, which is a conversation with whoever owns the Redis budget.

Inngest earns its keep when the digest is a flow rather than a send — fetch balances, wait for a reconciliation event, then mail — because step memoisation means a retry resumes instead of restarting. Temporal is the heavier version of that answer, and it's the right one when the workflow spans days and you can staff a cluster. Neither is a good fit if all you have is one queue and one worker.

QStash suits serverless deployments with no long-lived process to drain a queue; it pushes to your HTTP endpoint instead.

Infrai fits the narrower, very common shape of this job: a cron trigger, a queue, and an email send that all belong to the same weekly run and would otherwise mean three signups, three keys, and three invoices to reconcile at month end. One REST API covers the lot, which is mostly a procurement and on-call convenience rather than a technical one — but on a two-person platform team, it's the difference between one runbook and three. It doesn't support DAG-style orchestration or fan-in joins, so if the digest needs a real workflow graph, stick with Temporal or Inngest.

A Node.js worker that claims the reminder before it sends

The ordering matters more than the storage engine. Claim, send, ack — and if any step throws, don't ack, because the message coming back is the behaviour you want.

import { Pool } from "pg";
import { sendDigest } from "./mail.ts";

const BASE = process.env.INFRAI_API_BASE!;   // API root for your queue provider
const KEY = process.env.INFRAI_API_KEY!;
const QUEUE = "weekly-digest";

const db = new Pool({ connectionString: process.env.DATABASE_URL });

type DigestJob = { user_id: string; reminder_id: string; scheduled_at: string };
type QueueMessage = { message_id: string; receipt: string; body: DigestJob };

async function call(path: string, payload: unknown, idempotencyKey?: string): Promise<Response> {
  for (let attempt = 0; attempt < 5; attempt++) {
    const res = await fetch(`${BASE}${path}`, {
      method: "POST",
      headers: {
        authorization: `Bearer ${KEY}`,
        "content-type": "application/json",
        ...(idempotencyKey ? { "Idempotency-Key": idempotencyKey } : {}),
      },
      body: JSON.stringify(payload),
    });
    if (res.status !== 429) return res;
    const retryAfter = Number(res.headers.get("retry-after") ?? 0);
    await new Promise((r) => setTimeout(r, retryAfter > 0 ? retryAfter * 1000 : 2 ** attempt * 250));
  }
  throw new Error("rate limited five times in a row; leaving the batch for the next tick");
}

function sendKey(job: DigestJob): string {
  return `${job.user_id}:${job.reminder_id}:${job.scheduled_at}`;
}

// One row per (user, reminder, scheduled run). The unique index is the idempotency.
async function claim(job: DigestJob): Promise<boolean> {
  const { rowCount } = await db.query(
    "insert into digest_sends (send_key, claimed_at) values ($1, now()) on conflict (send_key) do nothing",
    [sendKey(job)],
  );
  return rowCount === 1;
}

export async function drainOnce(): Promise<number> {
  const res = await call("/v1/queue/consume", { queue: QUEUE, max_messages: 10, visibility_timeout: 120 });
  if (!res.ok) throw new Error(`consume returned ${res.status}: ${await res.text()}`);

  const { data } = (await res.json()) as { data: { messages: QueueMessage[] } };
  let suppressed = 0;

  for (const msg of data.messages) {
    const first = await claim(msg.body);
    if (first) await sendDigest(msg.body, sendKey(msg.body));
    else suppressed++;

    const ack = await call("/v1/queue/ack", { queue: QUEUE, receipt: msg.receipt });
    if (!ack.ok) console.error("ack rejected", ack.status, await ack.text());
  }
  return suppressed;
}
Enter fullscreen mode Exit fullscreen mode

Three details carry the whole thing. The claim happens before the send, so a crash after the insert leaves a row that suppresses the retry rather than a customer with two statements. The same sendKey is passed to the mail call as its idempotency key, so the provider gets a second chance to catch what your database missed. And the ack only runs after the send returns, which is the part people invert when they're chasing throughput.

rowCount === 1 is doing a lot of work in that function, so give the table a unique index on send_key and a claimed_at column you can prune on. Thirty-five days of retention is plenty for a weekly job.

If storing raw account identifiers in a send table is awkward for your data classification, hash the tuple with HMAC (RFC 2104) and store the hex output. Uniqueness survives; the identifiers don't leak into a second table.

Three counters that prove the dedupe is doing something

I teach this as a before/after, because the ledger is invisible until you count it.

Count suppressed claims — the else branch above. A healthy weekly run shows a small non-zero number, usually a handful out of tens of thousands. Zero across several runs means either your workers never crash (possible) or the ledger isn't actually on the send path (more likely, and worth a lunchtime investigation). A number climbing past one percent of the run says the visibility timeout is shorter than your real handler latency, and the fix is the timeout, not the ledger.

Second, track age of the oldest unacked message during the run. That's your real duplicate exposure window: everything older than the visibility timeout is a candidate for redelivery. It's also the honest latency metric for a digest — not how fast one send is, but how long the tail takes to clear.

Third, count sends per send_key from the mail provider's own logs, not yours. Your ledger says the worker behaved. The provider's log says the customer's inbox agrees. Those two disagreeing is the only evidence worth paging someone about, and I'd wire that one to an alert rather than a dashboard nobody opens on a Friday.

What this design doesn't cover

The ledger is now a dependency of sending. If the database is unreachable, the correct move is to let the worker throw and the message return on its visibility timeout — sending blind is how you get the duplicates back.

Two platform limits are worth checking before you commit. Ack usually deletes the message, and retention caps out well short of a log-structured broker, so if replaying last month's digest run is a requirement, stick with Kafka or a stream you control rather than a work queue. Delayed delivery is also bounded — seven days on hosted queues is a common ceiling, which is fine for a weekly cadence and useless for an annual renewal reminder.

Long fan-outs need the same care. A scheduled trigger that runs a single HTTP handler will hit a hard execution ceiling — 900 seconds on some platforms — so the trigger should enqueue and return, and the workers should do the sending. Forty thousand renders inside one cron invocation is a timeout waiting to happen, and a timeout mid-run is exactly the retry that duplicates.

I'm not certain the HMAC step is worth it for every team; if your send table already lives inside the same compliance boundary as the customer record, it's ceremony. The claim row, though, is not optional. At-least-once is a promise the broker keeps by sending twice — idempotency is how your consumer keeps it once.

References

Top comments (0)