Most job queues and message brokers advertise "at least once" delivery as a feature. It is, compared to the alternative of losing messages silently. What gets left out of the marketing is the second half of the sentence: at least once means a message can and eventually will be delivered more than once, and every handler that assumes otherwise is carrying a bug that just hasn't triggered yet.
This isn't a rare edge case. It's the normal, expected behavior of any durable queue with a visibility timeout, a retry policy, or a consumer that acknowledges messages after processing rather than before. The gap between knowing this abstractly and actually auditing your handlers for it is where most production incidents in this category come from.
Where the assumption sneaks in
The most common failure mode looks like this: a worker pulls a job, starts processing, and completes the side effect, an email sent, a charge captured, a row updated, but crashes or times out before it acknowledges the message back to the queue. From the queue's perspective, the message was never confirmed as processed, so it gets redelivered. From the handler's perspective, if it isn't written defensively, that redelivery means the side effect happens again.
Nobody sits down and decides to build a handler that breaks on duplicate delivery. It happens because the happy path, one message, one execution, one visible result, works fine in every test and in every demo, and the duplicate-delivery case only shows up under real network conditions, real timeouts, and real worker crashes that local development rarely reproduces.
Worse, the duplicate-delivery case is often rare enough in absolute terms, maybe a fraction of a percent of all jobs, that it takes weeks or months of production traffic before anyone notices a pattern, and by then the handler has usually been copied as a template for several other job types built the same way.
Where at-least-once semantics actually come from
Almost every message broker in production use today, Kafka, RabbitMQ, Redis-backed queues, cloud provider queue services, guarantees at-least-once delivery by design, not as a bug. The mechanism is consistent across all of them: a consumer receives a message, the broker starts a redelivery timer, and if the consumer doesn't acknowledge within that window, the broker assumes something went wrong and redelivers. That's a deliberate tradeoff favoring "the message definitely got processed eventually" over "the message got processed exactly once," because guaranteeing exactly-once delivery across a network boundary is a much harder problem, and most systems that claim to solve it are actually doing at-least-once delivery plus idempotent processing underneath.
This is worth explaining to a team explicitly rather than assuming everyone already knows it, because the phrase "exactly once" shows up often enough in marketing copy and conference talks that engineers reasonably assume some systems have actually solved the harder problem. In practice, the honest framing is closer to "at least once delivery, with tooling that makes idempotent processing easier to implement correctly," and understanding that distinction changes how a team designs handlers from the start rather than discovering the gap after an incident.
The fix is not a smarter broker
Teams sometimes go looking for an "exactly once" queue technology, assuming the current stack is just missing a feature. In practice, true exactly-once delivery requires either a single atomic operation spanning the message read and the side effect, which most systems can't offer across arbitrary external calls, or idempotent handlers that make duplicate delivery harmless regardless of how many times a message arrives. The second approach is the one that actually scales to real systems with external side effects like emails, webhooks, and payment charges.
def process_order_confirmation(order_id):
if already_sent(order_id):
return
send_confirmation_email(order_id)
mark_sent(order_id)
That pattern looks trivial written out, and the mechanics of doing it correctly under concurrent retries, where two workers might both check already_sent before either has recorded a result, take more care than the snippet suggests. A unique constraint on whatever table tracks "already sent" state, checked as part of the same write rather than a separate read-then-write, is what actually closes the race.
Photo by Oktay Köseoğlu on Pexels
The check-then-act version, a SELECT followed by an INSERT in separate statements, looks correct in a single-threaded test and fails exactly the moment two workers happen to run it within milliseconds of each other, which is precisely the scenario that redelivery creates. Combining the check and the write into one atomic statement, using a unique constraint and catching the conflict, is the difference between a fix that works in theory and one that actually holds up under real concurrent retries.
Auditing your own handlers for this
Go through every job handler that has an external side effect and ask, specifically, what happens if this exact message arrives twice within a few seconds of each other. Emails get duplicated. Charges get captured twice. Webhooks fire twice to a partner system that might not deduplicate on their end either. Database writes that aren't naturally idempotent, incrementing a counter rather than setting a value, are a particularly easy trap to miss because the bug doesn't show up until the second delivery actually happens, which might be days after the handler shipped.
Handlers that are naturally idempotent, recalculating a value from source data, upserting a row keyed on a stable identifier, don't need any extra work. The audit is really about finding the subset of handlers that aren't naturally safe and adding an explicit guard to each one.
A quick heuristic that speeds this up: if a handler's core action is a SET or an UPSERT, it's probably already safe. If it's an INCREMENT, an APPEND, or anything that sends a request to an external system, it almost certainly needs an explicit guard, because none of those operations are naturally idempotent on their own.
What this means for queue design generally
We covered the durability side of this problem, keeping job state outside process memory so a crash doesn't lose work entirely, in our guide to building a job queue that survives a server restart. At-least-once delivery is the natural consequence of that durability, not a separate design choice, and treating it as an afterthought is how teams end up with a queue that's technically reliable and a set of handlers that quietly misbehave under load.
The Wikipedia entry on idempotence is a useful shorthand to hand to a team that hasn't internalized this yet: an operation is idempotent if applying it multiple times produces the same result as applying it once. That's the property every side-effecting handler in a queue needs, and it's worth stating explicitly in code review rather than assuming everyone already thinks about it that way.
Where we see this most often
This comes up constantly in the backend architecture reviews 137Foundry runs for client teams, almost always in a handler nobody thought to question because it had "always worked," meaning it had simply never been redelivered yet in a way anyone noticed. If your job system has been running for a while without an explicit idempotency audit, it's worth doing one before the first duplicate charge or duplicate email turns it into a support ticket instead of a code review comment.
Start with the handlers that would actually hurt if they ran twice. Everything else can wait.
Top comments (0)