Idempotent Jobs & Exactly-Once Effects: Designing Safe Retries

Durable execution, async LLM workflows and message-delivery debates keep showing up on DEV: MQTT QoS posts, local agent pipelines, tool-call cost control. The fancy orchestration only works if the boring layer underneath is honest about failure. Most SaaS queues deliver at least once. Your job will run again after a worker crash, a deploy, a lock timeout or a broker redelivery. If that job charges a card, sends a welcome email or decrements inventory, "just retry" becomes double billing. Part 1 of this series is the foundation every queue needs before you chase orchestration frameworks: idempotent jobs and exactly-once effects.

I wrote this with help from AI tools and checked every example and claim myself against Laravel queue behaviour and common BullMQ patterns. I build Loop Origin, where we design and ship Laravel and Node SaaS systems for clients, and this post opens a new series on reliable queues and async work.

📚 Series Roadmap — Reliable Queues & Async Work in SaaS

• Part 1 (This Post): Idempotent Jobs & Exactly-Once Effects: Designing Safe Retries
• Part 2: Queue Backpressure, Fairness & Tenant Isolation in Multi-Tenant SaaS
• Part 3: Dead Letter Queues, Poison Messages & Safe Replay Playbooks
• Part 4: Delayed Jobs, Schedulers & Clock Skew Gotchas
• Part 5: Observability for Async Work: Correlation IDs & Job Traces
• Part 6: Choosing Redis, SQS or Database Queues for Laravel and Node SaaS

Earlier Loop Origin series: Production MCP servers · Multi-tier caching & stampede protection · Usage-based billing metering

How do you make queue jobs safe under at-least-once delivery?

Direct answer: Assume every job can run more than once. Give each logical unit of work a stable idempotency key, record successful side effects in a dedupe store before or atomically with the effect, and make retries no-ops when that key is already completed. Exactly-once effects are designed; exactly-once delivery from the broker is not something you should bet the business on.

At-least-once is the default

Brokers and workers fail independently. A worker may finish Stripe, crash before acking the message, and another worker will process the same payload. SQS visibility timeouts, Redis requeues after SIGKILL, and database queue reserved_at reclaim all produce the same shape: duplicate execution of a job you thought was done.

That is fine for pure recomputation (rebuild a cache entry, regenerate a report file). It is not fine for:

• Sending email or SMS
• Charging or refunding via Stripe
• Mutating inventory or wallet balances
• Creating external tickets or webhooks your partners treat as unique events

Those need an exactly-once effect built on top of at-least-once delivery. The same discipline shows up in metering pipelines from the usage-based billing architecture post: count once, bill once.

Idempotency keys

An idempotency key names the business operation, not the queue attempt.

Good keys:

• invoice:{invoice_id}:finalize
• user:{user_id}:welcome_email:v1
• order:{order_id}:reserve_stock
• Stripe-style keys you already send as Idempotency-Key on the API call

Bad keys:

• Random UUIDs generated inside handle() (every retry looks new)
• The queue job UUID alone (re-dispatch creates a new one)
• Timestamps

Generate the key when the command is accepted (HTTP handler, domain service), store it on the job payload, and reuse it for every attempt.

Dedupe store patterns

You need a place that remembers "this key already completed" with clear semantics.

Approach	Pros	Cons	Best for
DB unique constraint on idempotency_key	Strong, transactional with business rows	Needs a table; watch long transactions	Payments, inventory
Redis SET key NX EX	Fast, simple TTL	Must choose TTL carefully; Redis loss risk	Emails, notifications
Provider idempotency (Stripe)	Effect-level guarantee at the vendor	Only covers that vendor call	Card charges
Outbox + unique event id	Ties to durable domain events	More moving parts	Cross-service workflows

A practical pattern for side effects:

1. Begin transaction / acquire lock for the key.
2. If status is completed, return success (no-op).
3. If status is in_progress and not stale, exit and let the other worker finish (or retry later).
4. Perform the side effect (or call Stripe with the same Idempotency-Key).
5. Mark completed and commit.

Order matters. Marking complete before the email sends can skip a real send after a crash. Marking complete only after a non-idempotent charge can double-charge if you crash between charge and write. Prefer vendor idempotency for money; for email, accept rare duplicates or use a provider-side dedupe id.

AI-generated image.

Laravel: unique jobs and explicit keys

Laravel can prevent overlapping duplicates while a job is on the queue, which is necessary but not sufficient for exactly-once effects:

use Illuminate\Contracts\Queue\ShouldBeUnique;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Mail;

class SendWelcomeEmail implements ShouldQueue, ShouldBeUnique
{
    use Dispatchable, InteractsWithQueue, Queueable;

    public function __construct(
        public int $userId,
        public string $idempotencyKey, // e.g. user:42:welcome_email:v1
    ) {}

    public function uniqueId(): string
    {
        return $this->idempotencyKey;
    }

    public function handle(): void
    {
        $claimed = DB::table('job_idempotency')->insertOrIgnore([
            'key' => $this->idempotencyKey,
            'status' => 'completed',
            'created_at' => now(),
            'updated_at' => now(),
        ]);

        if ($claimed === 0) {
            return; // already sent (or claimed)
        }

        $user = User::query()->findOrFail($this->userId);
        Mail::to($user)->send(new WelcomeMail($user));
    }
}

// At the HTTP edge — key is stable before dispatch
$key = "user:{$user->id}:welcome_email:v1";
SendWelcomeEmail::dispatch($user->id, $key);

ShouldBeUnique stops two welcome jobs sitting on the queue at once. The job_idempotency row stops a second send after the first succeeded and the unique lock released. For charges, call Stripe with the same key the job carries; Stripe will not create a second PaymentIntent for that Idempotency-Key.

If you are shaping Laravel services for production SaaS, the same boundaries show up in how we approach Laravel development work: commands accept idempotency keys at the edge, jobs never invent them.

Node / BullMQ equivalent

// producer
await welcomeQueue.add(
  'welcome',
  { userId, idempotencyKey: `user:${userId}:welcome_email:v1` },
  {
    jobId: `user:${userId}:welcome_email:v1`, // BullMQ dedupe by jobId
    attempts: 5,
    backoff: { type: 'exponential', delay: 2000 },
    removeOnComplete: true,
  }
);

// worker
welcomeQueue.process(async (job) => {
  const { userId, idempotencyKey } = job.data;
  const set = await redis.set(`idem:${idempotencyKey}`, '1', 'EX', 60 * 60 * 24 * 30, 'NX');
  if (set !== 'OK') return; // already completed

  await sendWelcomeEmail(userId);
});

jobId gives you broker-level dedupe for identical enqueues. The Redis SET NX gives you effect-level dedupe across retries and re-dispatches after the original job record is gone. Mirror money moves with Stripe's Idempotency-Key header using the same string.

Cache stampede locks from the Redis/Valkey singleflight patterns are cousins of this idea: one winner does the work, everyone else observes the result.

Retry strategies compared

Strategy	Behaviour	Use when	Risk if misused
No retry	Fail once	Non-critical, manually replayable	Lost work
Fixed delay	Retry every N seconds	Simple internal APIs	Thundering herd
Exponential backoff + jitter	Growing delays	Downstream 429/5xx	Still doubles effects without keys
Unique / jobId dedupe	One in-flight job per key	Duplicate enqueues	Does not cover post-success redelivery alone
Effect idempotency key	Side effect runs once	Email, payments, stock	Key design bugs

Retries without idempotency keys multiply damage. Keys without retries leave users stuck on transient blips. You want both.

For broader product work that spans Laravel and Node services, the same queue contract applies across custom web application builds: define the key at command acceptance, not inside the worker.

Diagnostic checklist

1. Can every critical job run twice without corrupting money, stock or messaging?
2. Is the idempotency key created before dispatch and stored on the payload?
3. Do you avoid random keys inside handle() / process callbacks?
4. Is there a dedupe record (DB unique, Redis NX, or vendor key) for each side effect?
5. Are Stripe (or similar) calls passing a stable Idempotency-Key?
6. Does uniqueness middleware cover in-flight duplicates and a completed-state check?
7. Are TTLs on Redis dedupe keys longer than any realistic late redelivery?
8. Do poison retries go somewhere observable instead of silent infinite loops? (Part 3)
9. Can you explain in one sentence what happens after a crash between side effect and ack?
10. Have you load-tested duplicate delivery (replay the same payload) in staging?

FAQ

Is exactly-once delivery possible?

In general distributed systems, end-to-end exactly-once delivery is fragile. Design for at-least-once delivery plus idempotent handlers. That combination gives you exactly-once effects where it matters.

Should every job be idempotent?

Every job with an external side effect should be. Pure recomputation can be naturally idempotent if it overwrites the same artifact. Still guard expensive LLM or vendor calls; retries without caches burn money, which is why agent and tool-cost threads keep trending.

What TTL should Redis dedupe keys use?

Longer than your maximum job retention plus any manual replay window. Thirty days is a common starting point for emails; financial keys often live in Postgres forever.

Does Laravel ShouldBeUnique replace a dedupe table?

No. It mostly prevents concurrent duplicates while a job is queued or running. After success, a re-dispatch with a new attempt still needs a completed-state check.

How do I test this?

In staging, dispatch the same job payload twice, kill a worker mid-handler, and replay DLQ messages. Assert one email, one charge, one inventory row.

What about MCP and agent tool calls?

Treat each tool invocation that mutates state like a job side effect: stable ids, dedupe, and budgets. The production MCP server patterns for approval gates pair naturally with idempotent execution.

Next in this series, Part 2 covers queue backpressure, fairness and tenant isolation so one noisy customer cannot starve everyone else's workers.

---

Follow & Connect

• Website: Loop Origin
• LinkedIn: Loop Origin
• X: Muhammad Ali Mughal (@AliMughal887)
• GitHub: Loop Origin (@LoopOrigin)