Retry-Safe Queue Jobs in Laravel: ShouldBeUnique, WithoutOverlapping and Idempotency Keys

A queue job that runs twice is not a rare edge case. It is the normal behaviour of every queue system you will ever use. Workers get killed during a deploy, a job times out halfway through an HTTP call, a user double-clicks "Send invoice", a scheduler fires while the previous run is still going. Laravel's queue will happily retry for you. The question is whether your job is safe to retry.

In business software the cost of getting this wrong is very concrete: two invoices sent to the same customer, stock decremented twice, a supplier receiving the same purchase order on two emails. This post walks through the toolkit I use to make Laravel jobs retry-safe: dispatching after commit, ShouldBeUnique, the WithoutOverlapping middleware, and an idempotency key that survives crashes.

The problem: "at least once" means "sometimes twice"

Consider a typical job: when an invoice is finalized, send it to an external e-invoicing provider and store the provider's reference.

// The naive version
class SendInvoiceToProvider implements ShouldQueue
{
    use Queueable;

    public function __construct(public Invoice $invoice) {}

    public function handle(): void
    {
        $response = Http::post('https://provider.example/api/invoices', [
            'number' => $this->invoice->number,
            'total' => $this->invoice->total_minor,
        ])->throw();

        $this->invoice->update(['provider_reference' => $response->json('id')]);
    }
}

There are at least four ways this sends the same invoice twice:

1. Dispatched twice. Two requests (a double-click, a retried HTTP request, an observer and a controller both dispatching) put two copies on the queue.
2. Processed concurrently. Two workers pick up two copies at the same time.
3. Retried after a partial success. The provider accepted the invoice, then the worker died (timeout, OOM, deploy) before update() ran. The retry posts again.
4. Dispatched before commit. The job is queued inside a database transaction, a worker picks it up before the commit, and it sees stale or missing data. The transaction rolls back, but the job has already run.

Each tool below closes one of these gaps. None of them closes all four alone.

Step 1: Dispatch after the transaction commits

If you dispatch inside DB::transaction(), a fast worker can process the job before the data it depends on is committed. Laravel lets you defer the dispatch until all open transactions have committed:

use Illuminate\Support\Facades\DB;

DB::transaction(function () use ($invoice) {
    $invoice->finalize();   // writes rows inside the transaction

    SendInvoiceToProvider::dispatch($invoice)->afterCommit();
});

You can also set after_commit to true on the queue connection in config/queue.php to make this the default. If the transaction rolls back, the job is never dispatched. That alone removes a whole class of "the job ran on data that never existed" bugs.

Step 2: Deduplicate dispatches with ShouldBeUnique

Implementing ShouldBeUnique makes Laravel acquire a cache lock keyed by uniqueId() when the job is dispatched. If the lock is already held, the new dispatch is silently ignored. The lock is released when the job completes or fails all its retry attempts.

namespace App\Jobs;

use App\Models\Invoice;
use Illuminate\Contracts\Queue\ShouldBeUnique;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;

class SendInvoiceToProvider implements ShouldQueue, ShouldBeUnique
{
    use Queueable;

    /** The number of times the job may be attempted. */
    public $tries = 5;

    /** Release the unique lock after 10 minutes even if the job never finished. */
    public $uniqueFor = 600;

    public function __construct(public Invoice $invoice) {}

    public function uniqueId(): string
    {
        return (string) $this->invoice->id;
    }

    // handle() comes in step 4
}

Two things to know:

• Unique jobs need a cache driver that supports atomic locks (redis, memcached, dynamodb, database, file or array), and every server that dispatches must share the same cache. A file cache on three web servers gives you three separate locks.
• If you want the lock released as soon as processing starts (so a new dispatch can queue up while the current one runs), implement ShouldBeUniqueUntilProcessing instead.

On Laravel 13 the documentation shows the same options as attributes, for example #[Tries(5)] and #[UniqueFor(600)] on the class. Use whichever style your version's docs show; the behaviour is the same.

Step 3: Serialize processing with WithoutOverlapping

ShouldBeUnique guards the dispatch. It does not help when two copies are already on the queue (for example, one was dispatched before you added the interface, or the unique lock expired). For that, use the WithoutOverlapping job middleware, which holds a lock while the job is running:

use Illuminate\Queue\Middleware\WithoutOverlapping;

/**
 * Get the middleware the job should pass through.
 *
 * @return array<int, object>
 */
public function middleware(): array
{
    return [
        (new WithoutOverlapping((string) $this->invoice->id))
            ->releaseAfter(30)   // an overlapping copy is retried in 30 seconds
            ->expireAfter(120),  // the lock expires even if the worker crashes
    ];
}

By default, an overlapping copy is released back onto the queue, and each release counts as an attempt. If the duplicate is useless, call dontRelease() to delete it instead. If two different job classes must not touch the same invoice at once, add ->shared() so the key applies across classes.

Set expireAfter() comfortably above the job's expected runtime. If it is shorter, a slow job loses its lock mid-run and a second copy can start.

Step 4: Make the side effect idempotent

Locks reduce concurrency. They do not fix gap number 3: the provider accepted the invoice, the worker died before saving the reference, and the retry posts again. For that, the job needs to recognise its own previous work. Two layers:

Layer A: check your own state first. If the reference is already stored, there is nothing to do.

Layer B: send a stable idempotency key. Many payment and invoicing APIs accept an Idempotency-Key (or similar) header and return the original result when they see the same key again. Derive it from your data, never from Str::uuid() inside handle(), because a new UUID per attempt defeats the purpose.

use Illuminate\Support\Facades\Http;

public function handle(): void
{
    $invoice = $this->invoice->fresh();

    // Layer A: already done on a previous attempt.
    if ($invoice->provider_reference !== null) {
        return;
    }

    // Layer B: same key on every attempt for this invoice.
    $key = "invoice-{$invoice->id}";

    $response = Http::withHeaders(['Idempotency-Key' => $key])
        ->timeout(15)
        ->post(config('services.einvoice.url').'/invoices', [
            'number' => $invoice->number,
            'total' => $invoice->total_minor,
            'currency' => $invoice->currency,
        ])
        ->throw();

    $invoice->update(['provider_reference' => $response->json('id')]);
}

fresh() matters: the serialized model in the payload is a snapshot from dispatch time, and a previous attempt may have updated the row since.

When the provider has no idempotency support

Then you need your own record of what was sent. A small table with a unique key turns "did I already do this?" into a database constraint instead of a hope:

// Migration excerpt
Schema::create('outbound_requests', function (Blueprint $table) {
    $table->id();
    $table->string('idempotency_key')->unique();
    $table->string('status');            // pending, sent
    $table->string('external_id')->nullable();
    $table->timestamps();
});

// Claim the key. Only one attempt can ever insert it.
$claimed = DB::table('outbound_requests')->insertOrIgnore([
    'idempotency_key' => $key,
    'status' => 'pending',
    'created_at' => now(),
    'updated_at' => now(),
]);

if ($claimed === 0) {
    // Someone already claimed it. If it is still "pending", a previous attempt
    // may have crashed mid-call: reconcile with the provider instead of re-sending.
    return;
}

Be honest about the limit here: without provider support there is always a small window between "the remote side accepted" and "I wrote that down". The unique key guarantees you never send twice blindly; the pending rows are the list you reconcile (for example, by looking the invoice up on the provider by its number) before sending again.

Here is the same claim logic as a runnable plain-PHP script with SQLite, simulating a worker that crashes after the remote call and a retry that follows:

<?php
// idempotency_demo.php: run with `php idempotency_demo.php`
$db = new PDO('sqlite::memory:');
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$db->exec('CREATE TABLE outbound_requests (
    idempotency_key TEXT PRIMARY KEY,
    status TEXT NOT NULL
)');

$remoteCalls = 0;

function attempt(PDO $db, string $key, int &$remoteCalls, bool $crashAfterCall): string
{
    $stmt = $db->prepare('INSERT OR IGNORE INTO outbound_requests (idempotency_key, status) VALUES (?, ?)');
    $stmt->execute([$key, 'pending']);

    if ($stmt->rowCount() === 0) {
        $status = $db->query("SELECT status FROM outbound_requests WHERE idempotency_key = " . $db->quote($key))
            ->fetchColumn();
        return "skipped (already claimed, status: {$status})";
    }

    $remoteCalls++; // the side effect: the HTTP call to the provider

    if ($crashAfterCall) {
        return 'crashed after remote call';
    }

    $db->prepare('UPDATE outbound_requests SET status = ? WHERE idempotency_key = ?')
        ->execute(['sent', $key]);
    return 'sent';
}

echo 'Attempt 1: ', attempt($db, 'invoice-42', $remoteCalls, true), PHP_EOL;
echo 'Attempt 2: ', attempt($db, 'invoice-42', $remoteCalls, false), PHP_EOL;
echo 'Attempt 3: ', attempt($db, 'invoice-43', $remoteCalls, false), PHP_EOL;
echo "Remote calls: {$remoteCalls}", PHP_EOL;

Output:

Attempt 1: crashed after remote call
Attempt 2: skipped (already claimed, status: pending)
Attempt 3: sent
Remote calls: 2

Invoice 42 was sent exactly once despite the retry, and its pending status tells you it needs reconciliation rather than a second send.

Common pitfalls

• Treating ShouldBeUnique as "exactly once". It only deduplicates dispatches while the lock is held. Retries of the same job, expired locks and copies already on the queue are not covered.
• Unique jobs inside batches. The documentation is explicit: unique job constraints do not apply to jobs within batches.
• A cache that isn't shared. Locks live in the cache. Per-server file or array caches mean per-server locks, which means no locks at all in a multi-server setup.
• expireAfter() shorter than the job. The lock disappears mid-run and a duplicate starts. Measure your slow cases and leave margin.
• Releases eat your attempts. Every WithoutOverlapping release counts as an attempt. With $tries = 3 and a busy key, a job can fail without ever running handle(). Raise $tries, use dontRelease(), or switch to retryUntil() with a time window.
• Random idempotency keys. A key generated inside handle() changes on every attempt. Derive it from stable data (invoice ID, plus a version if the invoice can be legitimately re-sent after a correction).
• Trusting the serialized model. Call fresh() (or re-query) before deciding whether work is already done.
• Dispatching inside transactions. Use afterCommit() or the after_commit connection option, or a rolled-back invoice can still be sent.

Summary

Retry-safety in Laravel is layered, and each layer has a different job:

1. afterCommit() so the job never runs on uncommitted data.
2. ShouldBeUnique to drop duplicate dispatches.
3. WithoutOverlapping so two workers never process the same record at once.
4. An idempotency check plus a stable key so a retry after a partial success recognises its own work.

The first three are a few lines of configuration. The fourth is the one that actually protects your customers from a second invoice, and it is the one most codebases skip.

Which side effects in your app would hurt the most if a job ran twice, and how are you guarding them today?