The Reconciler Pattern: When a Queued Job Simply Never Runs

TL;DR

• A job dispatched once, at row-creation time, has a silent failure mode: if that dispatch is lost, the row sits pending forever — and never appears in failed_jobs.
• Retries only exist for jobs that ran. A job that never entered the queue has nothing to retry.
• Fix: a reconciler — a scheduled sweep for stale, unclaimed rows that re-dispatches them. The database is the source of truth; the queue is just delivery.

The bug that leaves no trace

Records stuck in pending. No error, no failed job, no log line. The shape of the code:

$status = SyncStatus::create([...]);   // sync_status = 'pending'
ProcessSyncStatus::dispatch($status->id);

One dispatch, at creation. Fine 99% of the time. The 1%:

What happened	Where the job went	What you see
queue:clear during an incident	Payload deleted	Stuck pending, no error
No worker running at dispatch	Flushed	Stuck pending, no error
Redeploy dropped the connection mid-enqueue	Never enqueued	Stuck pending, no error
Job ran and threw	failed_jobs	An error you can act on

Only the last row is visible. The first three are invisible failures — dashboards green, data quietly rotting.

The bad mental model: "the queue is durable, so a dispatched job will eventually run." Your database row is the durable thing. The queue is a delivery hint.

Retry ≠ reconcile

	Retry	Reconcile
Triggered by	A job that ran and failed	A state that looks wrong
Source of truth	The queue / failed_jobs	The database
Catches lost dispatches?	No	Yes

Retry is a queue-level mechanism. Reconciliation is a domain-level one. You need both.

The reconciler

Periodically ask the database "does anything look stuck?", and fix it.

  every 15 min
       |
       v
+----------------------------+
|  sweep pending rows        |
|  updated_at < now()-15m    |  <- stale
|  AND claimed_at IS NULL    |  <- never picked up by a worker
+----------------------------+
       | touch(row)   <- so the next tick won't sweep it again
       v
   re-dispatch ProcessSyncStatus

Two predicates carry the weight. Stale: updated_at older than a threshold — fresh pending rows are just working, give the queue time. Unclaimed: the job stamps the row the moment a worker picks it up, so a pending-and-claimed row is in flight right now; leave it alone. Pending and unclaimed after 15 minutes was never delivered.

final class ReconcilePendingSync extends Command
{
    protected $signature = 'sync:reconcile-pending
                            {--minutes=15 : Only sweep rows untouched this long}
                            {--limit=200  : Cap re-dispatches per run}
                            {--dry-run}';

    public function handle(): int
    {
        $rows = SyncStatus::query()
            ->where('sync_status', 'pending')
            ->whereNull('claimed_at')
            ->where('updated_at', '<', now()->subMinutes((int) $this->option('minutes')))
            ->limit((int) $this->option('limit'))
            ->get();

        foreach ($rows as $row) {
            if ($this->option('dry-run')) { continue; }   // ...report only

            $row->touch();                                // don't sweep again next tick
            ProcessSyncStatus::dispatch($row->id);
            Log::info('reconciler: re-dispatched', ['id' => $row->id]);
        }

        return self::SUCCESS;
    }
}

Three things that are easy to get wrong:

• touch() before dispatch. Otherwise the next tick sees the same stale row and dispatches it again — a job multiplier, not a reconciler.
• --limit. If something breaks for a day, the first sweep shouldn't dump 40,000 jobs into Redis.
• Idempotent job. Reconciliation will occasionally re-run something that was fine. That must be a no-op: claim the row at the top of handle(), make the work an upsert.

Tests

Queue::fake() plus a backdated updated_at covers the cases that matter:

function agePendingRow(SyncStatus $row, int $minutes): void
{
    // Bypass Eloquent timestamps to backdate updated_at.
    DB::table('sync_status')->where('id', $row->id)
        ->update(['updated_at' => now()->subMinutes($minutes)]);
}

it('re-dispatches a stale unclaimed pending row', function () {
    Queue::fake();
    $stale = SyncStatus::factory()->create(['sync_status' => 'pending', 'claimed_at' => null]);
    agePendingRow($stale, 60);

    $this->artisan('sync:reconcile-pending', ['--minutes' => 15])->assertSuccessful();

    Queue::assertPushed(ProcessSyncStatus::class, 1);
});

it('skips a pending row already claimed by a worker', function () {
    Queue::fake();
    $claimed = SyncStatus::factory()->create(['sync_status' => 'pending', 'claimed_at' => now()]);
    agePendingRow($claimed, 60);

    $this->artisan('sync:reconcile-pending', ['--minutes' => 15])->assertSuccessful();

    Queue::assertNotPushed(ProcessSyncStatus::class);
});

Plus a third: a fresh pending row must be left alone. Anyone can write a sweeper that re-dispatches everything — the value is in what it refuses to touch.

Takeaway

If a database row implies "a job should be running for this", something needs to check that claim on a schedule. Dispatch-on-create is a happy-path optimisation, not a guarantee.

The rule I'm keeping: the queue delivers, the database decides. Any state that can only be advanced by a queued job needs a reconciler behind it — otherwise "eventually consistent" really means "usually consistent, and silent when it isn't."