Running Laravel Queues in Production: Redis, Supervisor, Retries and Failed Jobs

Laravel queues feel easy in development. You set QUEUE_CONNECTION, run php artisan queue:work in a terminal, and emails send, PDFs generate and webhooks fire in the background. Then you deploy, close the terminal, and a week later a customer asks why their order confirmation never arrived.

I'm Muhammad Ali Mughal, and at Loop Origin, our Karachi team builds and maintains Laravel applications, including e-commerce stores that rely on queues for order emails, courier bookings, SMS and payment callbacks. This is the production setup we use, and the mistakes it's designed to prevent.

1. Pick a real queue driver

The sync driver runs jobs immediately inside the request, which is fine for tests but defeats the point in production. The database driver works and is a reasonable start for small apps. For anything busy, I use Redis: it's fast, it's what Laravel Horizon is built on, and it keeps queue traffic off your main database.

In .env:

QUEUE_CONNECTION=redis
REDIS_HOST=127.0.0.1
REDIS_PORT=6379

If Redis also holds your cache and sessions, consider a separate Redis database or connection for queues, so clearing the cache never wipes pending jobs.

2. Split work into named queues

Not all jobs are equally urgent. A password reset email shouldn't wait behind a 5,000-row export. Dispatch onto named queues:

SendOrderConfirmation::dispatch($order)->onQueue('high');
GenerateMonthlyReport::dispatch($month)->onQueue('low');

Then tell the worker to process them in priority order:

php artisan queue:work redis --queue=high,default,low

The worker always empties high before looking at default, and default before low.

3. Keep workers alive with Supervisor

queue:work is a long-running process. If it crashes, runs out of memory or the server reboots, nothing restarts it unless you have a process monitor. On Ubuntu, Supervisor is the standard choice:

sudo apt install supervisor

Create /etc/supervisor/conf.d/laravel-worker.conf:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work redis --queue=high,default,low --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/app/storage/logs/worker.log
stopwaitsecs=3600

Load it:

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start "laravel-worker:*"

A few notes on those options:

• numprocs is how many workers run in parallel. Start with two and increase only if the queue backs up; each worker holds your whole app in memory.
• --max-time=3600 makes each worker exit after an hour so Supervisor starts a fresh one. This limits slow memory leaks. --max-jobs does the same by job count, and --memory sets a memory ceiling.
• stopwaitsecs should be longer than your longest job, so Supervisor doesn't kill a worker mid-job during a stop.
• user should match the user that owns your app files, or you'll get permission errors on logs and storage.

4. Restart workers on every deploy

This one catches almost everyone. A running worker has already loaded your code into memory. Deploy new code and the workers keep running the old version until they restart.

Add this to the end of your deploy script:

php artisan queue:restart

It signals workers to finish their current job and exit gracefully; Supervisor then starts them again with the new code. Clear and rebuild config cache first (php artisan config:cache) so the new workers pick up the right settings.

5. Set timeouts and retries deliberately

Three settings work together, and getting them out of sync causes jobs to run twice:

• Job $timeout: the maximum seconds a job may run before the worker kills it.
• retry_after in config/queue.php: how long the queue waits before assuming a job died and handing it to another worker.
• $tries / $backoff: how many attempts, and how long to wait between them.

Rule: retry_after must be longer than your longest job timeout. If a job legitimately takes 120 seconds and retry_after is 90, a second worker picks it up while the first is still running.

On the job class:

class BookCourierShipment implements ShouldQueue
{
    use Queueable;

    public int $tries = 5;
    public int $timeout = 60;

    public function __construct(public Order $order) {}

    public function backoff(): array
    {
        return [10, 60, 300]; // seconds before retries 1, 2, 3+
    }

    public function handle(CourierClient $courier): void
    {
        $courier->book($this->order);
    }
}

Increasing backoff matters for third-party APIs. If a courier or SMS gateway is down, hammering it every three seconds won't help, and may get you rate-limited.

6. Handle failed jobs instead of losing them

When a job runs out of attempts, Laravel records it in the failed_jobs table (recent Laravel versions include that migration by default). Learn these commands:

php artisan queue:failed            # list failed jobs
php artisan queue:retry <uuid>      # retry one
php artisan queue:retry all         # retry all
php artisan queue:forget <uuid>     # delete one
php artisan queue:prune-failed --hours=168

Add a failed() method to important jobs so someone actually finds out:

public function failed(?Throwable $e): void
{
    Log::error('Courier booking failed', [
        'order' => $this->order->id,
        'error' => $e?->getMessage(),
    ]);
    // notify the ops team by email or Slack
}

Schedule the prune command so the table doesn't grow forever.

7. Make jobs safe to run twice

Even with correct settings, a job can occasionally run more than once: a worker crashes after doing the work but before acknowledging it, or someone retries a failed job that half-succeeded. Design for it:

• Check before acting. Before booking a shipment, check whether the order already has a tracking number.
• Use unique jobs. Implement ShouldBeUnique to stop duplicate jobs for the same model from being queued at the same time.
• Prevent overlap. The WithoutOverlapping job middleware stops two jobs for the same key from running simultaneously.
• Dispatch after commit. If you dispatch inside a database transaction, use ->afterCommit() (or set after_commit on the connection) so the worker doesn't look for a record that isn't saved yet.

This matters most for payment callbacks and inventory updates, where a duplicate means a double charge or a wrong stock count.

8. Watch the queue

You can't fix what you can't see. Options, from simplest:

• Tail storage/logs/worker.log and check supervisorctl status after deploys.
• Schedule php artisan queue:monitor redis:high,redis:default --max=100 so Laravel fires an event when a queue grows past a threshold, and send yourself an alert from it.
• Install Laravel Horizon if you're on Redis. It gives you a dashboard of throughput, wait times and failed jobs, and manages worker processes from a config file. With Horizon, Supervisor runs php artisan horizon instead of queue:work, and you restart with php artisan horizon:terminate on deploy.

Production checklist

• Redis (or database) driver, never sync
• Named queues with priority order
• Supervisor running workers with --max-time or --max-jobs
• queue:restart (or horizon:terminate) in every deploy
• retry_after longer than the longest job $timeout
• $tries and increasing backoff on jobs that call external APIs
• failed() handlers and a scheduled prune
• Idempotent jobs for payments, stock and shipments
• Queue-length alerts

If you'd like a team to build or review this for your app, our Laravel development services cover queued jobs, integrations and deployment. You can see our other work at looporigin.com.

What's the worst queue bug you've hit in production? Tell me in the comments.

laravel #php #redis #webdev #backend #devops #queues #supervisor #ubuntu #webdevelopment #pakistan #karachi

Comments

[Muhammad Ali Mughal]

Great callout, Eusebiu! That distinction between ShouldBeUnique and ShouldBeUniqueUntilProcessing trips teams up all the time when a job dispatches the next batch or re-queues itself before handle() returns. Using ShouldBeUniqueUntilProcessing (or releasing the lock right as execution starts) avoids silently dropping that follow-up dispatch. Appreciate you sharing that!

[Abhinav Sharma]

The separate Redis database for queues is the detail that quietly saves people: one cache flush on a shared instance and every pending job disappears without an error anywhere. Splitting work by named queues pays off the first time a callback spike should not delay order emails.

[Muhammad Ali Mughal]

Thanks, Abhinav! That silent part is what makes it nasty: nothing fails, nothing lands in failed_jobs, the jobs are just gone. Named queues also give you cheap priority, since running the worker with --queue=high,default means order emails always get picked up first, even in the middle of a callback spike.

[Eusebiu Balan]

One catch with ShouldBeUnique that cost me a live send: it holds the lock until the job finishes, not until it starts.

I had a job that re-runs an email send command, and the command queues the next wave plus the next copy of that job, because the managed queue refused long delays. With ShouldBeUnique the next copy couldn't be queued while the current one still held the lock, so the chain stopped after the first wave. ShouldBeUniqueUntilProcessing releases it when processing begins, and that's what a self-continuing job needs.

[Sanad Kit]

Good rundown. One addition on the timeout and retry section: the worker's own --timeout flag (default 60 seconds) is separate from the job's $timeout property, and the job's property wins when both are set. The ordering to aim for is job timeout shorter than worker --timeout, and worker --timeout shorter than retry_after. Also remember that timeouts are enforced through the pcntl extension, so a worker without it will not kill a stuck job and retry_after will hand the job to a second worker anyway. Two smaller details: a $tries on the job class overrides the --tries=3 in your Supervisor command, and if you dispatch from inside a database transaction, add ->afterCommit() (or set after_commit to true on the connection) so the worker never picks up a job before the row it needs is committed.

[Muhammad Ali Mughal]

Thanks for adding these, all spot on. The ordering is the part I'd underline: job $timeout < worker --timeout < retry_after. Get that wrong and a slow job gets handed to a second worker while the first is still running it, which is how you end up with duplicate side effects. The pcntl point is easy to miss in slim Docker images, so it's worth running php -m | grep pcntl inside the worker container. And afterCommit() is a great catch; if you dispatch from inside transactions a lot, setting after_commit to true on the queue connection is the safer default.