Laravel Queue Workers: Processing Background Jobs at Scale

You're three months post-launch and your app is gaining traction. Then the complaints start rolling in: "Why does sending an invoice take 8 seconds?" The culprit is almost always the same — synchronous execution of work that should be deferred. Laravel's queue system exists precisely to solve this, but running it in production at scale is a different beast from getting it working locally. This article digs into the real mechanics of queue workers, job design, scaling strategies, and the failure modes that will bite you if you're not prepared.

How Laravel Queues Actually Work

Before optimizing, it's worth understanding the pipeline. When you dispatch a job, Laravel serializes it and writes a record to your configured queue driver (database, Redis, SQS, etc.). A queue worker is a long-running PHP process that polls that driver, deserializes the job, executes it, and acknowledges completion — or handles failure.

php artisan queue:work redis --queue=high,default --tries=3 --timeout=60

The key insight: queue:work keeps the application bootstrapped in memory between jobs. This is fast, but it means code changes require a worker restart. This trips up nearly every developer the first time they deploy a fix and wonder why their jobs are still running old logic.

queue:listen, by contrast, re-bootstraps for each job — safe but slow. Use it in development; never in production.

Structuring Jobs for Reliability

A job that fails halfway through can cause real damage if it re-runs and repeats destructive side effects. Design jobs to be idempotent — running them twice should produce the same outcome as running them once.

class ProcessSubscriptionRenewal implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public int $tries = 3;
    public int $backoff = 60; // seconds between retries

    public function __construct(private readonly Subscription $subscription) {}

    public function handle(BillingService $billing): void
    {
        // Guard against double-processing
        if ($this->subscription->renewed_at?->isToday()) {
            return;
        }

        $billing->renew($this->subscription);
    }

    public function failed(Throwable $exception): void
    {
        Log::error('Subscription renewal failed', [
            'subscription_id' => $this->subscription->id,
            'error' => $exception->getMessage(),
        ]);

        $this->subscription->owner->notify(new RenewalFailedNotification());
    }
}

The failed() method is your safety net. Always implement it for jobs with business consequences.

Choosing the Right Queue Driver

Database driver works fine for low-volume applications and requires zero additional infrastructure. The downside: polling creates database load, and high-frequency jobs can hammer your jobs table.

Redis is the sweet spot for most production applications. It's fast, supports atomic operations (eliminating race conditions), and Laravel Horizon gives you a beautiful dashboard on top of it.

Amazon SQS makes sense when you're already in the AWS ecosystem and want managed infrastructure with automatic scaling and dead-letter queues baked in.

For most Laravel shops, Redis + Horizon is the answer.

Laravel Horizon: The Missing Dashboard

If you're running Redis queues without Horizon, you're flying blind. Horizon gives you real-time throughput metrics, per-queue balancing, failed job management, and — critically — a configuration-as-code approach to worker pools.

// config/horizon.php
'environments' => [
    'production' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['critical', 'high', 'default', 'low'],
            'balance' => 'auto',
            'minProcesses' => 2,
            'maxProcesses' => 20,
            'balanceMaxShift' => 5,
            'balanceCooldown' => 3,
            'tries' => 3,
            'timeout' => 90,
        ],
    ],
],

The auto balance strategy is powerful: Horizon monitors queue depth and dynamically scales worker count between minProcesses and maxProcesses. During a spike — say, a bulk email campaign fires — workers scale up automatically and wind back down once the queue clears.

Prioritizing Work with Multiple Queues

Not all jobs are equal. A password reset email and a low-priority analytics event shouldn't compete for the same worker. Define explicit queues and dispatch accordingly:

// Dispatch to named queues
PasswordResetEmail::dispatch($user)->onQueue('critical');
AnalyticsEvent::dispatch($data)->onQueue('low');

When starting a worker, list queues in priority order:

php artisan queue:work redis --queue=critical,high,default,low

The worker exhausts higher-priority queues before touching lower ones. Be careful with this pattern though — a flood of critical jobs can starve low indefinitely. Consider dedicated workers per tier in high-throughput environments.

Handling Timeouts and Memory Limits

Two flags that developers often ignore until they cause a production incident:

• --timeout: Kills a job that runs longer than N seconds. Set it slightly below your job's expected maximum runtime. A PDF generation job might need --timeout=120; a simple notification should finish in under 5.
• --memory: Restarts the worker when memory usage exceeds N megabytes. PHP processes leak memory over time, especially with ORM-heavy jobs. Setting --memory=256 is a reasonable baseline.

php artisan queue:work redis --timeout=60 --memory=256 --sleep=1

--sleep=1 controls how long the worker pauses (in seconds) when no jobs are available. Lower values reduce latency but increase polling overhead.

Keeping Workers Alive: Supervisor Configuration

Workers die — from memory limits, crashes, or deploys. Supervisor is the standard tool to keep them running.

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/artisan queue:work redis --queue=critical,high,default --sleep=1 --tries=3 --timeout=90 --memory=256
autostart=true
autorestart=true
numprocs=4
stopasgroup=true
killasgroup=true
user=www-data
redirect_stderr=true
stdout_logfile=/var/log/worker.log

numprocs=4 spins up four parallel worker processes. Adjust this based on your server's CPU count and job characteristics — CPU-bound jobs benefit from more processes; I/O-bound jobs (API calls, email sends) can run many more per core.

Graceful Deploys with Zero Job Loss

This is where teams cut corners and pay for it. When you deploy new code, workers need to restart — but not mid-job.

php artisan queue:restart

This sets a flag in cache. Workers finish their current job, check the flag, and exit gracefully. Supervisor then respawns them with fresh code. Wire this into your deployment script (Envoyer, GitHub Actions, Forge deploy hooks — wherever your deploys live).

Pair this with ShouldBeUniqueUntilProcessing on jobs that should never queue duplicates:

class SyncUserProfile implements ShouldQueue, ShouldBeUniqueUntilProcessing
{
    public function uniqueId(): string
    {
        return $this->user->id;
    }
}

Monitoring in Production

Horizon's dashboard is great for development and internal monitoring, but for production alerting you need to go further. At https://hanzweb.ae, working across multiple high-traffic Laravel deployments, we wire Horizon metrics into Prometheus/Grafana and set alerts on failed job rates and queue depth thresholds — a queue depth spiking above 500 on the critical channel at 2AM is worth waking someone up.

Also implement regular pruning to prevent your failed_jobs table from becoming a graveyard:

# In your scheduler
$schedule->command('queue:prune-failed --hours=168')->daily();

Conclusion

Laravel's queue system is one of its most powerful features — but it rewards developers who understand the failure modes, not just the happy path. Design idempotent jobs, choose your driver deliberately, use Horizon for visibility, configure Supervisor for resilience, and always hook queue:restart into your deploy pipeline. These aren't advanced techniques — they're table stakes for any application where background jobs matter. Get this foundation right, and you can scale your worker fleet horizontally with confidence when the traffic comes.