DEV Community

Cover image for Scaling Laravel Queues: A Practical Guide to AWS SQS
Noni Gopal Sutradhar Rinku
Noni Gopal Sutradhar Rinku

Posted on

Scaling Laravel Queues: A Practical Guide to AWS SQS

#laravel #redis #aws #devops

When your Laravel application moves from MVP to a production system handling thousands of jobs per hour, the default database or Redis queue can quickly become a bottleneck. The solution for high availability and elastic scaling is almost always AWS SQS (Simple Queue Service).

Here's how to integrate SQS effectively to handle burst traffic and improve your application's reliability.

1. Why SQS Over Redis or Database Queues?

Feature Redis/Database AWS SQS
Durability Data loss possible if the server crashes. High durability, messages are replicated across multiple availability zones.
Elasticity Scaling requires scaling up the host server. Automatically scales to handle millions of messages per second.
Visibility Timeout Basic control, sometimes buggy. Robust control, ensuring workers don't process the same job simultaneously.

2. Setting up AWS IAM and SQS

Before touching Laravel, you need a queue in AWS and credentials to access it.

Create an SQS Queue: Choose a Standard Queue. Name it something logical (e.g., production-default-queue).

IAM User: Create a dedicated IAM user with the minimal necessary policy (Least Privilege Principle). This user needs:

sqs:SendMessage

sqs:ReceiveMessage

sqs:DeleteMessage

sqs:GetQueueAttributes
Enter fullscreen mode Exit fullscreen mode

Credentials: Get the Access Key ID and Secret Access Key for this IAM user.

3. Laravel Configuration

Laravel makes the switch simple in your environment files.

Update your .env file with the SQS credentials and configuration:

# .env File Changes
QUEUE_CONNECTION=sqs

AWS_ACCESS_KEY_ID="your_iam_access_key"
AWS_SECRET_ACCESS_KEY="your_iam_secret_key"
AWS_DEFAULT_REGION="your_aws_region" # e.g., ap-southeast-2
AWS_QUEUE="[https://sqs.ap-southeast-2.amazonaws.com/123456789012/production-default-queue](https://sqs.ap-southeast-2.amazonaws.com/123456789012/production-default-queue)" 
AWS_QUEUE_PREFIX="${AWS_QUEUE}/" # SQS uses the full URL as the queue name
Enter fullscreen mode Exit fullscreen mode

Next, ensure your config/queue.php file is correctly configured under the 'sqs' connection:

//config/queue.php
'sqs' => [
    'driver' => 'sqs',
    'key' => env('AWS_ACCESS_KEY_ID'),
    'secret' => env('AWS_SECRET_ACCESS_KEY'),
    'prefix' => env('AWS_QUEUE_PREFIX', '[https://sqs.us-east-1.amazonaws.com/](https://sqs.us-east-1.amazonaws.com/)'), // Ensure this is set!
    'queue' => env('AWS_QUEUE', 'default'),
    'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
]
Enter fullscreen mode Exit fullscreen mode

4. Tuning Workers and Visibility

This is where performance is won or lost. SQS uses a Visibility Timeout—the time during which a message is hidden from other workers after being picked up.

Queue Setting (AWS Console): Set the SQS queue's Visibility Timeout to match the maximum runtime of your longest job (e.g., 300 seconds for 5 minutes).

Worker Timeout (Supervisor/Horizon): Set your worker process timeout to match, or slightly exceed, the SQS timeout. If the worker crashes, SQS will re-release the message after the timeout.

Running the Workers

If you use Supervisor to manage your workers, ensure the SQS connection is specified:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/artisan queue:work sqs --daemon --tries=3 --timeout=300
numprocs=5 # Start with 5 processes and scale up
Enter fullscreen mode Exit fullscreen mode

Conclusion

Migrating to SQS immediately gives your application a durable, infinitely scalable message broker. It moves the responsibility of message handling and retry logic to a cloud service, freeing up your Laravel application to focus solely on processing business logic. This is the cornerstone of any highly available, modern Laravel deployment on AWS. Feel free to share your valuable suggestions.

Top comments (0)