Configuring Laravel Queues with AWS SQS

Introduction

Recently, I had to migrate my queue driver on a project from Redis to SQS and for some weird reason the information provided on the official Laravel Queues documentation didn't do justice to this and I couldn't find an article online that could help, that's why I'm writing this, with the hope that you don't spend two hours of your Saturday morning figuring stuff that should have been documented. I'd try to keep this article as brief as possible, I'd attach links to other articles that explain some steps so I can focus on the important parts.

Prerequisites

• An existing Laravel application
• An AWS account
• An understanding of Laravel queues
• Some AWS knowledge

Step One - Install the AWS SDK

Run the following command to install the SDK:

composer require aws/aws-sdk-php

Step Two - Get your AWS keys

Click on this link to watch the guide.

When you get to the point where you need to set permissions, search for AmazonSQSFullAccess and select it.

If you already have existing credentials, modify its policies and do the same.

Step Three - Create a queue

Head over to your SQS console, click on create queue.

Out of the box, Laravel dispatches jobs to the default queue, therefore, your first queue should be named default on SQS.

If you would be dispatching to a queue other than default, the queue name should be the same as the queue you're dispatching to.

Leave everything else at its default, scroll to the bottom and click on the create queue button.

Step Four - Set .env variables

Open your .env with your preferred text editor. Add your AWS keys in the following format:

AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION=YOUR_SQS_REGION

Ensure you replace YOUR_* with the right credentials. Next, set your SQS Prefix:

SQS_PREFIX=https://sqs.<your-region>.amazonaws.com/<your-account-id>

| I don't think I need to explain what <your-region> means

You can get <your-account-id> on your AWS console, click on your account name on the top right of your dashboard.

you'd find your account ID just beside My Account as shown above. Finally, change the queue driver to SES.

QUEUE_CONNECTION=sqs

Step Five - Start queue worker

Run the following command to start your queue worker:

php artisan queue:work sqs

Dispatch your job on the default queue and it should be processed.

Modification for Laravel horizon

If you use Laravel Horizon to manage your queues, you'd be making some modifications to your horizon.php file. For each worker defined, change the queue connection from redis which is the default to env('QUEUE_CONNECTION').

From this:

<?php
...
'defaults' => [
    'queue_name' => [
        ...
        'connection' => 'redis',
    ],
...
]
...

To this:

<?php
...
'defaults' => [
    'queue_name' => [
        ...
        'connection' => env('QUEUE_CONNECTION'),
    ],
...
]
...

After these modifications have been made, restart horizon and all should be good.

Conclusion

I have nothing else to add, have fun, keep building.

Comments

[Oliver Kurmis]

The Horizon docs at laravel.com/docs/8.x/horizon#insta... say:

"Laravel Horizon requires that you use Redis to power your queue. Therefore, you should ensure that your queue connection is set to redis in your application's config/queue.php configuration file."

So I wonder how this configuration (Horizon+SQS) is actually running,

[Chigozie Ekwonu]

Great article. Worthy of note are the limitations associated with using SQS such as maximum job payload size, maximum job delay, maximum job life span, etc.

[Madalin Ignisca]

This will not provide full Horizon functionality.
Horizon uses an extended RedisQueue class with additional methods used to provide queue metrics and extra functionality.
You must extend from SqsQueue and add the missing pieces and register the new driver for Horizon to achieve complete functionality.

Here is an example of a full alternative driver for queues on RabbitMQ, which also has complete support for Horizon: github.com/vyuldashev/laravel-queu...

With the above you might have broken jobs and you'll have to monitor your error log constantly to catch what Horizon will not.

[Sean Hood]

I found the docs quite lacking too so wrote it up here for personal notes:
snippets.snhd.co/posts/neat-secure...

I actually forget to start the queue runner in my post, but it was more about configuring things within the AWS Ecosystem.

[Logan Lindquist]

Sounds like a two hours well spent! I guess also now we have a time to beat?

[ToMmiE DaRkU]

You don't necessarily have to name your queue "default" on AWS, you can set any name and specify it in your Laravel project's env/config

[Ankur K]

Horizon does not play well with SQS