Job Queues and Workers In Laravel Apps

This article was originally written by Darius D on the Honeybadger Developer Blog.

We would like to start this article with a real-life example. Imagine an ice cream van. It is a very hot day, and people want to refresh themselves with delicious ice cream. The buyers are all screaming about the kind of ice cream they want; one buyer just wants a single portion, while another has a big family and wants a whole box. This situation doesn’t seem difficult; buyers are sending requests, and sellers are receiving money, preparing orders and providing ice cream to the customers. However, if there are many buyers, the orders are big, or several people are working in the ice cream van simultaneously, it could become chaotic. Many angry buyers could be waiting on their big orders, and there’s only one frantic seller who might not be able to fulfill an order due to tiredness. This is likely one of the reasons queues were invented.

In this article, we will learn how to use queues, define a worker, and analyze some examples using different situations, including how to add queues for one worker, how the program will behave when we have several workers, and how to combine tasks into groups.

What are Queues in Laravel?

The most common situation where we want to use queues is heavy processes, such as importing or synchronizing big data, which takes more time and happens quite often. When we run a big process on page loading, it means the user has to wait until the process finishes to access the content. If we want to improve the user experience, we could make this process run in the background, allow the user the continue working with other stuff, and when all the tasks are finished, we can send the user a notification or message via email.

The main idea of Laravel queues is to create PHP methods, register them in a database, and then tell the server to call these methods sequentially by reading the database. This explanation is perhaps over-simplified because, as well will see later, the registration of these methods (or jobs) can be more complicated if we want to get more information about executed processes and create, for example, a progress bar to show the user his or her current status.

Configuring and Installing Queues

Our installation will start with an artisan command:

php artisan queue:table

This command will create a migration file containing information about the new Jobs database table. So, let's migrate this file:

php artisan migrate

When we have prepared a database table, we can create our first job. Let's name it CalculateDataJob, and by running the following command, we will create a new PHP file:

/app/Jobs/CalculateDataJob.php

php artisan make:job CalculateDataJob

The task we want to run has to be described in a handle public method, such as the following:

public function handle()
{
    for ($x = 1; $x <= 10; $x++) {
        sleep(2);
        // do calculation
    }
}

In this example, we added a very small job that can be executed in milliseconds. If we want to see slower progress, let's add a sleep function that delays execution by two seconds. The handle method should control your task's main logic, or if you have a more complicated situation, you can write a method that calls other methods.

NOTE: Every time you use jobs and make changes in the method, you need to clean the cache. Otherwise, it could cost you additional time for looking bugs in the code; the problem is that the server is running your old code, because it was cached.

php artisan cache:clear

Another new term is Dispatching Jobs. It means we need to queue this job. Let's create a Livewire component, which is a simple button that will activate the job with one click.

php artisan make:livewire JobButton

It will create two files: a class file /app/Http/Livewire/JobButton.php and a view file /resources/views/livewire/job-button.blade.php. In job-button.blade.php, let's add this simple code:

<div wire:click="runJob()">Run job</div>

Anywhere in your website template, insert the following component tag: <livewire:job-button /> or blade directive @livewire('job-button'). In the JobButton.php file, add this new method:

...
use App\Jobs\CalculateDataJob;

public function runJob()
{
    CalculateDataJob::dispatch();
}

Additionally, before triggering this method, we to configure the queue driver. In Laravel, there are several options to choose from:

• Database - Information about jobs will be saved in a database table;
• Redis - Good for big applications and when more flexibility is needed;
• Other - Three dependencies, which can be installed by using a Composer package manager: Amazon SQS, Beanstalkd, or Redis (phpredis PHP extension);

For this example, we will use the database as the queue driver. Therefore, in the .env file, let's find the QUEUE_CONNECTION=sync row (by default, the QUEUE_CONNECTION value is synched, which means that we want to execute the job immediately) and change it to QUEUE_CONNECTION=database.
Finally, if we will click our "Run job" button, information about the job will be saved in the database table "jobs". To process this job, we just need to write a simple command in the console:

php artisan queue:work

After running this commend, we will see messages about processing jobs in the console. Thus, when users click a button, they won't need to wait while the task finishes. The process will be executed in the background.

Regardless of whether a process is executed by a user or the server, we could encounter issues with memory, especially if we are working with big data. Instead of performing one big job, we can split it into several smaller jobs. In the JobButton.php file, make the following change:

public function runJob()
{
    for ($x = 1; $x <= 10; $x++) {
        CalculateDataJob::dispatch($x);
    }
}

With this change, we will create jobs by sending parameters. Thus, in CalculateDataJob.php, we need to add a new public variable, assign it in the constructor, and use it in the handle method.

...

public $x;

public function __construct($x)
{
    $this->x = $x;
}

public function handle()
{
    sleep(2);
    // do calculation with $this->x
}

Now we can see ten registered jobs in the database. These jobs are quite small to help avoid server memory problems.

Laravel Queue Workers

All these examples explore the basics of using queues in Laravel. When we run the artisan command queue:work, we activate workers. When we changed from one job to several smaller jobs, we created ten independent workers, which means that these ten jobs will be executed like there are ten different invisible users in the background.

After activation, queue workers will "live" in the server indefinitely and hold job information in memory. Therefore, when we change something in the code, we need to both clean the cache and restart the workers:

php artisan queue:restart

NOTE: For example, in Linux, after closing the console window, workers will be stopped. Thus, if we want to continue our processes after closing the console window, we need to run the same queue:work but with extra commands:

nohup php artisan queue:work &

As mentioned previously, it is possible to send parameters to a job and split a big task into smaller ones, such as reading a big csv file and splitting it into smaller files. However, a major disadvantage of this approach is that we need to create many small files, read the data, and then delete the files. The best way to resolve this issue is to read the file, chunk the data, and send the chunks of data as parameters. Thus, the data will be saved in a database, and when workers are activated, the data will be read from the jobs table and imported into specific tables.

Other important aspects of queues are handling errors and notifying users. When we only have a few jobs, it may not be crucial, but if we are talking about a big system performing many tasks per day, it is very important to register failed jobs. Hence, in the same CalculateDataJob.php file, we can add a new public method: failed.

...
public function failed(Throwable $exception)
{
    // Send user notification of failed job
}

Laravel Queue Workers Process Progress Information

To run processes in the background and avoid having to wait, it is a very good solution. However, what if the user wants to know what's going on in the background? The best way is to present live information or simply a progress bar. From Laravel version 8, we have Job Batching. This functionality requires new database tables to save detailed information about jobs.

php artisan queue:batches-table
php artisan migrate

Therefore, in our JobButton.php file, we need to change the old code to the following:

...
use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;
use Throwable;

public function runJob()
{
    $this->batchHolder = Bus::batch([])->then(function (Batch $batch) {
        // All jobs completed successfully...
    })->catch(function (Batch $batch, Throwable $e) {
        // First batch job failure detected...
    })->finally(function (Batch $batch) {
        // The batch has finished executing...
    })->name('data_calculation')->dispatch();
    for ($x = 1; $x <= 10; $x++) {
        CalculateDataJob::dispatch($x);
        $this->batchHolder->add(new CalculateDataJob($x));
    }
    return $this->batchHolder;
}

If we have different groups of jobs, we can name them. In this example, it’s data_calculation, so it will be easier to separate individual jobs of a particular task and only receive information about these jobs. When you are using Job batching, add Batchable to the CalculateDataJob.php file.

use Illuminate\Bus\Batchable;
...
class ApiUpdateJob implements ShouldQueue
{
    use Batchable,  Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
    ...
}

To create a progress bar, we need to add code to job-button.blade.php:

<div wire:poll.1000ms="checkStatus()">
    <div style="background-color: black; padding: 3px;">
        <div style="background-color: red; text-align:center; height: 20px; width: {{ $loaded_value }}%;">{{ $loaded_value }}%</div>
    </div>
</div>

The main html tag will call the checkStatus() method each second and refresh the $loaded_value value. In JobButton.php, we will add a public checkStatus method:

...
public function checkStatus()
{
    $batches = DB::table('job_batches')->where([['pending_jobs','>',0],['name','=','data_calculation']])->orderBy('created_at', 'desc')->limit(10)->get();
    if(count($batches) > 0){
        $job_status = Bus::findBatch($batches[0]->id)->toArray();
        $this->loaded_value = round($job_status['progress'],0);
    }else{
        $this->loaded_value = 0;
    }
}

In this method, we will send requests to the database to get, for example, the last ten created jobs with the name data_calculation, and pick from the first item to provide job status information. If we print the variable $job_status, we will see 5 different elements: totalJobs, pendingJobs, processedJobs, progress, and failedJobs. For now, we are only interested in progress, but other elements could be very important, too, if we want to give user more detailed information.

Bonus Example

One of the most popular use cases for queue workers is sending email messages to users. These messages could be a welcome letter sent to users after registration or after users perform a specific action. First, add the credentials of your mailbox to the .env file. If you don't want to use your private email, you can easily register for a free account at https://mailtrap.io, which is very good tool for sending and testing email functionality. We need to create a "mailable" class:

php artisan make:mail SendEmail

It will automatically create a default email template app/Mail/SendEmail.php, which can be changed to meet your needs. As in previous examples, we need to create a job for this purpose:

php artisan make:job SendWelcomeEmailJob

In the newly created app/Jobs/SendWelcomeEmailJob.php file, a handle method needs to be added:

use App\Mail\SendEmail;
...
public function handle()
{
    $test_email = '[e-mail]';
    Mail::to($test_email)->send(new SendEmail());
}

Finally, we can create a simple route to call this job; in the web.php file, add a new route: send-email

use App\Jobs\SendWelcomeEmailJob;
...
Route::get('send-email', function(){
    dispatch(new SendWelcomeEmailJob());
});

This is another good example where we can use queues.

NOTE: If we want to delay our job process, we can use the delay method:

$job = (new SendWelcomeEmailJob())
    ->delay(Carbon::now()->addMinutes(10));
dispatch($job);

Applying this method will cause the job to be dispatched after ten minutes.

Queue Workers on a Live Server

All the examples we’ve shown so far should work and on either a local host or a live server. However, on a live server, the administrator can't constantly check whether the queue workers are active and run the php artisan queue:work command when necessary, especially if the project is international and involves users from different time zones. What happens when a worker encounters an error or an execution timeout? Your application queue workers must be active 24 hours per day and react every time a user performs a specific action. On a live server, a process monitor is needed. This monitor controls the queue workers and automatically restarts processes if they fail or are impacted by other processes. A supervisor is commonly used on Linux server. The first step of installing a supervisor on a live server is to run the following command:

sudo apt-get install supervisor

After installation, in the /etc/supervisor/conf.d directory, prepare a configuration laravel-worker.conf file with the following content:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app.com/artisan queue:work sqs --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=root
numprocs=8
redirect_stderr=true
stdout_logfile=/var/www/app.com/worker.log
stopwaitsecs=3600

All directories depend on your server structure. After the configuration files are created, you’ll need to activate the supervisor using the following commands:

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start laravel-worker:*

If configuring queue workers on a live server seems too complicated, there are many hosting providers with special tools that perform these functions in a more user-friendly manner, such as Laravel Forge or Digitalocean.

Conclusion

Laravel queues is a very powerful tool to improve your application’s performance, especially when the application has many heavy, frequently executed tasks. We don't want to make our users wait until a heavy job is finished. Thus, after one click, the user should still be free to navigate to other pages or perform other actions. However, keep in mind that background processes can take down a server if not properly configured or tested. It is advisable to split big jobs into smaller ones to avoid process execution timeout.