[Resolved] MySQL lock wait timeout dealing with Laravel queues and jobs

Hi, I'm Valerio, software engineer and CTO at Inspector.

One of the most-read articles I've posted on our blog relates to a queue and jobs configuration in a Laravel application.

Queues and Jobs introduce parallel task execution in your app. And it's also the most important step to enable a new level of scalability while keeping server resources cost-friendly.

So I decided to write about some side effects that Laravel queues adoption can cause in your application. I also want to provide solutions based on my real-life experience.

Use MySQL database transactions

One of the best-known uses of database transactions is when an error occurs, the transaction can be rolled back. Any changes you made to the database within that transaction are rolled back as well.

Data Integrity

DB::transaction(function () {
    DB::table('users')->update(['votes' => 1]);

    DB::table('posts')->delete();
});

This result is beneficial when your code needs to update several tables. Or to run some complicated tasks that could produce exceptions.

If an error occurs in one statement all previous data changes will not apply to the database, keeping your data consistent with itself.

Concurrency

Another thing database transaction can help us with is "concurrency".

In a highly concurrent environment (that's what we want to build using Laravel queues), where the application needs to update resources on the database (such as multiple jobs that want to update the same record in the same table), you can handle this situation retrying to execute the blocked transaction:

DB::transaction(function () {
    DB::table('users')->update(['votes' => 1]);

    DB::table('posts')->delete();
}, 5); // <-- Number of attempts

Here, the second parameter, "5" is the number of times a transaction should reattempt before closing it (rolled back).

And than throwing the "Lock wait timeout" exception.

After becoming familiar with queues and jobs, you may see more exceptions appearing in your log files.

That was my case.

My experience using the Laravel Queue system

I struggled with this issue in a scenario where I needed to update a date-time field in a table called "recently_executed_at". As described above, many concurrent jobs trying to execute the update on the same record caused the "Lock wait timeout" exception.

The code abelow is not a solution:

DB::transaction(function () use ($task) {
    $task->update(['recently_executed_at' => now()]);
}, 5);

because after attempting the execution five times, it throws an exception.

In my case, it was not essential to have this field update with the last execution time from the previously executed job. I only needed a reasonable recent timestamp to show in the frontend as extra information.

When another job is locking the row for an update, I want to skip this statement in my other jobs.

Reading and dealing with Pessimistic/Optimistic locking sent me off the rails for several days. In such a scenario, the solution was to deal with "Race Condition". This is when multiple jobs try to update the same database record at the same time.

In the Laravel documentation, you can find the solution in the Cache section as "Atomic Lock". Unfortunately, it took me almost a week of research to understand that the Atomic Lock in the Cache driver could solve my problem 🤕 🤕 🤕.

Final code

Cache::lock("task-{$task->id}-update")->get(function () use ($task) {
    $task->update(['recently_executed_at' => now()]);
});

It was enough to wrap the update statement in a closure where the LOCK conditions the execution. The code block will be skipped if the lock isn't available (another job is executing the update).

Note that you need to create the lock name based on the "record id". This strategy lets you manage the race condition on one record at a time and not on the entire table.

Laravel application monitoring

If you found this post interesting and want to supercharge your development team with a more effective way to monitor your applications, you can try Inspector.

Inspector is a Code Execution Monitoring tool that helps you to identify bugs and bottlenecks in your applications automatically. Before your customers do.

It is completely code-driven. You won't have to install anything at the server level or make complex configurations in your cloud infrastructure.

It works with a lightweight software library that you can install in your application like any other dependencies. You can try the Laravel package, it's free.

Create an account, or visit our website for more information: https://inspector.dev/laravel

Comments

[Bertug Korucu]

Huge thanks for this post!

[Valerio]

Thank you for your feedback, Happy to help.