⚙️ Laravel Queues: Rate-Limiting jobs

Hi guys, today I'd like to talk a bit about rate-limiting Laravel queued jobs 👨🏻‍💻.

This might be a common scenario, where we want to make sure jobs are not processed too fast in order to not overload other system resources.

🤔 In one of my projects there was a case where certain type of jobs had to be processed at specific rate as they had to call a 3rd party API which was rate-limited.

Let's explore different approaches we could take to solve this problem.

1️⃣ Using Laravel's Built-in Rate Limiting

Laravel provides a built-in rate limiting feature that you can use to limit the rate at which jobs are processed.
You can define rate limiters using the RateLimiter::for() method and apply them to your jobs using middleware or directly within your job classes.
Rate limiters can be configured based on various criteria such as the number of attempts per minute, per user, or per resource.

Simply add this to your job class:

    /**
     * The rate limiting middleware for the job.
     */
    public function middleware(): array
    {
        return [
            new \Illuminate\Queue\Middleware\RateLimitedWithRedis('process-job', 10, 60) // Example: 10 jobs per minute
        ];
    }

Pros:

• Easy peasy to use and works out of the box.

Cons:

• Not for every use case: it marks jobs as failed and drops them if the limiter doesn't have enough quota to process the job at the moment.
• This might be useful if the action is triggered too often by an impatient user so some jobs will be dropped but won't work in scenarios where each job must eventually be processed.

2️⃣ Custom Rate Limiting Middleware

If Laravel's built-in rate limiting does not meet your requirements, you can implement custom rate limiting logic within your application thanks to Job Middlewares.
Instead of putting the job back to the queue and decreasing $maxAttempts, you could make the job wait a bit before proceeding to processing it.

Pros:

• Quite simple to implement and reusable - you could put it in a trait and use it across several jobs. Cons:
• It involve quite a lot of custom logic and is relatively difficult to test.
• Time spent waiting for a quota from rate limiting would count towards the job execution time which not always might be desired.

3️⃣ Create a custom Queue Worker

If you are not satisfied with above solutions an you'd like to fetch the job from the queue only when there is quota available in your rate limiter logic, the easiest way to do so is to create a custom Queue Worker:

<?php

namespace App\Queue;

use Illuminate\Queue\Worker;
use Illuminate\Queue\WorkerOptions;

class RateLimitedWorker extends Worker
{
    /**
     * Pop the next job off of the queue with rate limiting.
     *
     * @param  string|null  $connectionName
     * @param  string|null  $queue
     * @param  \Illuminate\Queue\WorkerOptions  $options
     * @return \Illuminate\Contracts\Queue\Job|null
     */
    public function pop($connectionName = null, $queue = null, WorkerOptions $options = null)
    {
        // Implement rate limiting logic here before popping the job
        // For example, use Laravel's rate limiting feature or any custom logic

        return parent::pop($connectionName, $queue, $options);
    }
}

Registering this worker as default application queue worker in your AppServiceProvider will apply this logic to all queues and connections - you can conditionally run the custom logic on certain queues then:

$this->app->extend('queue.worker', function ($worker, $app) {
    return new RateLimitedWorker(
        $app['queue'], 
        $app['events'], 
        $app['queue.failer'], 
        $app['cache.store']
    );
});

The simplest way to rate-limit such jobs is using a sleep() and calculating the number of processes vs allowed limit. More elegant solution would be to use Laravel's RateLimit component. Just make sure you always take into account that there might be many concurrent workers running at any point of time, so use some kind of distributed lock for that. Cheers!

❌ That's a lot of code...

Pros:

• Quite easy to implement ✨
• Jobs will only be picked up when there is quota available to process them 👍🏻

Cons:

• Overriding/extending framework's internal classes might cause problems when upgrading - make sure you got this behaviour covered by tests
• Not so elegant solution

4️⃣ Use Queue::popUsing()

If you don't want to extend framework's classes, you might want to use composition over inheritance to customize the way your app picks up the jobs from the queue:

<?php

use Illuminate\Support\Facades\Queue;

Queue::popUsing(function ($connection, $queue) {
    // Implement rate limiting logic here before popping the job
    // For example, use Laravel's rate limiting feature or any custom logic

    return Queue::getConnection($connection)->pop($queue);
});

Just put this code in one of your service providers and enjoy avoiding nasty inheritance. Well, that didn't sound right.

Pros:

• Elegant,easy to implement and quite flexible solution, without polluting the app with framework internals too much
• Jobs will only be picked up when there is quota to process them

Cons:

• Quite difficult to test it other than manually running it

😎 Summary

Each option offers flexibility and customization to suit different use cases and requirements. Depending on your specific needs, you can choose the approach that best fits your application architecture and rate limiting requirements.

🤓☝🏻 Actually...

In my case, none of them were good enough.

That's because I looked at it from the wrong angle - the fact that the 3rd party API has some limits shouldn't be the queue's concern but should rather be handled in the SDK or the client class that connects to it and solved there. You might one day call this API directly from a request or CLI and you'd have to rewrite the whole thing all over again.

One way to achieve it is to use Guzzle Rate Limiter Middleware which waits for the quota to replenish before calling the API another time.
Simply plug it wherever you are hitting the external API and boom, job done! 🤘🏻

See you again soon!

Comments

[Alex]

I needed to rate limit a group of jobs that used throttled API and keep the sequence of execution, which is hard with simple per job rate limit. The easiest way was to send all jobs to separate queue and just use --rest=5 --queue=route_throttled_queue. Not perfect, but does the thing.