Avoid Laravel Cache Locks For Indefinite Period For ShouldBeUnique Job and WithoutOverlapping Job…

Avoid Laravel Cache Locks For Indefinite Period For ShouldBeUnique Job and WithoutOverlapping Job Middleware

Maravel-Framework

This article was inspired by an investigation made for this discussion. I introduced in Maravel-Framework version 10.53.3 default 2 hours (7200 seconds) for the WithoutOverlapping and ShouldBeUnique job locks instead of indefinite period (which is generated by default 0 seconds in Laravel ).

The issue is that a job that should timeout will generate a SIGKILL signal (force termination) in a queue worker, preventing any finally block from being executed and also any cache locks from being released:

    protected function registerTimeoutHandler($job, WorkerOptions $options)
    {
        // We will register a signal handler for the alarm signal so that we can kill this
        // process if it is running too long because it has frozen. This uses the async
        // signals supported in recent versions of PHP to accomplish it conveniently.
        pcntl_signal(SIGALRM, function () use ($job, $options) {
            if ($job) {
                $this->markJobAsFailedIfWillExceedMaxAttempts(
                    $job->getConnectionName(),
                    $job,
                    (int)$options->maxTries,
                    $e = $this->timeoutExceededException($job)
                );

                $this->markJobAsFailedIfWillExceedMaxExceptions(
                    $job->getConnectionName(),
                    $job,
                    $e
                );

                $this->markJobAsFailedIfItShouldFailOnTimeout(
                    $job->getConnectionName(),
                    $job,
                    $e
                );

                $this->events->dispatch(
                    new JobTimedOut(
                        $job->getConnectionName(),
                        $job
                    )
                );
            }

            $this->kill(static::EXIT_ERROR, $options);
        }, true);

        pcntl_alarm(
            max($this->timeoutForJob($job, $options), 0)
        );
    }

    public function kill($status = 0, $options = null)
    {
        $this->events->dispatch(new WorkerStopping($status, $options));

        if (extension_loaded('posix')) {
            posix_kill(getmypid(), SIGKILL);
        }

        exit($status);
    }

The best solution is to ALWAYS give a lock period when using these kind of restrictions because the PHP process can stop at any time leaving the lock active. Here is an example for both ShouldBeUnique and WithoutOverlapping :

<?php

namespace App\Jobs;

class YourJob implements ShouldQueue, ShouldBeUnique
{
    use InteractsWithQueue;
    use Queueable;
    use SerializesModels;

    public int $tries = 1;
    public int $timeout = 300;
    public bool $failOnTimeout = true;
    public int $uniqueFor = 300; // you should define this always on a ShouldBeUnique job

    public function uniqueId(): string
    {
        return '123';
    }

    public function middleware(): array
    {
        return [new WithoutOverlapping($this->uniqueId(), 0, $this->timeout)]; // Always give a ttl
    }

    public function failed(?Throwable $exception): void
    {
       // the job instance will not be the same when failing due to timeout.
    }
}

For WithoutOverlapping there is an uglier solution for releasing the locks but this will pile up the shutdown callbacks from each job needing the worker to be restarted after a certain number of jobs processed:

// Child of WithoutOverlapping
public function handle($job, $next)
    {
        $lock = Container::getInstance()->make(Cache::class)->lock(
            $this->getLockKey($job),
            $this->expiresAfter
        );

        if ($lock->get()) {
            $lockWrapper = new \stdClass();
            $lockWrapper->lock = $lock;

            try {
                \register_shutdown_function(function () use ($lockWrapper): void {
                    if (\isset($lockWrapper->lock)) {
                        $lockWrapper->lock->release();
                    }
                });

                $next($job);
            } finally {
                $lock->release();
                $lockWrapper->lock = null;
            }
        } elseif (!is_null($this->releaseAfter)) {
            $job->release($this->releaseAfter);
        }
    }

Run your queue worker with limited amount of jobs to be processed in a PID to avoid the shutdown functions piling up and eating memory.

php artisan queue:work --max-jobs=50

or

php artisan queue:work --once

The same is applicable when scheduling tasks in app/Console/Kernel.php even if this has a default of 24 hours which might be too long:

// \Illuminate\Console\Scheduling\Event::withoutOverlapping
$schedule->command('...')->withoutOverlapping(30) // 30 minutes, default is 1440 minutes (24h)