Robust queue job handling in Craft CMS

Craft CMS uses queue jobs for long-run­ning tasks. Here’s how to avoid stalled or failed queue jobs by using an inde­pen­dent async queue runner

Andrew Welch / nystudio107

Queu­ing in line is some­thing few peo­ple are fond of, but it allows each per­son to be ser­viced in order, in an effi­cient and fair manner.

Craft CMS also uses a queue, too. Any lengthy task is pack­aged off into a job , and jobs are put into a queue to be exe­cut­ed in order.

In com­put­er sci­ence terms, this is a First In, First Out (FIFO) data struc­ture that ser­vices things in what we’d con­sid­er to be a nat­ur­al order.

Here are some lengthy tasks that end up as jobs in the Craft CMS queue:

• Updat­ing entry drafts and revisions
• Delet­ing stale tem­plate caches
• Gen­er­at­ing pend­ing image transforms
• Re-sav­ing Elements
• Updat­ing search indexes
• Plu­g­in cre­at­ed queue jobs

The queue exists to allow these often lengthy jobs to run in the back­ground with­out block­ing access to the Craft CMS Con­trol Pan­el (CP) while they execute.

See How They Run

So now what we know what type of jobs will end up in the queue, let’s have a look at how these jobs are run. This ends up being a cru­cial­ly impor­tant bit of infor­ma­tion to under­stand about queue jobs.

Giv­en that Craft CMS is built on top of the Yii2 frame­work, it makes sense that it lever­ages the exist­ing Yii2-queue by lay­er­ing a QueueIn­ter­face on top of it.

It does this so that it can pro­vide a nice friend­ly sta­tus UI in the CP. The default Craft set­up uses a data­base (Db) dri­ver for queue jobs, stor­ing them in the queue table.

Although there are oth­er queue dri­vers avail­able, you’re unlike­ly to see any gains from using any­thing oth­er than the Craft CMS Db dri­ver. And you’d lose the afore­men­tioned sta­tus UI in the CP.

Where queue jobs are stored and how they are run are com­plete­ly sep­a­rate things.

But how do the queue jobs run? Here’s how:

• When some­one access the CP
• Via web request

Here­in lies the rub with the default queue imple­men­ta­tion in Craft CMS. No queue jobs will run at all unless some­one logs into the CP, and even then, they are run via web request.

Here’s an excerpt from the Craft CMS CP.js JavaScript that kicks off the the web-based queue running:

runQueue: function() {
    if (!this.enableQueue) {
        return;
    }

    if (Craft.runQueueAutomatically) {
        Craft.queueActionRequest('queue/run', $.proxy(function(response, textStatus) {
            if (textStatus === 'success') {
                this.trackJobProgress(false, true);
            }
        }, this));
    }
    else {
        this.trackJobProgress(false, true);
    }
},

And here’s the queue/​run con­troller method that ends up get­ting exe­cut­ed via AJAX:

/**
 * Runs any waiting jobs.
 *
 * @return Response
 */
public function actionRun(): Response
{
    // Prep the response
    $response = Craft::$app->getResponse();
    $response->content = '1';
    // Make sure Craft is configured to run queues over the web
    if (!Craft::$app->getConfig()->getGeneral()->runQueueAutomatically) {
        return $response;
    }
    // Make sure the queue isn't already running, and there are waiting jobs
    $queue = Craft::$app->getQueue();
    if ($queue->getHasReservedJobs() || !$queue->getHasWaitingJobs()) {
        return $response;
    }
    // Attempt to close the connection if this is an Ajax request
    if (Craft::$app->getRequest()->getIsAjax()) {
        $response->sendAndClose();
    }
    // Run the queue
    App::maxPowerCaptain();
    $queue->run();
    return $response;
}

They are doing what they can to make sure the web-based queue jobs will run smooth­ly, but there’s only so much that can be done. Bran­don Kel­ly from Pix­el & Ton­ic even not­ed ​“Craft 2, and Craft3 by default, basi­cal­ly have a poor-man’s queue”.

Run­ning lengthy oper­a­tions via web request rarely ends well

Web requests are real­ly meant for rel­a­tive­ly short, atom­ic oper­a­tions like load­ing a web page. For this rea­son, both web servers and PHP process­es have time­outs to pre­vent them from tak­ing too long.

This can result in bad things happening:

• Queue jobs stack up until some­one logs into the CP
• Queue jobs can fail or get stuck if they take too long and time out
• Queue jobs can fail or get stuck if they run out of memory
• Lengthy queue jobs can impact CP or fron­tend per­for­mance as they run

For­tu­nate­ly, we can alle­vi­ate all of these prob­lems, and it’s not hard to do!

If you do have tasks that have failed due to errors, check out the Zen and the Art of Craft CMS Log File Read­ing arti­cle to learn how to debug it.

Queue Run­ning Solutions

For­tu­nate­ly Yii2, and by exten­sion Craft CMS, comes with a robust set of com­mand line tools that I detail in the Explor­ing the Craft CMS 3 Con­sole Com­mand Line Inter­face (CLI) article.

The queue com­mand gives us the abil­i­ty to shift our queue jobs into high gear by run­ning them direct­ly. PHP code that is run via the con­sole com­mand line inter­face (CLI) uses a sep­a­rate php.ini file than the web-ori­ent­ed php-fpm. Here’s an exam­ple from my server:

/etc/php/7.1/fpm/php.ini
/etc/php/7.1/cli/php.ini

The rea­son this is impor­tant is that while the web-based php-fpm might have rea­son­able set­tings for memory_​limit and max_​execution_​time, the CLI-based PHP will typ­i­cal­ly have no memory_limit and no max_execution_time.

This is because web process­es are exter­nal­ly trig­gered and there­fore untrust­ed, where­as things run­ning via the CLI is inter­nal and trusted.

So run­ning our queue jobs via PHP CLI gives us the fol­low­ing benefits:

• Can be run any time, not just when some­one access­es the CP
• Won’t be ter­mi­nat­ed due to lengthy exe­cu­tion time
• Are unlike­ly to run out of memory

The queue con­sole com­mands we use are:

• queue/run — Runs all jobs in the queue, then exits
• queue/listen — Per­pet­u­al­ly runs a process that lis­tens for new jobs added to the queue and runs them

There are sev­er­al ways we can run these com­mands, dis­cussed in the Yii2 Work­er Start­ing Con­trol arti­cle, and we’ll go into detail on them here.

Solu­tion #1: Async Queue Plugin

If you want a no fuss, no muss solu­tion that just involves installing a plu­g­in, check out Oliv­er Stark’s Async Queue plu­g­in. From the plug­in’s description:

[Async Queue plu­g­in] replaces Craft’s default queue han­dler and moves queue exe­cu­tion to a non-block­ing back­ground process. The com­mand craft queue/run gets exe­cut­ed right after you push a Job to the queue.

If you want a sim­ple solu­tion that does­n’t require doing any­thing more than installing a plu­g­in, go this route. You’ll end up with a more robust solu­tion for run­ning your queue jobs.

That’s it, you’re done!

Solu­tion #2: Forge Daemons

If you use the won­der­ful Lar­avel Forge to pro­vi­sion your servers (dis­cussed in detail in the How Agen­cies & Free­lancers Should Do Web Host­ing arti­cle), Forge pro­vides a nice GUI inter­face to what it calls Dae­mons.

Go to your Serv­er, then click on Dae­mons, and you’ll see a GUI that looks like this:

Here’s the com­mand we’ve added for the dev​Mode​.fm website:

/usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose

Let’s break this down:

• /usr/bin/nice -n 10 — this uses the Unix nice com­mand to run the process using a low­er pri­or­i­ty, so it won’t inter­fere with the CP or frontend
• /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose — this starts a PHP CLI process to lis­ten for any queue jobs, and run them

You should swap in your own path to the craft CLI exe­cutable; ours is /home/forge/devmode.fm/craft

We tell it that we want to run this as the user forge and we want 2 of these process­es run­ning. Click Start Dae­mon and you’ll be greet­ed with:

Then ensure that you’ve set the run­QueueAu­to­mat­i­cal­ly Craft CMS general.php con­fig set­ting to false for your live pro­duc­tion environment:

    // Live (production) environment
    'live' => [
        // Craft defined config settings
        'runQueueAutomatically' => false,
    ],

This makes sure that Craft will no longer try to run its queue via web requests.

That’s it, you’re done!

The fol­low­ing is an entire­ly option­al peek under the hood at how this works with Lar­avel Forge. Feel free to skip it if you’re not inter­est­ed in the gory details.

What Forge calls Dae­mons are actu­al­ly com­mands run via the Super­vi­sor com­mand. When we cre­at­ed the above dae­mon in the Forge GUI, it cre­at­ed a file in /etc/supervisor/conf.d:

forge@nys-production /etc/supervisor/conf.d $ ls -al
-rw-r--r-- 1 root root 293 Aug 3 14:55 daemon-157557.conf

The con­tents of the file look like this:

[program:daemon-157557]
command=/usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose

process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
user=forge
numprocs=2
redirect_stderr=true
stdout_logfile=/home/forge/.forge/daemon-157557.log

And indeed, this is exact­ly one of the meth­ods rec­om­mend­ed in the in the Yii2 Work­er Start­ing Con­trol arti­cle. Huzzah!

Solu­tion #3 Using Heroku

If you’re using Heroku for your host­ing needs, you can use Work­er Dynos to run your queue/listen command:

./craft queue/listen --verbose

Then ensure that you’ve set the run­QueueAu­to­mat­i­cal­ly Craft CMS general.php con­fig set­ting to false for your live pro­duc­tion environment:

    // Live (production) environment
    'live' => [
        // Craft defined config settings
        'runQueueAutomatically' => false,
    ],

This makes sure that Craft will no longer try to run its queue via web requests.

That’s it, you’re done!

Solu­tion #4: Uti­liz­ing systemd

If you’re using a Lin­ux serv­er, but you’re not using Forge or Heroku, or per­haps you just want to con­fig­ure things your­self, we can accom­plish the same thing using sys­temd.

systemd is a way to start, stop, and oth­er­wise man­age dae­mon process­es that run on your serv­er. You’re already using it whether you know it or not, for things like your web serv­er, MySQL, and so on.

Here’s what I did for dev​Mode​.fm to use systemd to run our queue jobs. You’ll need sudo access to be able to do this, but for­tu­nate­ly most mod­ern VPS host­ing ser­vices give you this ability.

First I cre­at­ed a .service file in /etc/systemd/system/ named devmode-queue@.service via:

forge@nys-production ~ $ sudo nano /etc/systemd/system/devmode-queue@.service

Here’s what the con­tents of the file looks like:

[Unit]
Description=devMode.fm Queue Worker %I
After=network.target
After=mysql.service
Requires=mysql.service

[Service]
User=forge
Group=forge
ExecStart=/usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose
Restart=always

[Install]
WantedBy=multi-user.target

Let’s break this down:

• /usr/bin/nice -n 10 — this uses the Unix nice com­mand to run the process using a low­er pri­or­i­ty, so it won’t inter­fere with the CP or frontend
• /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose — this starts a PHP CLI process to lis­ten for any queue jobs, and run them

You should swap in your own path to the craft CLI exe­cutable; ours is /home/forge/devmode.fm/craft

We tell it that we want to run this as the user forge , we want it to restart the process on fail­ure and that it depends on both the network and mysql ser­vices being available.

The strange @ in the .service file name allows us to have mul­ti­ple instances of this ser­vice run­ning. We can then use the range syn­tax {1..2} to spec­i­fy how many of these process­es we want.

We’ve cre­at­ed the file, but we need to start the ser­vice via:

forge@nys-production ~ $ sudo systemctl start devmode-queue@{1..2}

And then just like our oth­er ser­vices such as nginx and mysqld we want it to start up auto­mat­i­cal­ly when we restart our server:

forge@nys-production ~ $ sudo systemctl enable devmode-queue@{1..2}

Then ensure that you’ve set the run­QueueAu­to­mat­i­cal­ly Craft CMS general.php con­fig set­ting to false for your live pro­duc­tion environment:

    // Live (production) environment
    'live' => [
        // Craft defined config settings
        'runQueueAutomatically' => false,
    ],

This makes sure that Craft will no longer try to run its queue via web requests.

That’s it, you’re done!

Now you can start, stop, enable, dis­able, etc. your Craft CMS queue run­ner just like you would any oth­er sys­tem ser­vice. You can also mon­i­tor it via the jour­nalctl command:

forge@nys-production ~ $ sudo journalctl -f -u devmode-queue@*.service
-- Logs begin at Sat 2019-08-03 13:23:17 EDT. --
Aug 03 14:06:04 nys-production nice[1364]: Processing element 42/47 - Neutrino: How I Learned to Stop Worrying and Love Webpack
Aug 03 14:06:04 nys-production nice[1364]: Processing element 43/47 - Google AMP: The Battle for the Internet's Soul?
Aug 03 14:06:04 nys-production nice[1364]: Processing element 44/47 - The Web of Future Past with John Allsopp
Aug 03 14:06:04 nys-production nice[1364]: Processing element 45/47 - Web Hosting with ArcusTech's Nevin Lyne
Aug 03 14:06:04 nys-production nice[1364]: Processing element 46/47 - Tailwind CSS utility-first CSS with Adam Wathan
Aug 03 14:06:04 nys-production nice[1364]: Processing element 47/47 - Talking Craft CMS 3 RC1!
Aug 03 14:06:04 nys-production nice[1364]: 2019-08-03 14:06:04 [1427] Generating episodes sitemap (attempt: 1) - Done (1.377 s)
Aug 04 09:25:25 nys-production nice[1364]: 2019-08-04 09:25:25 [1429] Generating calendar sitemap (attempt: 1) - Started
Aug 04 09:25:25 nys-production nice[1364]: Processing element 1/1 - Calendar of Upcoming Episodes
Aug 04 09:25:25 nys-production nice[1364]: 2019-08-04 09:25:25 [1429] Generating calendar sitemap (attempt: 1) - Done (0.174 s)

The above com­mand will tail the systemd logs, only show­ing mes­sages from our devmode-queue service.

You can learn more about using journalctl in the How To Use Jour­nalctl to View and Manip­u­late Sys­temd Logs article.

Get­ting Dotenvious

This is an entire­ly option­al sec­tion you can skip if you’re not inter­est­ed in learn­ing how to not use .env files in pro­duc­tion, as rec­om­mend­ed by the php­dotenv pack­age authors:

php­dotenv is made for devel­op­ment envi­ron­ments, and gen­er­al­ly should not be used in pro­duc­tion. In pro­duc­tion, the actu­al envi­ron­ment vari­ables should be set so that there is no over­head of load­ing the .env file on each request. This can be achieved via an auto­mat­ed deploy­ment process with tools like Vagrant, chef, or Pup­pet, or can be set man­u­al­ly with cloud hosts like Pagod­abox and Heroku.

You do not need to do any of this unless you do not use .env in production

Addi­tion­al­ly some setups have envi­ron­men­tal vari­ables intro­duced by alter­nate means. I made a Doten­vy pack­age that makes it easy to gen­er­ate .env key/​value vari­able pairs as Apache, Nginx, and shell equivalents.

In this case, we’ll need to mod­i­fied the com­mand we run slight­ly, so that we’re defin­ing our envi­ron­men­tal vari­ables in the shell before run­ning Craft via the CLI.

All we need to do is change our com­mand to this:

/bin/bash -c ". /home/forge/devmode.fm/.env_cli.txt && /usr/bin/nice -n 10 /usr/bin/php /home/forge/devmode.fm/craft queue/listen --verbose"

All we’re doing is exe­cut­ing the .env_cli.txt file (cre­at­ed by Doten­vy) that con­tains our envi­ron­men­tal vari­ables we export to the shell environment:

forge@nys-production ~/devmode.fm (master) $ cat .env_cli.txt
# CLI (bash) .env variables
# Paste these inside your .bashrc file in your $HOME directory:
export ENVIRONMENT="live"
export SECURITY_KEY="XXXXX"
export DB_DRIVER="pgsql"
export DB_SERVER="localhost"
export DB_USER="devmode"
export DB_PASSWORD="XXXXX"
export DB_DATABASE="devmode"
export DB_SCHEMA="public"
export DB_TABLE_PREFIX=""
export DB_PORT="5432"
export SITE_URL="https://devmode.fm/"
export BASE_PATH="/home/forge/devmode.fm/web/"
export BASE_URL="https://devmode.fm/"
export REDIS_HOSTNAME="localhost"
export REDIS_PORT="6379"
export REDIS_DEFAULT_DB="0"
export REDIS_CRAFT_DB="3"
export PUBLIC_PATH="/dist/"
export DEVSERVER_PUBLIC="http://192.168.10.10:8080"
export DEVSERVER_HOST="0.0.0.0"
export DEVSERVER_POLL="1"
export DEVSERVER_PORT="8080"
export DEVSERVER_HTTPS="0"

The rest is the same as the com­mands we’ve dis­cussed above for run­ning queue/listen, and then we wrap it all in /bin/sh -c so it appears as a sin­gle com­mand that can be executed.

Deeply Queued

Here’s some queue job triv­ia that you may find interesting.

We men­tioned ear­li­er that the only way queue jobs are run nor­mal­ly is if you vis­it the CP. This is most­ly true.

There’s actu­al­ly a case where queue jobs can be run via fron­tend page load. If you have gen­er­ate­Trans­forms­Be­forePage­Load set to false (which is the default), and there are pend­ing trans­forms that need to be generated.

What hap­pens is the Assets::getAssetUrl() method push­es the image trans­form queue job, which calls Queue::push():

    /**
     * @inheritdoc
     */
    public function push($job)
    {
        // Capture the description so pushMessage() can access it
        if ($job instanceof JobInterface) {
            $this->_jobDescription = $job->getDescription();
        } else {
            $this->_jobDescription = null;
        }

        if (($id = parent::push($job)) === null) {
            return null;
        }

        // Have the response kick off a new queue runner if this is a site request
        if (Craft::$app->getConfig()->getGeneral()->runQueueAutomatically && !$this->_listeningForResponse) {
            $request = Craft::$app->getRequest();
            if ($request->getIsSiteRequest() && !$request->getIsAjax()) {
                Craft::$app->getResponse()->on(Response::EVENT_AFTER_PREPARE, [$this, 'handleResponse']);
                $this->_listeningForResponse = true;
            }
        }

        return $id;
    }

So if this is a site (fron­tend) request, and it’s not an AJAX request, Craft will add a call to Queue::handleResponse() that gets fired when the Response is about to be sent back to the client.

In this case, Craft will inject some JavaScript into the fron­tend request that pings the queue/run con­troller via AJAX:

    /**
     * Figure out how to initiate a new worker.
     */
    public function handleResponse()
    {
        // Prevent this from getting called twice
        $response = Craft::$app->getResponse();
        $response->off(Response::EVENT_AFTER_PREPARE, [$this, 'handleResponse']);

        // Ignore if any jobs are currently reserved
        if ($this->getHasReservedJobs()) {
            return;
        }

        // Ignore if this isn't an HTML/XHTML response
        if (!in_array($response->getContentType(), ['text/html', 'application/xhtml+xml'], true)) {
            return;
        }

        // Include JS that tells the browser to fire an Ajax request to kick off a new queue runner
        // (Ajax request code adapted from http://www.quirksmode.org/js/xmlhttp.html - thanks ppk!)
        $url = Json::encode(UrlHelper::actionUrl('queue/run'));
        $js = <<<EOD
<script type="text/javascript">
/*<![CDATA[*/
(function(){
    var XMLHttpFactories = [
        function () {return new XMLHttpRequest()},
        function () {return new ActiveXObject("Msxml2.XMLHTTP")},
        function () {return new ActiveXObject("Msxml3.XMLHTTP")},
        function () {return new ActiveXObject("Microsoft.XMLHTTP")}
    ];
    var req = false;
    for (var i = 0; i < XMLHttpFactories.length; i++) {
        try {
            req = XMLHttpFactories[i]();
        }
        catch (e) {
            continue;
        }
        break;
    }
    if (!req) return;
    req.open('GET', $url, true);
    if (req.readyState == 4) return;
    req.send();
})();
/*]]>*/
</script>
EOD;

        if ($response->content === null) {
            $response->content = $js;
        } else {
            $response->content .= $js;
        }
    }

While tech­ni­cal­ly this could hap­pen for any queue job that is Queue::pushed()​‘d in the queue dur­ing a fron­tend request, asset trans­forms are the only case I’m aware of that this hap­pens in practice.

For this rea­son, if you’re post­ing any XHR requests to Craft con­trollers or API end­points, make sure you add the header:

'X-Requested-With': 'XMLHttpRequest'

Craft looks for this head­er to deter­mine whether the request is AJAX or not, and will not attempt to attach its queue run­ner JavaScript to the request if the request is AJAX.

This head­er is set auto­mat­i­cal­ly by jQuery for $.ajax() requests, but it is not set auto­mat­i­cal­ly by pop­u­lar libraries such as Axios.

Wrap­ping Up

So which method do I use? If I’m spin­ning up a VPS with Forge (which I nor­mal­ly am), then I’ll use Solu­tion #2: Forge Dae­mons. If it’s not a Forge box, but I do have sudo access, then I’ll use Solu­tion #3: Uti­liz­ing sys­temd.

I use Solu­tion #1: Async Queue Plu­g­in on client sites where nei­ther of the above are the case. Oliv­er’s plu­g­in is great, and much bet­ter than the default way that queue jobs are run, but I like lever­ag­ing exist­ing sys­tems if pos­si­ble, and hav­ing nice con­trol over the process pri­or­i­ty is, well, nice.

Replac­ing the default web-based queue job sys­tem in Craft CMS with a more robust CLI-based sys­tem will pay dividends.

Your queue jobs will run more smooth­ly, fail less often, and have less of an impact on the CP/​frontend.

It’s worth it. Do it.

Hap­py queuing!

Further Reading

If you want to be notified about new articles, follow nystudio107 on Twitter.

Copyright ©2020 nystudio107. Designed by nystudio107