DEV Community: Andrew Welch

Speeding Up Tailwind CSS Builds

Andrew Welch — Tue, 13 Oct 2020 12:29:00 +0000

Speeding Up Tailwind CSS Builds

Learn how to optimize your Tailwind CSS PostCSS build times to make local development with Hot Module Replacement or Live Reload orders of magnitude faster!

Tailwind CSS is a utility-first CSS framework that we’ve been using for several years, and it’s been pretty fantastic. We first talked about it on the Tailwind CSS utility-first CSS with Adam Wathan episode of devMode.fm way back in 2017!

We use it in the base of every project we do, as part of a webpack build process described in the An Annotated webpack 4 Config for Frontend Web Development article.

One issue that we’ve run into recently is that build times can be slow in local development, where you really want the speed as you are building out the site CSS.

This article will lead you through how to optimize your Tailwind CSS build process so that your hot module replacement / live reload will be fast again, and explains what’s going on under the hood.

So let’s dive in!

Framing the Problem

The problem I was having was that a simple edit of one of my .pcss files would result in a good 10 second or more delay as the CSS was rebuilt.

Granted, one of the lovely things about Tailwind CSS is that you don’t have to write much custom CSS; mostly you’re putting utility classes in your markup.

However, when you do want to add custom CSS, it shouldn’t be this painful

So if I wanted to do something like change the background color of a particular CSS class, instead of the instant feedback I was used to from webpack hot module replacement, I’d be waiting a good while for it to recompile.

I wrote up Tailwind CSS issue #2544, and also created a minimal GitHub repo nystudio107/tailwind-css-performance to reproduce it.

But then started spelunking a bit further to see if I could find a way to mitigate the situation.

I found a technique that sped up my build times immeasurably, and you can use it too.

The Problem Setup

My original base Tailwind CSS setup looked roughly like this:


css
├── app.pcss
├── components
│ ├── global.pcss
│ ├── typography.pcss
│ └── webfonts.pcss
├── pages
│ └── homepage.pcss
└── vendor.pcss

The meat of this is the app.css, which looked like this:


/**
 * app.css
 *
 * The entry point for the css.
 *
 */

/**
 * This injects Tailwind's base styles, which is a combination of
 * Normalize.css and some additional base styles.
 */
 @import "tailwindcss/base";

/**
 * This injects any component classes registered by plugins.
 *
 */
@import 'tailwindcss/components';

/**
 * Here we add custom component classes; stuff we want loaded
 * *before* the utilities so that the utilities can still
 * override them.
 *
 */
@import './components/global.pcss';
@import './components/typography.pcss';
@import './components/webfonts.pcss';

/**
 * This injects all of Tailwind's utility classes, generated based on your
 * config file.
 *
 */
@import 'tailwindcss/utilities';

/**
 * Include styles for individual pages
 *
 */
@import './pages/homepage.pcss';

/**
 * Include vendor css.
 *
 */
 @import 'vendor.pcss';

This app.pcss file then gets imported into my app.ts via:


// Import our CSS
import '../css/app.pcss';

This causes webpack to be aware of it, so the .pcss gets pulled into the build pipeline, and gets processed.

And then my postcss.config.js file looked like this:


module.exports = {
    plugins: [
        require('postcss-import')({
            plugins: [
            ],
            path: ['./node_modules'],
        }),
        require('tailwindcss')('./tailwind.config.js'),
        require('postcss-preset-env')({
            autoprefixer: { },
            features: {
                'nesting-rules': true
            }
        })
    ]
};

And we have a super-standard tailwind.config.js file:


module.exports = {
  theme: {
    // Extend the default Tailwind config here
    extend: {
    },
    // Replace the default Tailwind config here
  },
  corePlugins: {},
  plugins: [],
};

The Problem

This setup above is pretty much what is laid out Tailwind CSS docs section on Using with Preprocessors: PostCSS as your preprocessor.

Tailwind CSS is built using PostCSS, so it makes sense that if you’re using a buildchain that uses webpack or Laravel Mix (which uses webpack under the hood) or Gulp that you’d leverage PostCSS there, too.

The documentation recommends that if you’re using the postcss-import plugin (which we are, and is a very commonly used PostCSS plugin), that you change this:


@tailwind base;
@import "./custom-base-styles.css";

@tailwind components;
@import "./custom-components.css";

@tailwind utilities;
@import "./custom-utilities.css";

To this:


@import "tailwindcss/base";
@import "./custom-base-styles.css";

@import "tailwindcss/components";
@import "./custom-components.css";

@import "tailwindcss/utilities";
@import "./custom-utilities.css";

This is because postcss-import strictly adheres to the CSS spec and disallows @import statements anywhere except at the very top of a file, so we can’t mix them in together with our other CSS or @tailwind directives.

And this is where things start to go pear-shaped

When we use @import "tailwindcss/utilities"; instead of @tailwind utilities; all that’s really happening is the tailwindcss/utilities.css file is imported:


@tailwind utilities;

So we’re just side-stepping the @import location requirement in postcss-import by adding a layer of indirection.

But we can have a look at node_modules/tailwindcss/dist/ to get a rough idea how large this generated file is going to be:


❯ ls -alh dist
total 43568
drwxr-xr-x 12 andrew staff 384B Sep 19 11:34 .
drwxr-xr-x 19 andrew staff 608B Sep 19 11:35 ..
-rw-r--r-- 1 andrew staff 11K Oct 26 1985 base.css
-rw-r--r-- 1 andrew staff 3.1K Oct 26 1985 base.min.css
-rw-r--r-- 1 andrew staff 1.9K Oct 26 1985 components.css
-rw-r--r-- 1 andrew staff 1.3K Oct 26 1985 components.min.css
-rw-r--r-- 1 andrew staff 5.4M Oct 26 1985 tailwind-experimental.css
-rw-r--r-- 1 andrew staff 4.3M Oct 26 1985 tailwind-experimental.min.css
-rw-r--r-- 1 andrew staff 2.3M Oct 26 1985 tailwind.css
-rw-r--r-- 1 andrew staff 1.9M Oct 26 1985 tailwind.min.css
-rw-r--r-- 1 andrew staff 2.2M Oct 26 1985 utilities.css
-rw-r--r-- 1 andrew staff 1.8M Oct 26 1985 utilities.min.css

(incidentally, if you look closely, you’ll also have learned Adam Wathan’s birthday)

We can see that the utilities.css file weighs in at a hefty 2.2M itself; and while this comes out in the wash for production builds when you’re using PurgeCSS as recommended in Tailwind CSS docs: Controlling file size, it can be problematic for local development.

So but why is this a problem? If we’re just @import’ing the file, why would this be so slow?

The reason twofold:

Although Tailwind CSS has optimizations in placeto mitigate it, there’s still a ton of CSS generation that has to happen each time a change is made in any of your .pcss files. We lumped them all in together, so they all get rebuild together.
The postcss-import plugin actually parses any files you @import, looking for other @import statements in that imported file, and it does this on the CSS that the @tailwind directive generates, too.

And our resulting utilities.css file has by default over 100,000 lines of CSS generated & parsed through:


❯ wc -l utilities.css
  102503 utilities.css

So that’s not good.

The Solution

So what can we do? It’s inherent to Tailwind CSS that it’s going to create a ton of utility CSS for you, and that generation can only be optimized so much.

I started thinking of various caching mechanism that could be added to Tailwind CSS, but I realized the right solution was to just leverage the platform.

I remembered an old Comp Sci maxim:

The fastest code is the code you never have to execute

We’re already using webpack, which adroitly handles tons of imports of various kinds through a variety of loaders… why not just break our .pcss up into chunks, and let webpack sort it out?

The Solution Setup

So that’s exactly what I did in the solution branch of nystudio107/tailwind-css-performance.

Now our CSS directory looks like this:


css
├── app-base.pcss
├── app-components.pcss
├── app-utilities.pcss
├── components
│ ├── global.pcss
│ ├── typography.pcss
│ └── webfonts.pcss
├── pages
│ └── homepage.pcss
├── tailwind-base.pcss
├── tailwind-components.pcss
├── tailwind-utilities.pcss
└── vendor.pcss

Our app.pcss file has been chopped up into 6 separate .pcss files that correspond with Tailwind’s base, components, and utilities methodology:


/**
 * This injects Tailwind's base styles, which is a combination of
 * Normalize.css and some additional base styles.
 */
@tailwind base;



/**
 * Here we add custom base styles, applied after the tailwind-base
 * classes
 *
 */



/**
 * This injects any component classes registered by plugins.
 *
 */
@tailwind components;



/**
 * Here we add custom component classes; stuff we want loaded
 * *before* the utilities so that the utilities can still
 * override them.
 *
 */
@import './components/global.pcss';
@import './components/typography.pcss';
@import './components/webfonts.pcss';



/**
 * This injects all of Tailwind's utility classes, generated based on your
 * config file.
 *
 */
@tailwind utilities;



/**
 * Include styles for individual pages
 *
 */
@import './pages/homepage.pcss';

/**
 * Include vendor css.
 *
 */
 @import 'vendor.pcss';

And then these .pcss files then get imported into our app.ts via:


// Import our CSS
import '../css/tailwind-base.pcss';
import '../css/app-base.pcss';
import '../css/tailwind-components.pcss';
import '../css/app-components.pcss';
import '../css/tailwind-utilities.pcss';
import '../css/app-utilities.pcss';

Nothing else was changed in our config or setup.

This allows webpack to handle each chunk of imported .pcss separately, so the Tailwind-generated CSS (and importantly the huge utilities.css) only needs to be rebuilt if something that affects it (like the tailwind.config.js) is changed.

Leverage the platform

Changes to any of the .pcss files that we write are rebuilt separately & instantly.

💥

Benchmarking Problem vs. Solution

I did a few informal benchmarks while testing all of this out. I used a 2019 MacBook Pro with 64gb RAM.

For the test, all I did was change the background-color: yellow; to background-color: blue; in the css/components/global.pcss file.

When we do a rebuild using the “Problem Setup”, the webpack-dev-server [WDS] output in the browser’s Developer JavaScript Console looks like this:


[WDS] App updated. Recompiling...
[WDS] App hot update...
[HMR] Checking for updates on the server...
[HMR] Updated modules:
[HMR] - ../src/css/app.pcss
[HMR] App is up to date.

…and it took 11.74s to do this recompile on my 2019 MacBook Pro.

Notice that it rebuild the entire app.pcss here.

When we do a rebuild using the “Solution Setup”, the webpack-dev-server [WDS] output in the browser’s Developer JavaScript Console looks like this:


[WDS] App updated. Recompiling...
[WDS] App hot update...
[HMR] Checking for updates on the server...
[HMR] Updated modules:
[HMR] - ../src/css/app-components.pcss
[HMR] App is up to date.

…and it only took 0.52s to do this HMR rebuild.

Notice that it rebuilt only the app-components.pcss here, and delivered just the diff of it to the browser in a highly efficient manner.

So with our changes, it’s now 22.5x faster

This is a nice gain, and makes the development experience much more enjoyable!

Happy sailing! ≈ 🚀

Running Node.js in Docker for local development

Andrew Welch — Mon, 14 Sep 2020 04:00:00 +0000

Running Node.js in Docker for local development

You don’t need to know Docker to benefit from running local dev Node.js buildchains & apps inside of Docker containers. You get easy onboarding, and less hassle.

Andrew Welch / nystudio107

Devops folks who use Docker often have no desire to use JavaScript, and JavaScript developers often have no desire to do devops.

However, Node.js + Docker really is a match made in heaven.

Hear me out.

You don’t have to learn Docker in depth to reap the benefits from using it.

Whether you’re just using Node.js as a way to run a buildchain to generate frontend assets that uses Grunt / Gulp / Mix / webpack / NPM scripts, or you’re developing full blown Node.js apps, you can benefit from running Node.js in Docker.

In this article, we’ll show you how you can utilize Docker to run your Node.js buildchains & apps & in local dev without needing to know a whole lot about how Docker works.

Unless you install every NPM package you use globally, you already understand the need for containerization

We’ll be running Node.js on-demand in Docker containers that run in local dev only when you’re or building assets with your buildchain or developing your application.

All you’ll need to have installed is Docker itself.

If you’re the TL;DR type, you can check out the example project we used eleventy-blog-base feature/docker branch, and look at the master..feature/docker diff.

Why in the world would I use Docker?

I think this tweet from Adam Wathan is a perfect example of why you would want to use Docker:

“Upgraded Yarn, which broke PHP, which needs Python to reinstall, which needs a new version of Xcode, which needs the latest version of Mojave, which means I need a beer and it’s only noon.” —Adam Wathan

Adam’s certainly not alone, this type of “dependency hell” is something that most developers have descended down into at some point or another.

And having one global install for your entire development environment only gets worse from here:

Updating a dependency like the Node.js version for one app may break other apps
You end up using the oldest possible version of everything to keep the teetering development environment running
Trying new technologies is costly, because your whole development environment is at risk
Updating operating system versions often means putting aside a day (or more) to rebuild your development environment
Getting a new computer similarly means putting aside a day (or more) to rebuild your development environment

Instead of having one monolithic local development environment, using Docker adds a layer of containerization that gives each app you are working on exactly what it needs to run.

Your computer isn’t disposable, but Docker containers are

Is it quicker to just start installing stuff via Homebrew on your computer? Sure.

But people often confuse getting started quickly with speed. What matters more is the speed (and sanity) with which you finish.

So let’s give Docker a whirl.

Docker setup overview

We’re not going to teach you the ins and outs of Docker here; if you want that, check out the An Annotated Docker Config for Frontend Web Development article.

I also highly recommend the Docker Mastery course (if it’s not on sale now, don’t worry, it will be at some point).

Instead, we’re just going to put Docker to work for us. Here’s an overview of how this is going to work:

We’re using make with a Makefile to provide a nice easy way to type our terminal commands (yes, Virginia, dependency managing build systems have been around since 1976).

Then we’re also using a Dockerfile that contains the information needed to build & run our Docker container.

We then leverage NPM scripts in the scripts section of our package.json to run our buildchain / application:

So we’ll type something like:


make npm build

And it will spin up our Node.js Docker container, and run the build script that’s in the scripts section of our package.json.

Since we can put whatever we want in the scripts section of our package.json, we can run whatever we want.

It may seem complicated, but it’s actually relatively simple how it all works

So let’s have a look at how this all works in detail.

Docker setup detail

So as to have a real-world example, what we’re going to do is create a Docker container that builds a website using the popular 11ty static site generator.

Keep in mind that this is just an example, we could be containerizing any Node.js buildchain or app

So what we’ll do is make a clone of the eleventy-base-blog repo:


git clone https://github.com/11ty/eleventy-base-blog

Then we’ll make just one change to the package.json that comes from the repository, adding an install npm script:


{
  "name": "eleventy-base-blog",
  "version": "5.0.2",
  "description": "A starter repository for a blog web site using the Eleventy static site generator.",
  "scripts": {
    "install": "npm install",
    "build": "eleventy",
    "watch": "eleventy --watch",
    "serve": "eleventy --serve",
    "start": "eleventy --serve",
    "debug": "DEBUG=* eleventy"
  },

MAKEFILE

Next we’ll create a Makefile in the project directory that looks like this:


TAG?=12-alpine

docker:
    docker build \
        . \
        -t nystudio107/node:${TAG} \
        --build-arg TAG=${TAG} \
        --no-cache
npm:
    docker container run \
        --name 11ty \
        --rm \
        -t \
        -p 8080:8080 \
        -p 3001:3001 \
        -v `pwd`:/app \
        nystudio107/node:${TAG} \
        $(filter-out $@,$(MAKECMDGOALS))
%:
    @:
# ref: https://stackoverflow.com/questions/6273608/how-to-pass-argument-to-makefile-from-command-line

The way make works is that if you type make, it looks for a Makefile in the current directory for the recipe to make. In our case, we’re just using it as a convenient way to create aliases that are local to a specific project.

So we can use make as a shortcut to run much more complicated commands that aren’t fun to type:

make docker — this will build our Node.js Docker image for us. You need to build a Docker image from a Dockerfile before you can run it as a container
make npm xxx — once built, this will run our Docker container, and execute the NPM script named xxx as listed in the package.json. For instance, make npm build will run the build script

The TAG?=12-alpine line provides a default Node.js tag to use when building the image, with the number part of it being the Node.js version (“alpine” is just a very slimmed down Linux distro).

If we wanted, say, Node.js 14, we could just change that to be TAG?=14-alpine and do a make docker or we could pass it in via the command line for a quick temporary change: make docker TAG=14.alpine

It’s just that easy to switch the Node.js version

While it’s not important that you learn the syntax of make, let’s have a look at the two commands we have in our Makefile.

The </kbd> you see in the Makefile is just a way to allow you to continue a shell command on the next line, for readability reasons.

docker: # the command alias, so we run it via make docker
- docker build </kbd> # Build a Docker container from a Dockerfile
- . </kbd> # …in the current directory
- -t nystudio107/node:${TAG} </kbd> # tag the image with nystudio107/node:12-alpine (or whatever ${TAG} is)
- --build-arg TAG=${TAG} </kbd> # pass in our ${TAG} variable as an argument to the Dockerfile
- --no-cache # Do not use cache when building the image
npm: # the command alias, so we run it via make npm xxx, where xxx is the npm script to run
- docker container run </kbd> # Run a Docker container from an image
- --name 11ty </kbd> # name the container instance “11ty”
- --rm </kbd> # remove the container when it exits
- -t </kbd> # provide a terminal, so we can have pretty colored text
- -p 8080:8080 </kbd> # map port 8080 from inside of the container to port 8080 to serve our hot reloaded files from http://localhost:8080
- -p 3001:3001 </kbd> # map port 3001 from inside of the container to port 3001 to serve the BrowserSync UI from http://localhost:3001
- -v pwd:/app </kbd> # mount a volume from the current working directory to /app inside of the Docker container
- nystudio107/node:${TAG} </kbd> # use the Docker image tagged with nystudio107/node:12-alpine (or whatever ${TAG} is)
- $(filter-out $@,$(MAKECMDGOALS)) # a fancy way to pass any additional arguments from the command line down to Docker

We do the port mapping to allow 11ty’s hot reloading to work during development.

DOCKERFILE

Now we’ll create a Dockerfile in the project root directory:


ARG TAG=12-alpine
FROM node:$TAG

WORKDIR /app

CMD ["build"]

ENTRYPOINT ["npm", "run"]

Our Dockerfile is pretty small, but let’s break down what it’s doing:

ARG TAG=12-alpine — Set the build argument TAG to default to 12-alpine. If a --build-arg is provided, it’ll override this so you can specify other Node.js version

FROM node:$TAG — Designate which base image our container will be built from

WORKDIR /app — Set the directory where the commands in the Dockerfile are run to /app

CMD ["build"] — Set the default command to build

ENTRYPOINT ["npm", "run"] — When the container is spun up, it’ll execute npm run xxx where xxx is an argument passed in via the command line, or it’ll fall back on the default build command

Taking Docker for a spin

So let’s take Docker for a spin on this project. First we’ll make sure we’re in the project root directory, and build our Docker container with make docker:


❯ make docker
docker build \
        . \
        -t nystudio107/node:12-alpine \
        --build-arg TAG=12-alpine \
        --no-cache
Sending build context to Docker daemon 438.8kB
Step 1/5 : ARG TAG=12-alpine
Step 2/5 : FROM node:$TAG
 ---> 18f4bc975732
Step 3/5 : WORKDIR /app
 ---> Running in 6f5191fe0128
Removing intermediate container 6f5191fe0128
 ---> 29e9346463f9
Step 4/5 : CMD ["build"]
 ---> Running in 38fb3db1e3a3
Removing intermediate container 38fb3db1e3a3
 ---> 22806cd1f11e
Step 5/5 : ENTRYPOINT ["npm", "run"]
 ---> Running in cea25ee21477
Removing intermediate container cea25ee21477
 ---> 29758f87c56c
Successfully built 29758f87c56c
Successfully tagged nystudio107/node:12-alpine

Next let’s execute the install script we added to our package.json via make npm install. This runs an npm install, which we only need to do once to get our node_module dependencies installed:


❯ make npm install
docker container run \
        --name 11ty \
        --rm \
        -t \
        -p 8080:8080 \
        -p 3001:3001 \
        -v `pwd`:/app \
        nystudio107/node:12-alpine \
        install

> eleventy-base-blog@5.0.2 install /app
> npm install

npm WARN deprecated core-js@2.6.11: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.

> core-js@2.6.11 postinstall /app/node_modules/core-js
> node -e "try{require('./postinstall')}catch(e){}"

> ejs@2.7.4 postinstall /app/node_modules/ejs
> node ./postinstall.js

Thank you for installing EJS: built with the Jake JavaScript build tool (https://jakejs.com/)

npm WARN lifecycle eleventy-base-blog@5.0.2~install: cannot run in wd eleventy-base-blog@5.0.2 npm install (wd=/app)
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.1.2 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 437 packages from 397 contributors and audited 439 packages in 30.004s

15 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

Finally, let’s fire up a hot reloading development server, and build our site via make npm serve. This is the only step you’ll normally need to do in order to work on your site:


❯ make npm serve
docker container run \
        --name 11ty \
        --rm \
        -t \
        -p 8080:8080 \
        -p 3001:3001 \
        -v `pwd`:/app \
        nystudio107/node:12-alpine \
        serve

> eleventy-base-blog@5.0.2 serve /app
> eleventy --serve

Writing _site/feed/feed.xml from ./feed/feed.njk.
Writing _site/sitemap.xml from ./sitemap.xml.njk.
Writing _site/feed/.htaccess from ./feed/htaccess.njk.
Writing _site/feed/feed.json from ./feed/json.njk.
Writing _site/posts/fourthpost/index.html from ./posts/fourthpost.md.
Writing _site/posts/thirdpost/index.html from ./posts/thirdpost.md.
Writing _site/posts/firstpost/index.html from ./posts/firstpost.md.
Writing _site/404.html from ./404.md.
Writing _site/posts/index.html from ./archive.njk.
Writing _site/posts/secondpost/index.html from ./posts/secondpost.md.
Writing _site/page-list/index.html from ./page-list.njk.
Writing _site/tags/second-tag/index.html from ./tags.njk.
Writing _site/index.html from ./index.njk.
Writing _site/tags/index.html from ./tags-list.njk.
Writing _site/about/index.html from ./about/index.md.
Writing _site/tags/another-tag/index.html from ./tags.njk.
Writing _site/tags/number-2/index.html from ./tags.njk.
Copied 3 files / Wrote 17 files in 0.74 seconds (43.5ms each, v0.11.0)
Watching…
[Browsersync] Access URLs:
 -----------------------------------
       Local: http://localhost:8080
    External: http://172.17.0.2:8080
 -----------------------------------
          UI: http://localhost:3001
 UI External: http://localhost:3001
 -----------------------------------
[Browsersync] Serving files from: _site

We can just point our web browser at http://localhost:8080 and we’ll see our website up and running:

If we make any changes, they’ll automatically be hot reloaded in the browser, so away we go!

“Yeah, so what?” you say?

Realize that with the Makefile and Dockerfile in place, we can hand our project off to someone else and onboarding becomes bliss:

We won’t need to care what version of Node.js they have installed
They don’t even have to have Node.js installed at all, in fact

Additionally, we can come back to the project at any time and:

The project is guaranteed to work, since the devops needed to run it is “shrink wrapped” around it
We can easily switch Node.js versions without affecting anything else

No more nvm. No more n. No more hassles switching Node.js versions.

Containerization as a way forward

Next time you have the opportunity to start fresh with a new computer or a new operating system, consider taking it.

Don’t install Homebrew.

Don’t install Node.js.

Don’t install dozens of packages.

Instead, take the containerization challenge and just install Docker, and run everything you need from containers

I think you may be pleasantly surprised at how it’ll make your life easier.

Atomic Deployments Without Tears

Andrew Welch — Tue, 30 Jun 2020 04:00:00 +0000

Atomic Deployments Without Tears

Learn how to use atomic deployments to automatically, deterministically, and safely deploy changes to your website using Continuous Integration (CI) tools

Andrew Welch / nystudio107

Once you have developed a website, you then have to face the challenge of deploying that website to a live production environment where the world can see it.

Back in the mean old days, this meant firing up an FTP client to upload the website to a remote server.

This type of “cowboy” deployment is not the best choice

The reason doing it this way isn’t so great is that it’s a manual, error-prone process. Many services in the form of Continuous Integration tools have sprung up to make the process much easier for you, and importantly, automated.

Let computers do the boring, repetitious work that they are good at

This article will show you how you can leverage the CI tool buddy.works to atomically deploy your Craft CMS websites like a pro.

However, the concepts presented here are universal, so if you’re using some other CI tool or CMS/platform, that’s totally fine. Read on!

Anatomy of a web project

Let’s have a look at what a typical project setup might look like:

We work on the project in our local development environment, whether individually or with a team of other developers. We push our code changes up to a git repository in the cloud.

Local development is “where the magic happens”

The git repository is where all source code is kept, and allows us to work with multiple people or multiple revisions without fear. This git repo can be hosted via GitHub, GitLab, or any number of other places.

We may also be using cloud file storage such as Amazon S3 as a place to store the client-uploaded content, as described in the Setting Up AWS S3 Buckets + CloudFront CDN for your Assets article.

A general workflow for code is:

Push code changes from local development up to your git repo
Pull code changes down from your git repo to your live production or staging servers

If you’re working on a team or in multiple environments, you may also be pulling code down to your local development environment from your git repo as well, to stay in sync with changes other people have made.

Non-Atomic Deployment Flow

But how do you pull code changes down to your live production or staging servers?

Deployment is getting your code from your local development environment to your live production server.

A simple method (dubbed the #YOLO method by Matthew Stein) could be to trigger a shell script when we push to the master branch of our project’s git repo:


cd /home/forge/devmode.fm
git pull origin master
cd /home/forge/devmode.fm/cms
composer install --no-interaction --prefer-dist --optimize-autoloader
echo "" | sudo -S service php7.1-fpm reload

In my case, this is how I was previously doing deployments for the devMode.fm website: it’s just a shell script that’s executed when a webhook that is triggered when we push to the master branch of our git repo.

Line by line, here’s what this shell script does:

Change directories to the root directory of our project
Pull down the latest changes from the master branch of the project’s git repo
Change directories to the root of the Craft CMS project
Run composer install to install the latest composer dependencies specified in the composer.lock file
Restart php-fpm to clear our opcache What could possibly go wrong?

For a hobby project site, this is totally fine.

But there are downsides to doing it this way:

The deployment is done in multiple steps
The work happens on the production server, which is also serving frontend requests
The entire git repo is deployed to the server, when only part of it is actually needed on the production server
If there’s a problem with the deploy, the site could be left broken
Any website CSS/JavaScript assets need to built in local development, and checked into the git repo

You might notice that there are a number of steps listed, and some of the steps such as git pull origin master and composer install can be quite lengthy processes.

And we’re doing them in situ, so if someone visits the website when we’re in the middle of pulling down our code, or Composer is in the middle of installing PHP packages… that person may see errors on the frontend.

The fact that there are multiple, lengthy steps in this process makes it a non-atomic deployment.

Atomic Deployment Flow

So while we have an automated deployment method, it’s a bit fragile in that there’s a period of time during which people visiting our website may see it broken. To solve this, let’s introduce how an atomic deployment would work.

An atomic deployment is just a fancy nomenclature for a deployment that happens in such a way that the switch to the new version of the site as a single — or atomic — step.

This allows for zero downtime, and no weirdness in partially deployed sites.

An atomic deployment is a magician’s finger-snap and “tada”!

We’re going to set up our atomic deployments using buddy.works, which is a tool that I’ve chosen because it is easy to use, but also very powerful.

There’s a free tier that you can use for up to 5 project while you’re testing it out, you can give it a whirl, or you can use some other deployment tool like Envoyer (and there are many others). The principle is the same.

Here’s what an atomic deployment setup might look like:

Note that we’re still doing the same work as in our non-atomic deployment, but we’re changing where and how that work is done.

This nicely solves all of the downsides we noted in our non-atomic deployment:

The switchover to the newly deployed website code happens in a single atomic step
No work is done on the live production server other than deploying the files
Only the parts of the project needed to serve the website are deployed
If there’s a problem with the build, it never reaches the server
Any website CSS/JavaScript assets are built “in the cloud”

So this is all wonderful, but how does it work? Continue on, dear reader!

Atomic Deployments Under the Hood

We’ll get to the actual setup in a bit, but first I think it’s instructive to see how it actually works under the hood.

As usual, we’ll be using the devMode.fm website as our guinea pig, the source code of which is available in the nystudio107/devmode repo.

Our project root directory looks like this on our production server:


forge@nys-production ~/devmode.fm $ ls -Al
total 32
lrwxrwxrwx 1 forge forge 49 Jun 28 19:08 current -> releases/33a5a7f984521811c5db597c7eef1c76c00d48e2
drwxr-xr-x 7 forge forge 4096 Jun 27 01:39 deploy-cache
-rw-rw-r-- 1 forge forge 2191 Jun 22 18:14 .env
drwxrwxr-x 12 forge forge 4096 Jun 28 19:08 releases
drwxrwxr-x 5 forge forge 4096 Jun 22 18:11 storage
drwxrwxr-x 2 forge forge 4096 Jun 26 12:30 transcoder

This may look a little foreign to you, but bear with me, you’ll get it!

The deploy-cache/ directory is where files are stored as they are being uploaded to the server. In our case, it looks like this:


forge@nys-production ~/devmode.fm $ ls -Al deploy-cache/
total 328
-rw-r--r-- 1 forge forge 2027 Jun 26 22:46 composer.json
-rw-r--r-- 1 forge forge 287399 Jun 27 01:39 composer.lock
drwxr-xr-x 4 forge forge 4096 Jun 27 01:39 config
-rwxr-xr-x 1 forge forge 577 Jun 23 07:25 craft
-rw-r--r-- 1 forge forge 330 Jun 23 07:25 craft.bat
-rw-r--r-- 1 forge forge 1582 Jun 23 07:25 example.env
drwxr-xr-x 3 forge forge 4096 Jun 23 07:25 modules
drwxr-xr-x 11 forge forge 4096 Jun 23 07:25 templates
drwxr-xr-x 60 forge forge 4096 Jun 27 01:40 vendor
drwxr-xr-x 5 forge forge 4096 Jun 28 19:08 web

This should look pretty familiar to you if you’re a Craft CMS developer, it’s the project root for the actual Craft CMS project. Check out the Setting up a New Craft CMS 3 Project article for more information on that.

Since this is a cache directory, the contents can be deleted without any ill effect, other than our next deployment will be slower since it’ll need to be done from scratch.

Next let’s have a look at the releases/ directory:


forge@nys-production ~/devmode.fm $ ls -Al releases/
total 48
drwxr-xr-x 7 forge forge 4096 Jun 27 14:17 2c8eef7c73f20df9d02f6f071656331ca9e08eb0
drwxr-xr-x 7 forge forge 4096 Jun 28 19:08 33a5a7f984521811c5db597c7eef1c76c00d48e2
drwxrwxr-x 7 forge forge 4096 Jun 26 22:48 42372b0cd7a66f98d7f4dc83d8d99c4d9a0fb1f6
drwxrwxr-x 7 forge forge 4096 Jun 27 01:43 7b3d57dfedf5bf275aeddc6d799e3264e02d2b88
drwxrwxr-x 8 forge forge 4096 Jun 26 21:21 8c2448d252651b8cb0d69a72e327dac3541c9ba9
drwxr-xr-x 7 forge forge 4096 Jun 27 14:08 9b5c8c7cf6a7111220b66d21d811f8e5a1800507
drwxrwxr-x 8 forge forge 4096 Jun 23 08:16 beaef13f5bda9d7c2bb0e88b300f68d3b663528e
drwxrwxr-x 8 forge forge 4096 Jun 26 21:26 c56c13127b4a5ff779a155a211c07f604a4dcf8b
drwxrwxr-x 7 forge forge 4096 Jun 27 14:04 ce831a76075f57ceff8822641944e255ab9bf556
drwxrwxr-x 8 forge forge 4096 Jun 23 07:57 ebba675ccd2bb372ef82795f076ffd933ea14a31

Here we see 10 really weirdly named directories. The names here don’t really matter (they are automatically generated hashes), but what does matter is that each one of these directories contains a full deployment of your website.

You can set how many of these directories should be kept on the server, in my case I have it set to 10.

If you look carefully at the current symlink:


lrwxrwxrwx 1 forge forge 49 Jun 28 19:08 current -> releases/33a5a7f984521811c5db597c7eef1c76c00d48e2

…you’ll see that it actually points to the current deployment in the releases/ directory (notice that the hash-named directory it points to has the latest modification date on it, too).

So when a deployment happens:

Files are synced into the deploy-caches/ directory (we’ll get into this more later)
Then those files are copied from deploy-caches/ directory to a hash-named directory in the releases/ directory
After everything is done, the current symlink is updated to point to the latest deployment

That’s it! That’s the atomic part: the changing of the current symlink is the single atomic operation that makes that version of the website live.

We just have to make sure that our web server root path contains the symlink, so we can swap out where it points to as needed:


    root /home/forge/devmode.fm/current/web;

If you ever encounter a regression, you can roll your website back to a previous revision just by changing the current symlink.

Also note that we have storage/ and transcoder/ directories in our project root, as well as a .env file.

These are all directories & files that we want to persist between and be shared by each atomic deployment. Since each deployment is a clean slate, we just move anything we need to keep persistent into the root directory, and symlink to them from each deployment.

The .env file is something you’ll have to create yourself manually, using the example.env as a guide.

The storage/ directory is Craft’s runtime storage directory. We keep this as a persistent directory so that log files and other Craft runtime files can persist across atomic deployments.

The transcoder/ directory is used to store the transcoded audio for the podcast, as created by our Transcoder plugin. It’s very project specific, so you’re unlikely to need it in your projects.

Let’s have a look at the current deployment in the releases/ directory:


forge@nys-production ~/devmode.fm $ ls -Al releases/33a5a7f984521811c5db597c7eef1c76c00d48e2/
total 320
-rw-r--r-- 1 forge forge 2027 Jun 29 14:10 composer.json
-rw-r--r-- 1 forge forge 287399 Jun 29 14:10 composer.lock
drwxr-xr-x 4 forge forge 4096 Jun 29 14:10 config
-rwxr-xr-x 1 forge forge 577 Jun 29 14:10 craft
-rw-r--r-- 1 forge forge 330 Jun 29 14:10 craft.bat
lrwxrwxrwx 1 forge forge 27 Jun 29 14:10 .env -> /home/forge/devmode.fm/.env
-rw-r--r-- 1 forge forge 1582 Jun 29 14:10 example.env
drwxr-xr-x 3 forge forge 4096 Jun 29 14:10 modules
lrwxrwxrwx 1 forge forge 30 Jun 29 14:10 storage -> /home/forge/devmode.fm/storage
drwxr-xr-x 11 forge forge 4096 Jun 29 14:10 templates
drwxr-xr-x 60 forge forge 4096 Jun 29 14:10 vendor
drwxr-xr-x 6 forge forge 4096 Jun 29 14:11 web

N.B.: this is the exact same as doing ls -Al current/ since the current symlink points to this latest deployment.

Here we can see the current deployment root, with the .env & storage aliases in place, pointing back to the persistent files/directories in our project root.

Something that might not be immediately apparent is that we’re only deploying part of what is in our project git repo. The git repo root looks like this:


❯ ls -Al
total 80
-rw-r--r-- 1 andrew staff 868 Jun 22 17:24 .gitignore
-rw-r--r-- 1 andrew staff 1828 Feb 18 10:22 CHANGELOG.md
-rw-r--r-- 1 andrew staff 1074 Feb 4 09:54 LICENSE.md
-rw-r--r-- 1 andrew staff 7461 Jun 29 09:03 README.md
-rw-r--r-- 1 andrew staff 5094 Jun 27 14:15 buddy.yml
drwxr-xr-x 16 andrew staff 512 Jun 27 14:06 cms
-rwxr-xr-x 1 andrew staff 2064 Mar 17 16:37 docker-compose.yml
drwxr-xr-x 10 andrew staff 320 Feb 17 16:58 docker-config
drwxr-xr-x 7 andrew staff 224 Mar 17 16:37 scripts
drwxr-xr-x 12 andrew staff 384 Feb 17 15:51 src
lrwxr-xr-x 1 andrew staff 47 Jun 27 14:06 tsconfig.json -> docker-config/webpack-dev-devmode/tsconfig.json
lrwxr-xr-x 1 andrew staff 45 Jun 27 14:06 tslint.json -> docker-config/webpack-dev-devmode/tslint.json

So instead of deploying all of the source code and build tools that aren’t needed to serve the website (they are only needed to build it), we instead just deploy just what’s in the cms/ directory.

Nice.

Now that we know how it works under the hood, let’s create the atomic deployment pipeline!

Step 1: Creating a new project

We’ll go step by step through how to build a simple but effective atomic deployment with buddy.works.

The deployment pipeline we’re going to set up will:

Automatically deploy to our production server whenever we push to the master branch of our git repo
Utilize the Docker containers we already use for local development for building the website in the cloud, as discussed in the An Annotated Docker Config for Frontend Web Development article
Build all of our CSS & JavaScript assets via the webpack setup discussed in the An Annotated webpack 4 Config for Frontend Web Development article
Efficiently sync just the changed files down to our live production server
Do an atomic deployment by swapping the current site
Prep Craft CMS by running all migrations, syncing Project Config, and clearing all caches

So let’s get down to it

After logging into buddy.works, make sure that you’ve linked buddy.works to your git repo provider (such as GitHub, GitLab, etc.). It needs this to allow you to choose a git repo for your atomic deployment setup, and also to be notified when you push code to that git repo.

You can configure this and other settings by clicking on your user icon in the upper-right corner of the screen, and choosing Manage your project.

Once that’s all set, click on New Project from your Dashboard:

Next click on the Add a new pipeline button to create a new deployment pipeline for this project. A pipeline is just a series of instructions to execute in sequence.

Set the Name to Build & Deploy to Production , set Trigger Mode to On push and then set the Trigger to Single Branch and master (or whatever the name of your primary git repo branch is).

Then click on + Site URL, Currently deployed revision, Clone depth & Visibility to display more options, and set the Target website URL to whatever your live production website URL is.

We won’t be changing anything else here, so click on Add a new pipeline to create a new empty pipeline (you can have as many pipelines as you like per project).

Step 2: Setting Variables

Before we add any actions to our pipeline, we’re going to set some environment variables for use in the buddy.works build pipeline.

Click on the Edit pipeline settings link on the right, then click on Variables :

We’re adding these variables to our pipeline to make it easier to build our individual actions, and make our pipeline generic so it can be used with any project.

Add the following key/value pair variables by clicking on Add a new variable , changing them to suit your project (by convention, environment variables are SCREAMING_SNAKE_CASE):

PROJECT_SHORTNAME — devmode — a short name for the project with no spaces or punctuation; it’s used to create working directories in the buddy.works containers
PROJECT_URL — https://devmode.fm — a URL to your live production website
REMOTE_PROJECT_ROOT — /home/forge/devmode.fm — a path to the root directory of the project on the server
REMOTE_SSH_HOST — devmode.fm — the host name that should be used to ssh into your server
REMOTE_SSH_USER — forge — the user name that should be used to ssh into your server

N.B.: the buddy.works docs say to use the variables in a ${VARIABLE_NAME} format, but you can also use them just as $VARIABLE_NAME (in fact the latter is how they are auto-completed for you).

These variables are defined inside of the pipeline, but you can also have variables that are project-wide, as well as workspace-wide in buddy.works.

Step 3: Execute: webpack build

Now that our variables are all set, click on Actions and then click on the Add the first action button.

Type webpack into the search field to find the Webpack action, and click on it.

We’re assuming you are using the webpack setup described in the An Annotated webpack 4 Config for Frontend Web Development article and the Docker setup described in the An Annotated Docker Config for Frontend Web Development article.

Add the following script under the Run tab; it installs our npm packages via npm ci and then executes webpack to build our build:


cd docker-config/webpack-dev-devmode
npm ci
npm run build

You can change this to be whatever you need to execute your CSS & JavaScript build, if you’re using something other than the aforementioned setups.

Next click on the Environment tab, and change the Image to our custom webpack-dev-base that we used in the An Annotated Docker Config for Frontend Web Development article, since it has everything we need in it for building our CSS & JavaScript:

This Environment tab allows you to pick any Docker image you like — public or private — to use when running you webpack build in the cloud. The default is an old (but official) Node 6 image at the time of this writing.

Clicking on the Action tab allows you to change the name of the action; change it to: Execute: webpack build.

Step 4: Execute: composer install

Next up we’ll create another action to our pipeline by clicking on the + icon below the Execute: webpack build action.

Type php into the search field to find the PHP action, and click on it.

We’re assuming you are using the Docker setup described in the An Annotated Docker Config for Frontend Web Development article.

Add the following script under the Run tab; it changes directories to the cms/ directory, and then runs composer install with some flags:


cd cms
composer install --no-scripts --no-interaction --prefer-dist --optimize-autoloader --ignore-platform-reqs

You can change this to be whatever you need to execute to install your Composer packages, if you’re using something other than the aforementioned setup.

Next click on the Environment tab, and change the Image to our custom php-dev-base that we used in the An Annotated Docker Config for Frontend Web Development article, since it has everything we need for our PHP application:

This Environment tab allows you to pick any Docker image you like — public or private — to use when running your composer install in the cloud. The default is php 7.4 image at the time of this writing.

Still on the Environment tab, scroll down to CUSTOMIZE ENVIRONMENT and paste this in:


echo "memory_limit=-1" >> /usr/local/etc/php/conf.d/buddy.ini
apt-get update && apt-get install -y git zip
curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
# php ext pdo_pgsql
docker-php-ext-install pdo_pgsql pgsql

This script runs inside the Docker container to customize the environment by setting PHP to have no memory limit, installing Composer, and then installing some Postgres php extensions. If you’re using MySQL, you’d change it to:


# php ext pdo_mysql
docker-php-ext-install pdo_mysql mysql

In actuality, it doesn’t matter, because we’re not even doing anything with the database on deploy currently.

Clicking on the Action tab allows you to change the name of the action; change it to: Execute: composer install.

Step 3: Rsync files to production

Now that we have our updated website code from our git repo, our built CSS & JavaScript assets, and all of our Composer packages in the Docker container in the cloud, we need to deploy them to our production server.

To do this, we’re going to use rsync to sync only the files that have changed to our deploy-cache/ directory.

Create another action to our pipeline by clicking on the + icon below the Execute: composer install action.

Type rsync into the search field to find the RSync action, and click on it.

Here we’ve chosen to synchronize just the cms/ directory of our project with the deploy-caches/ directory on our live production server.

To allow buddy.works to access our live production server, we have to provide it with how to connect to our server. Fortunately, we can use the environment variables set up in step #1.

So set Hostname & Port to $REMOTE_SSH_HOST, Login to $REMOTE_SSH_USER, and Authentication mode to Buddy workspace key.

We’re using ssh keys here because the provisioner I use, Laravel Forge, disables password-based auth by default as a security best practice.

If you’re going to use Buddy workspace key too, you’ll need to ssh into your live production server, and run the code snippet. This will add Buddy’s workspace key to your live production server’s list of hosts that are authorized to connect to it.

Then set Remote path to $REMOTE_PROJECT_ROOT/deploy-cache. This tells the rsync action what directory on the live production server should be synced with the cms/ directory in our buddy.works Docker container in the cloud.

Finally, check the following:

✓ Compress file data during the transer
✓ Archive mode
✓ Delete extraneous files
✓ Recurse into directories

Using Rsync for our deployment allows it to be very smart about deploying only files that have actually changed, and also compress the files before they are transferred over the wire.

N.B.: In the Ignore paths tab, you can add any directories you want ignored during the sync

Clicking on the Action tab allows you to change the name of the action; change it to: Rsync files to production.

Step 4: Atomic Deploy

Finally, we’re getting to the actual atomic deployment!

Create another action to our pipeline by clicking on the + icon below the Rsync files to production action.

This time we’re going to click on Templates and then click on Atomic Deployment. You’ll see some documentation on what the Atomic Deployment template does; click on Configure this template :

For Source , click on Pipeline Filesystem and leave Source path set to /

Set Hostname & Port to $REMOTE_SSH_HOST, Login to $REMOTE_SSH_USER, and Authentication mode to Buddy workspace key just like we did in step #3.

Again we’re using the same Buddy workspace key we used in step #3, so we won’t need to re-add this key to our live production server.

Leave Remote path set to ~/ and the double-negative Don’t delete files set to Off. You can also configure how many releases to keep on your server via How many old releases should be kept.

Then click on Add this action.

We’re not quite done with this action though. Click on it again in the list of pipeline actions to edit it, and you’ll see some shell code the template added for us under RUN SSH COMMANDS :


if [-d "releases/$BUDDY_EXECUTION_REVISION"] && ["$BUDDY_EXECUTION_REFRESH" = "true"];
then
 echo "Removing: releases/$BUDDY_EXECUTION_REVISION"
 rm -rf releases/$BUDDY_EXECUTION_REVISION;
fi
if [! -d "releases/$BUDDY_EXECUTION_REVISION"];
then
 echo "Creating: releases/$BUDDY_EXECUTION_REVISION"
 cp -dR deploy-cache releases/$BUDDY_EXECUTION_REVISION;
fi
echo "Linking current to revision: $BUDDY_EXECUTION_REVISION"
rm -f current
ln -s releases/$BUDDY_EXECUTION_REVISION current
echo "Removing old releases"
cd releases && ls -t | tail -n +11 | xargs rm -rf

This is the code that handles creating the hash-named revision directories, copying files from the deploy-cache/ directory, updating the current symlink, and trimming old releases.

You needn’t grok all that it’s doing, we’re just going to make a small addition to it to create and symlink our persistent directories & files:


if [-d "releases/$BUDDY_EXECUTION_REVISION"] && ["$BUDDY_EXECUTION_REFRESH" = "true"];
then
 echo "Removing: releases/$BUDDY_EXECUTION_REVISION"
 rm -rf releases/$BUDDY_EXECUTION_REVISION;
fi
if [! -d "releases/$BUDDY_EXECUTION_REVISION"];
then
 echo "Creating: releases/$BUDDY_EXECUTION_REVISION"
 cp -dR deploy-cache releases/$BUDDY_EXECUTION_REVISION;
fi
echo "Creating: persistent directories"
mkdir -p storage
mkdir -p transcoder
echo "Symlinking: persistent files & directories"
ln -nfs $REMOTE_PROJECT_ROOT/.env $REMOTE_PROJECT_ROOT/releases/$BUDDY_EXECUTION_REVISION
ln -nfs $REMOTE_PROJECT_ROOT/storage $REMOTE_PROJECT_ROOT/releases/$BUDDY_EXECUTION_REVISION
ln -nfs $REMOTE_PROJECT_ROOT/transcoder $REMOTE_PROJECT_ROOT/releases/$BUDDY_EXECUTION_REVISION/web
echo "Linking current to revision: $BUDDY_EXECUTION_REVISION"
rm -f current
ln -s releases/$BUDDY_EXECUTION_REVISION current
echo "Removing old releases"
cd releases && ls -t | tail -n +11 | xargs rm -rf

Here we’re ensuring that the storage/ and transcoder/ directories exist, and then we’re symlink’ing them and our .env file from their persistent location in the project root in the appropriate places in the deployed website.

The transcoder/ directory is used to store the transcoded audio for the podcast, as created by our Transcoder plugin. It’s very project specific, so you’re unlikely to need it in your projects.

Clicking on the Action tab allows you to change the name of the action; change it to: Atomic deploy.

Step 5: Prep Craft CMS

Create another action to our pipeline by clicking on the + icon below the Atomic deploy action.

Technically this action could be combined with Step #4, but logically they do different things, so keeping them separate seems appropriate.

Type ssh into the search field to find the SSH action, and click on it.

Under RUN SSH COMMANDS we have the following shell script:


# Ensure the craft script is executable
chmod a+x craft
# Run pending migrations, sync project config, and clear caches
./craft migrate/all
./craft project-config/sync
./craft clear-caches/all

This ensures that all of the migrations are run, Project Config is synced, and all caches are cleared on every deploy.

Set Hostname & Port to $REMOTE_SSH_HOST, Login to $REMOTE_SSH_USER, and Authentication mode to Buddy workspace key just like we did in steps #3 & #4.

Again we’re using the same Buddy workspace key we used in steps #3 & #4, so we won’t need to re-add this key to our live production server.

Then set Working directory to $REMOTE_PROJECT_ROOT/deploy-cache to tell buddy.works which directory should be current when the script above is run.

Clicking on the Action tab allows you to change the name of the action; change it to: Prep Craft CMS.

Step #6: Send notification to nystudio107 channel

Create another action to our pipeline by clicking on the + icon below the Prep Craft CMS action.

This optional action sends a notification on deploy to the #nystudio107 channel on the private nystudio107 Slack.

Type slack into the search field to find the Slack action, and click on it.

You’ll have to grant buddy.works access to your Slack by auth’ing it, then set the Send message to:


[#$BUDDY_EXECUTION_ID] $BUDDY_PIPELINE_NAME execution by $BUDDY_EXECUTION_REVISION_COMMITTER_NAME .

Or customize it however you like, and configure the Integration and Target channel as appropriate for your Slack.

Clicking on the Action tab allows you to change the name of the action; change it to: Send notification to nystudio107 channel.

The Golden Road (to unlimited deployment)

If all of this setup seems like a whole lot of work to you, it’s really not so bad once you are familiar with the buddy.works GUI.

However, I also have good news for you. There’s a reason why we used environment variables: buddy.works allows you to save your entire configuration out to a buddy.yml file.

Go to your project view, and click on YAML configuration: OFF and you’ll see:

If you have a buddy.yml in your project root and switch your project to YAML configuration: ON, then you’ll get your pipelines configured for you automatically by the buddy.yml file:


- pipeline: "Build & Deploy to Production"
  trigger_mode: "ON_EVERY_PUSH"
  ref_name: "master"
  ref_type: "BRANCH"
  target_site_url: "https://devmode.fm/"
  trigger_condition: "ALWAYS"
  actions:
    - action: "Execute: webpack build"
      type: "BUILD"
      working_directory: "/buddy/$PROJECT_SHORTNAME"
      docker_image_name: "nystudio107/webpack-dev-base"
      docker_image_tag: "latest"
      execute_commands:
        - "cd docker-config/webpack-dev-devmode"
        - "npm ci"
        - "npm run build"
      volume_mappings:
        - "/:/buddy/$PROJECT_SHORTNAME"
      trigger_condition: "ALWAYS"
      shell: "BASH"
    - action: "Execute: composer install"
      type: "BUILD"
      working_directory: "/buddy/$PROJECT_SHORTNAME"
      docker_image_name: "nystudio107/php-dev-base"
      docker_image_tag: "latest"
      execute_commands:
        - "cd cms"
        - "composer install --no-scripts --no-interaction --prefer-dist --optimize-autoloader --ignore-platform-reqs"
      setup_commands:
        - "echo \"memory_limit=-1\" >> /usr/local/etc/php/conf.d/buddy.ini"
        - "apt-get update && apt-get install -y git zip"
        - "curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer"
        - "# php ext pdo_mysql"
        - "docker-php-ext-install pdo_pgsql pgsql"
      volume_mappings:
        - "/:/buddy/$PROJECT_SHORTNAME"
      trigger_condition: "ALWAYS"
      shell: "BASH"
    - action: "Rsync files to production"
      type: "RSYNC"
      local_path: "cms/"
      remote_path: "$REMOTE_PROJECT_ROOT/deploy-cache"
      login: "$REMOTE_SSH_USER"
      host: "$REMOTE_SSH_HOST"
      port: "22"
      authentication_mode: "WORKSPACE_KEY"
      archive: true
      delete_extra_files: true
      recursive: true
      compress: true
      deployment_excludes:
        - "/.git/"
      trigger_condition: "ALWAYS"
    - action: "Atomic deploy"
      type: "SSH_COMMAND"
      working_directory: "$REMOTE_PROJECT_ROOT"
      login: "$REMOTE_SSH_USER"
      host: "$REMOTE_SSH_HOST"
      port: "22"
      authentication_mode: "WORKSPACE_KEY"
      commands:
        - "if [-d \"releases/$BUDDY_EXECUTION_REVISION\"] && [\"$BUDDY_EXECUTION_REFRESH\" = \"true\"];"
        - "then"
        - " echo \"Removing: releases/$BUDDY_EXECUTION_REVISION\""
        - " rm -rf releases/$BUDDY_EXECUTION_REVISION;"
        - "fi"
        - "if [! -d \"releases/$BUDDY_EXECUTION_REVISION\"];"
        - "then"
        - " echo \"Creating: releases/$BUDDY_EXECUTION_REVISION\""
        - " cp -dR deploy-cache releases/$BUDDY_EXECUTION_REVISION;"
        - "fi"
        - "echo \"Creating: persistent directories\""
        - "mkdir -p storage"
        - "echo \"Symlinking: persistent files & directories\""
        - "ln -nfs $REMOTE_PROJECT_ROOT/.env $REMOTE_PROJECT_ROOT/releases/$BUDDY_EXECUTION_REVISION"
        - "ln -nfs $REMOTE_PROJECT_ROOT/storage $REMOTE_PROJECT_ROOT/releases/$BUDDY_EXECUTION_REVISION"
        - "ln -nfs $REMOTE_PROJECT_ROOT/transcoder $REMOTE_PROJECT_ROOT/releases/$BUDDY_EXECUTION_REVISION/web"
        - "echo \"Linking current to revision: $BUDDY_EXECUTION_REVISION\""
        - "rm -f current"
        - "ln -s releases/$BUDDY_EXECUTION_REVISION current"
        - "echo \"Removing old releases\""
        - "cd releases && ls -t | tail -n +11 | xargs rm -rf"
      trigger_condition: "ALWAYS"
      run_as_script: true
      shell: "BASH"
    - action: "Prep Craft CMS"
      type: "SSH_COMMAND"
      working_directory: "$REMOTE_PROJECT_ROOT/current"
      login: "$REMOTE_SSH_USER"
      host: "$REMOTE_SSH_HOST"
      port: "22"
      authentication_mode: "WORKSPACE_KEY"
      commands:
        - "# Ensure the craft script is executable"
        - "chmod a+x craft"
        - "# Run pending migrations, sync project config, and clear caches"
        - "./craft migrate/all"
        - "./craft project-config/sync"
        - "./craft clear-caches/all"
      trigger_condition: "ALWAYS"
      run_as_script: true
      shell: "BASH"
    - action: "Send notification to nystudio107 channel"
      type: "SLACK"
      content: "[#$BUDDY_EXECUTION_ID] $BUDDY_PIPELINE_NAME execution by $BUDDY_EXECUTION_REVISION_COMMITTER_NAME ."
      blocks: "[{\"type\":\"section\",\"fields\":[{\"type\":\"mrkdwn\",\"text\":\"*Successful execution:* <$BUDDY_EXECUTION_URL|Execution #$BUDDY_EXECUTION_ID $BUDDY_EXECUTION_COMMENT>\"},{\"type\":\"mrkdwn\",\"text\":\"*Pipeline:* <$BUDDY_PIPELINE_URL|$BUDDY_PIPELINE_NAME>\"},{\"type\":\"mrkdwn\",\"text\":\"*Branch:* $BUDDY_EXECUTION_BRANCH\"},{\"type\":\"mrkdwn\",\"text\":\"*Project:* <$BUDDY_PROJECT_URL|$BUDDY_PROJECT_NAME>\"}]}]"
      channel: "CAYN15RD0"
      channel_name: "nystudio107"
      trigger_condition: "ALWAYS"
      integration_hash: "5ef0d26820cfeb531cb10738"
  variables:
    - key: "PROJECT_SHORTNAME"
      value: "devmode"
    - key: "PROJECT_URL"
      value: "https://devmode.fm"
    - key: "REMOTE_PROJECT_ROOT"
      value: "/home/forge/devmode.fm"
    - key: "REMOTE_SSH_HOST"
      value: "devmode.fm"
    - key: "REMOTE_SSH_USER"
      value: "forge"

The fact that we refactored things that change from project to project into environment variables makes it super easy to re-use this config on multiple projects.

And here’s what the final pipeline looks like in the GUI:

One more deploy for the road

The advantages that I find with buddy.works over tools like Ansible & Puppet or services like DeployBot & Envoyer are that it’s very easy to set up, and you can run all of your build steps in Docker containers in the cloud.

Because everything runs in Docker containers in the cloud, you also do not need Composer or Node or anything else that’s used only to “build the thing” installed on your server.

GitLab CI/CD works similarly to this, and is also a solid choice. But I prefer buddy.works being decoupled from where the git repo is hosted, because this flexibility can be very handy when dealing with varied client needs & requirements.

There’s also plenty more that buddy.works can do that we haven’t explored here. For example, you’d typically set up another pipeline for your staging server, which would auto-deploy on pushes to the develop branch.

We also could go a step further with our deployments and do blue/green database deployments if the project warranted it.

Automated acceptance tests could be run in the buddy.works containers, and deployment would only happen if they pass.

Or we could run accessibility tests on deploy, and block deployment if there were regressions there.

The options are limitless, and buddy.works makes it easy for me to explore them.

But whatever deployment tool you use… happy deploying!

A taste of Vue.js 3: API Changes, Async Components, and Plugins

Andrew Welch — Fri, 29 May 2020 23:26:00 +0000

A taste of Vue.js 3: API Changes, Async Components, and Plugins

This article takes you through changes that have to be made when moving to Vue.js 3, covering API changes, async components, and adapting existing plugins

Andrew Welch / nystudio107

We’re currently in the planning phase for a project, and are choosing the technologies that we’ll be using as the basis for it.

Vue.js will be amongst those technologies, but should we go with Vue 2 or Vue 3, which is currently still a beta?

                            It’s at that awkward stage where it could go either way

At the time of this writing, Vue.js 3 is at version 3.0.0-beta 14, and is slated for release Q2 2020. For now, it can be found at the vuejs/vue-next GitHub repo.

What we decided to do was attempt to convert over the scaffolding we use in the nystudio107/craft repo and detailed in the An Annotated webpack 4 Config for Frontend Web Development article.

If everything went smoothly, then away we go… Vue.js 3 it is. If not, then we stick with Vue.js 2.

We were primarily interested in using the new Composition API, better TypeScript support, and some other key improvements in Vue.js 3.

                            But mostly, the version of Vue we picked would be in use for some time

This article discusses the changes we needed to make to convert the scaffolding over. It shows some real-world situations and annotates the changes we had to make to get the code up and running on Vue.js 3.

This article doesn’t detail every change in Vue.js 3, for that check out the Vue 3 Tutorial (for Vue 2 users) article and the New awesomeness coming in Vue.js 3.0 podcast.

Spoiler alert: it went well, we’re using Vue.js 3 for this project!

Overview of the Changes

The changes we’re going to be making here are really relatively trivial. We have a single JavaScript entry point app.js file, and a VueConfetti component.

This skeleton code is what I use for my scaffolding, because it’s nice to see some confetti to indicate that your code is working as intended.

The app.js is just a shell that doesn’t do much of anything other than load the VueConfetti component, but the project does demonstrate some interesting things:

Changes needed to your package.json file
Changes needed to your webpack config
The changes needed to instantiate a new Vue app
How to do webpack dynamic imports of Vue 3 APIs
How to use async components in Vue 3 using new Async Component API
How we can adapt a Vue plugin that assumes being able to globally inject instance properties via Vue.prototype

If you’re using vue-cli, there’s a vue-cli-plugin-vue-next plugin that will automate some of the project conversion for you, but I wanted to get my hands dirty.

If you’re interested in seeing all of the major changes in Vue.js 3, check out the Vue.js merged RFCs.

And now, without further ado… let’s get on with the show!

Package.json Changes

The first thing we need to do is convert over the package.json packages to versions that work with Vue.js 3.

Here are just the additions/changes needed (not the complete package.json):


{
    "devDependencies": {
        "@vue/compiler-sfc": "^3.0.0-beta.2",
        "css-loader": "^3.4.2",
        "file-loader": "^6.0.0",
        "mini-css-extract-plugin": "^0.9.0",
        "vue-loader": "^16.0.0-alpha.3"
    },
    "dependencies": {
        "vue": "^3.0.0-beta.14"
    }
}

webpack config changes

Next up we need to make some very minor changes to the webpack config detailed in the An Annotated webpack 4 Config for Frontend Web Development article.

We just need to make two changes in the webpack.common.js file, and we’re done.

First, we need to change how we import the VueLoaderPlugin:


const VueLoaderPlugin = require('vue-loader/lib/plugin');

To look like this:


const { VueLoaderPlugin } = require('vue-loader');

Next up we need to change what file we alias vue to, by changing:


        alias: {
            'vue$': 'vue/dist/vue.esm.js'
        },

To look like this:


        alias: {
            'vue$': 'vue/dist/vue.esm-bundler.js'
        },

app.js Changes

Housekeeping out of the way, now we can get into the actual changes in the JavaScript & Vue components.

Here’s what the skeleton app.js looked like for Vue.js 2:


// Import our CSS
import styles from '../css/app.pcss';

// App main
const main = async () => {
    // Async load the vue module
    const { default: Vue } = await import(/* webpackChunkName: "vue" */ 'vue');
    // Create our vue instance
    return new Vue({
        el: "#page-container",
        components: {
            'confetti': () => import(/* webpackChunkName: "confetti" */ '../vue/Confetti.vue'),
        },
        data: {
        },
    });
};

// Execute async function
main().then( (vm) => {
});

// Accept HMR as per: https://webpack.js.org/api/hot-module-replacement#accept
if (module.hot) {
    module.hot.accept();
}

We have an async function main() that awaits the promise returned by the webpack dynamic import of the Vue constructor.

This pattern allows the main thread to continue executing while webpack handles dynamically loading the vue chunk.

While this is somewhat pointless in the skeleton code, this type of dynamic importing allows for code splitting that becomes beneficial from a performance point of view as our application gets fleshed out.

Then we create a new ViewModel, adding in our async component Confetti.vue (we’ll get to the component in a bit).

Let’s have a look at the changes we need to make to this code to get it working on Vue.js 3:


// Import our CSS
import styles from '../css/app.pcss';

// App main
const main = async () => {
    // Async load the Vue 3 APIs we need from the Vue ESM
    const { createApp, defineAsyncComponent } = await import(/* webpackChunkName: "vue" */ 'vue');
    // Create our root vue instance
    return createApp({
        components: {
            'confetti': defineAsyncComponent(() => import(/* webpackChunkName: "confetti" */ '../vue/Confetti.vue')),
        },
        data: () => ({
        }),
    }).mount("#page-container");
};

// Execute async function
main().then( (root) => {
});

// Accept HMR as per: https://webpack.js.org/api/hot-module-replacement#accept
if (module.hot) {
    module.hot.accept();
}

The global Vue constructor is gone in Vue.js 3, and instead we need to explicitly import the functions from the Vue.js 3 API that we need.

In this case, we’re going to need createApp() to create our app instance, and we’ll need defineAsyncComponent() to utilize the new Async Component API for using async components.

createApp() returns an app instance, which has an app context that is available to all components in the component tree.

Unlike Vue.js 2, this app doesn’t automatically mount, so we call .mount("#page-container"), which returns the root component instance, which mounts on the DOM element with the id page-container.

To get our async component Confetti.vue working, all we need to do is wrap the function we used in Vue.js 2 with defineAsyncComponent().

Also of note is that data can no longer be an object, but rather needs to be a factory function that returns a data object. While you’d often do this in Vue.js 2 already, it’s now mandatory in Vue.js 3.

If you want to learn more about some of these global API changes, check out the Global API Change RFC.

Confetti.vue changes

Now onto the all important Confetti.vue component! 🎉

The existing code for the Confetti.vue component looks like this, and is roughly a copy & paste of the example on the vue-confetti GitHub repo:


<template>
    <main>
    </main>
</template>

<script>
    import Vue from 'vue'
    import VueConfetti from 'vue-confetti'

    Vue.use(VueConfetti);

    export default {
        mounted: function() {
            this.$confetti.start({
                shape: 'heart',
                colors: ['DodgerBlue', 'OliveDrab', 'Gold', 'pink', 'SlateBlue', 'lightblue', 'Violet', 'PaleGreen', 'SteelBlue', 'SandyBrown', 'Chocolate', 'Crimson'],
            });
            setTimeout(() => {
                this.$confetti.stop();
            }, 5000);
        },
        methods: {}
    }
</script>

Unfortunately, this didn’t work out of the box on Vue.js 3, giving us the error:

Uncaught TypeError: Cannot read property '$confetti' of undefined

So to figure out what was wrong here, I had a look at the Vue plugin VueConfetti we’re importing, which looks like this:


import Confetti from './confetti';

export { Confetti };

export default {
  install(Vue, options) {
    if (this.installed) {
      return;
    }
    this.installed = true;
    Vue.prototype.$confetti = new Confetti(options); // eslint-disable-line no-param-reassign
  },
};

The way plugins work is that they define an install() function that is called to do whatever they need to do to install themselves, when Vue.use() is called.

In Vue.js 2, the global Vue constructor is passed in as the first parameter, but in Vue.js 3 we’d actually be calling app.use(), and the first parameter then becomes the app context, which is not a constructor, and thus has no .prototype.

Indeed, if we console.log() the first parameter passed in via Vue.js 2, we’ll see the Vue constructor:


ƒ Vue (options) {
  if ( true &&
    !(this instanceof Vue)
  ) {
    warn('Vue is a constructor and should be called with the `new` keyword');
  }
  this._init(options);
}

But a console.log() the first parameter passed in via Vue.js 3, we’ll see the app context:


{_component: {…}, _props: null, _container: null, _context: {…}, …}
component: ƒ component(name, component)
config: (...)
directive: ƒ directive(name, directive)
mixin: ƒ mixin(mixin)
mount: (containerOrSelector) => {…}
provide: ƒ provide(key, value)
unmount: ƒ unmount()
use: ƒ use(plugin, ...options)
_component: {components: {…}, data: ƒ}
_container: null
_context: {config: {…}, mixins: Array(0), components: {…}, directives: {…}, provides: {…}}
_props: null
get config: ƒ config()
set config: ƒ config(v)
__proto__ : Object

So okay, how can we fix this? The problem is that VueConfetti is trying to inject a globally shared instance property $confetti via Vue.prototype.$confetti, but don’t have a global constructor in Vue.js 3, so .prototype isn’t a thing here.

One way would be to change the vue-confetti/index.js code to use the new app instance’s config.globalProperties to accomplish the same thing, something like:

app.config.globalProperties.$confetti = new Confetti(options);

c.f.: Attaching Globally Shared Instance Properties

But this would require changing the VueConfetti code via a fork/pull request. While I’m not against doing this, I realized there was an easier way to accomplish the same thing:


<template>
</template>

<script>
    import Confetti from 'vue-confetti/src/confetti.js';
    export default {
        data: () => ({
            confetti: new Confetti(),
        }),
        mounted: function() {
            this.confetti.start({
                shape: 'heart',
                colors: ['DodgerBlue', 'OliveDrab', 'Gold', 'pink', 'SlateBlue', 'lightblue', 'Violet', 'PaleGreen', 'SteelBlue', 'SandyBrown', 'Chocolate', 'Crimson'],
            });
            setTimeout(() => {
                this.confetti.stop();
            }, 5000);
        },
        methods: {}
    }
</script>

Here we change the Confetti.vue component to directly import 'vue-confetti/src/confetti.js' and assign the new Confetti() object to our local data state object, rather than having it be globally available.

This feels a little nicer to me in general, because there’s probably no great reason for the $confetti object to be globally available, if we’re creating a Confetti.vue component that can nicely encapsulate it.

Should you use Vue.js 3 now?

We decided to use Vue.js 3 now, but should you?

I think much depends on how heavily you lean on third party components, plugins, and mixins.

                            The more code you’ll be writing yourself, the safer it is to use Vue.js <span>3</span> now

While all software always has issues, Vue.js 3 itself seems quite solid, and the first-party packages like Vuex and Vue-Router are coming along great.

There will likely be some lag in third party packages getting updated for Vue.js 3, and some may never be.

Thus whether to go with Vue.js 3 now really depends on how much you rely on said third party packages.

For us, the benefits are worthwhile enough for us to begin learning and using Vue.js 3 now.

Wrapping up

Hopefully this small dive into what it looks like updating your code for Vue.js 3 is helpful to you. While it is relatively narrow in scope, it does touch on some topics I hadn’t seen covered elsewhere, at least not wrapped up in a neat package.

I’m excited to explore Vue.js 3 further, and will very likely document more of my journey learning the new hotness in Vue.js 3.

Happy coding!

Extending Craft CMS with Validation Rules and Behaviors

Andrew Welch — Fri, 24 Apr 2020 02:55:00 +0000

Extending Craft CMS with Validation Rules and Behaviors

Craft CMS is a web application that is amazingly flexible & customizable using the built-in functionality that the platform offers. Use the platform!

Andrew Welch / nystudio107

Craft CMS is built on the rock-solid Yii2 framework, which is something you normally don’t need to think about. It just works, as it should.

But there are times that you need or want to extend the platform into something truly custom, which we looked at in the Enhancing a Craft CMS 3 Website with a Custom Module article.

In this article, we’ll talk about two ways you can use the platform that you normally don’t even have to think about to your advantage.

Use the Platform

Something we commonly hear in frontend development is to “use the platform”, which I think is fantastic advice. Why re-invent an elaborate custom setup when the platform already provides you with a battle-worn way to accomplish your goals?

The same holds true with any kind of development. If you’ve writing something on top of a platform — whatever that platform may be — I think it always makes sense to try to leverage it as much as possible.

It’s there. It’s well thought-out. It’s tested. Use it!

When we’re using Craft CMS, we’re also using the Yii2 platform that it’s built on. Indeed, as we discussed in the So You Wanna Make a Craft 3 Plugin? article, to know Craft plugin development, you will want to learn some part of Yii2.

So let’s do just that! The Yii2 documentation is a great place to start.

Models & Rules

Models are a core building block of Yii2, and so also Craft CMS. They are at the core of the Model-View-Controller (MVC) paradigm that many frameworks use.

Models are objects representing business data, rules and logic.

Models are used to represent data and validate data via a set of rules. For instance, Craft CMS has a User element (which is also a model) that encapsulates all of the data needed to represent a User in Craft CMS.

It also has validation rules for the data:


/**
 * @inheritdoc
 */
protected function defineRules(): array
{
    $rules = parent::defineRules();
    $rules[] = [['lastLoginDate', 'lastInvalidLoginDate', 'lockoutDate', 'lastPasswordChangeDate', 'verificationCodeIssuedDate'], DateTimeValidator::class];
    $rules[] = [['invalidLoginCount', 'photoId'], 'number', 'integerOnly' => true];
    $rules[] = [['username', 'email', 'unverifiedEmail', 'firstName', 'lastName'], 'trim', 'skipOnEmpty' => true];
    $rules[] = [['email', 'unverifiedEmail'], 'email'];
    $rules[] = [['email', 'password', 'unverifiedEmail'], 'string', 'max' => 255];
    $rules[] = [['username', 'firstName', 'lastName', 'verificationCode'], 'string', 'max' => 100];
    $rules[] = [['username', 'email'], 'required'];
    $rules[] = [['username'], UsernameValidator::class];
    $rules[] = [['lastLoginAttemptIp'], 'string', 'max' => 45];

If this looks more like config than code to you, then you’d be right! Model validation rules are essentially a list of rules that the data must pass in order to be considered valid.

Yii2 has a base Validator class to help you write validators, and ships with a whole bunch of useful Core Validators built-in that you can leverage.

And we can see here that Craft CMS is doing just that in its craft\elements\User.php class. Any validation rule is an array:

Field — the model field (aka attribute or object property) or array of model fields to apply this validation rule to
Validator — the validator to use, which can be a Validator class, an alias to a validator class, PHP Callable, or even an anonymous function for inline validation
[params] — depending on the validator, there may be additional optional parameters you can define

So in the above User Element example, the email & unverifiedEmail fields are using the built-in email core validator that Yii2 provides.

The username has several validation rules listed, which are applied in order:

string — This validator checks if the input value is a valid string with certain length (100 in this case)
required — This validator checks if the input value is provided and not empty
UsernameValidator — This is a custom validator that P&T wrote to handle validating the username field

The username field actually gives us a fun little tangent we can go on, so let’s peek under the hood to see how simple it can be to write a custom validator.

Here’s what the class looks like:


<?php
/**
 * @link https://craftcms.com/
 * @copyright Copyright (c) Pixel & Tonic, Inc.
 * @license https://craftcms.github.io/license/
 */

namespace craft\validators;

use Craft;
use yii\validators\Validator;

/**
 * Class UsernameValidator.
 *
 * @author Pixel & Tonic, Inc. <support@pixelandtonic.com>
 * @since 3.0.0
 */
class UsernameValidator extends Validator
{
    /**
     * @inheritdoc
     */
    public function validateValue($value)
    {
        // Don't allow whitespace in the username
        if (preg_match('/\s+/', $value)) {
            return [Craft::t('app', '{attribute} cannot contain spaces.'), []];
        }

        return null;
    }
}

At its simplest form, this is all a Validator needs to implement! Given some passed in $value, return whether it passes validation or not.

In this case, it’s just checking if it passes a regular expression (RegEx) test.

And indeed, we can even simplify this further, and get rid of the custom validator altogether by using the match core validator:


    $rules[] = [['username'], 'match', '/\s+/', 'not' => true];

Then we’re really be using the platform, and getting rid of custom code.

Being able to delete code is one of the most underrated joys of programming

But let’s return from our tangent, and see how we can leverage these rules to our own advantage. Let’s say we have specific requirements for our username and password fields.

Well, we can easily extend the existing model validation rules for our User Element by listening for the User class triggering the EVENT_DEFINE_RULES event:


<?php
/**
 * Site module for Craft CMS 3.x
 *
 * Custom site module for the devMode.fm website
 *
 * @link https://nystudio107.com
 * @copyright Copyright (c) 2020 nystudio107
 */

namespace modules\sitemodule;

use modules\sitemodule\rules\UserRules;

use craft\elements\User;
use craft\events\DefineRulesEvent;

// ...

class SiteModule extends Module
{
    /**
     * @inheritdoc
     */
    public function init()
    {
        parent::init();

        // Add in our custom rules for the User element validation
        Event::on(
            User::class,
            User::EVENT_DEFINE_RULES,
            static function(DefineRulesEvent $event) {
                foreach(UserRules::define() as $rule) {
                    $event->rules[] = $rule;
                }
            });
    // ...
    }
}

We’re calling our custom class method UserRules::define() to return a list of rules we want to add, and then we’re adding them one by one to the $event->rules

Here’s what the UserRules class looks like:


<?php
/**
 * Site module for Craft CMS 3.x
 *
 * Custom site module for the devMode.fm website
 *
 * @link https://nystudio107.com
 * @copyright Copyright (c) 2020 nystudio107
 */

namespace modules\sitemodule\rules;

use Craft;

/**
 * @author nystudio107
 * @package SiteModule
 * @since 1.0.0
 */
class UserRules
{
    // Constants
    // =========================================================================

    const USERNAME_MIN_LENGTH = 5;
    const USERNAME_MAX_LENGTH = 15;
    const PASSWORD_MIN_LENGTH = 7;

    // Public Methods
    // =========================================================================

    /**
     * Return an array of Yii2 validator rules to be added to the User element
     * https://www.yiiframework.com/doc/guide/2.0/en/input-validation
     *
     * @return array
     */
    public static function define(): array
    {
        return [
            [
                'username',
                'string',
                'length' => [self::USERNAME_MIN_LENGTH, self::USERNAME_MAX_LENGTH],
                'tooLong' => Craft::t(
                    'site-module',
                    'Your username {max} characters or shorter.',
                    [
                        'min' => self::USERNAME_MIN_LENGTH,
                        'max' => self::USERNAME_MAX_LENGTH
                    ]
                ),
                'tooShort' => Craft::t(
                    'site-module',
                    'Your username must {min} characters or longer.',
                    [
                        'min' => self::USERNAME_MIN_LENGTH,
                        'max' => self::USERNAME_MAX_LENGTH
                    ]
                ),
            ],
            [
                'password',
                'string',
                'min' => self::PASSWORD_MIN_LENGTH,
                'tooShort' => Craft::t(
                    'site-module',
                    'Your password must be at least {min} characters.',
                    ['min' => self::PASSWORD_MIN_LENGTH]
                )
            ],
            [
                'password',
                'match',
                'pattern' => '/^(?=.*[a-z])(?=.*[A-Z])(?=.*[0-9])(?=.*[!@#\$%\^&\*])(?=.{7,})/',
                'message' => Craft::t(
                    'site-module',
                    'Your password must contain at least one of each of the following: A number, a lower-case character, an upper-case character, and a special character'
                )
            ],
        ];
    }
}

And then BOOM! Just like that we’ve extended the User Element model validation rules with our own custom rules.

We’re even giving it the custom message to display if the password field doesn’t match, as well as the message to display if the username field is tooLong or tooShort.

Nice.

As you can see, we even get the display of the validation errors “for free” on the frontend, without having to do any additional work.

The less code you write, the fewer chances you have to introduce bugs

Bear in mind that while we’re showing the User Element as an example, we can do this for any model that Craft uses.

For instance, if you want to make the address field in Craft Commerce required, this is your ticket!

Models & Behaviors

But what if we want to add some properties or methods to an existing model? Well, we can do that, too, via Yii2 Behaviors.

To extend our User Element with a custom Behavior, we can listen for the User class triggering the EVENT_DEFINE_BEHAVIORS event:


<?php
/**
 * Site module for Craft CMS 3.x
 *
 * Custom site module for the devMode.fm website
 *
 * @link https://nystudio107.com
 * @copyright Copyright (c) 2020 nystudio107
 */

namespace modules\sitemodule;

use modules\sitemodule\behaviors\UserBehavior;

use craft\elements\User;
use craft\events\DefineBehaviorsEvent;

// ...

class SiteModule extends Module
{
    /**
     * @inheritdoc
     */
    public function init()
    {
        parent::init();

        // Add in our custom behavior for the User element
        Event::on(
            User::class,
            User::EVENT_DEFINE_BEHAVIORS,
            static function(DefineBehaviorsEvent $event) {
                $event->behaviors['userBehavior'] = ['class' => UserBehavior::class];
            });
    // ...
    }
}

Here we just add our userBehavior by setting the $event->behaviors['userBehavior'] to a custom UserBehavior class we wrote that inherits from the Yii2 Behavior class:


<?php
/**
 * Site module for Craft CMS 3.x
 *
 * Custom site module for the devMode.fm website
 *
 * @link https://nystudio107.com
 * @copyright Copyright (c) 2020 nystudio107
 */

namespace modules\sitemodule\behaviors;

use craft\elements\User;

use yii\base\Behavior;

/**
 * @author nystudio107
 * @package SiteModule
 * @since 1.0.0
 */
class UserBehavior extends Behavior
{
    // Public Properties
    // =========================================================================

    // Public Methods
    // =========================================================================

    /**
     * @inheritDoc
     */
    public function events()
    {
        return [
            User::EVENT_BEFORE_SAVE => 'beforeSave',
        ];
    }

    /**
     * Save last names in upper-case
     *
     * @param $event
     */
    public function beforeSave($event)
    {
        $this->owner->lastName = mb_strtoupper($this->owner->lastName);
    }

    /**
     * Return a friendly name with a smile
     *
     * @return string
     */
    public function getHappyName()
    {
        $name = $this->owner->getFriendlyName();

        return ':) ' . $name;
    }
}

We’re using the events() method to define the Component Events we want our behavior to listen for.

In our case, we’re listening for the EVENT_BEFORE_SAVE event, and we’re calling a new method we added called beforeSave.

In the context of a behavior, $this->owner refers to the Model object that our behavior is attached to; in our case, that’s a User Element.

So our beforeSave() method just upper-cases the User::$lastName property before saving it. So everyone’s last name will be upper-case.

Then we’ve added a getHappyName() method that prepends a smiley face to the User Element’s name, so in our Twig templates we can now do:


{{ currentUser.getHappyName() }}

Pretty slick, we just piggybacked on the existing Craft User Element functionality without having to do a while lot of work.

In our Behavior, if we defined any additional properties, they’d be added to the User Element model as well… which opens up a whole world of possibilities.

In addition to writing our own custom behaviors, we can also leverage other built-in Behaviors that Yii2 offers, and add them to our own Models. My personal favorite is the AttributeTypecastBehavior.

Check out Zoltan’s article Extending entries with Yii behaviors in Craft 3 for even more on behaviors.

I’d also like to note what Behaviors are not. You can not override an existing method with a Behavior. You might want override an existing method and replace it with your own code dynamically… but Behaviors cannot do that.

Behaviors can only extend, not replace.

Wrapping Up

In addition to “use the platform”, whenever we’re adding code, I think we should add as little as possible.

Yes, there are other ways to add the functionality we’ve shown in this article, but the methods discussed here are simple, and require less code.

Simple is Good

When adding code to an existing project or framework, you typically want to go in like a surgeon, changing as little as possible to achieve the desired effect.

Happy coding!

An Annotated Docker Config for Frontend Web Development

Andrew Welch — Sun, 22 Mar 2020 16:12:00 +0000

An Annotated Docker Config for Frontend Web Development

A local development environment with Docker allows you to shrink-wrap the devops your project needs as config, making onboarding frictionless

Andrew Welch / nystudio107

Docker is a tool for containerizing your applications, which means that your application is shrink-wrapped with the environment that it needs to run.

This allows you to define the devops your application needs in order to run as config, which can then be easily replicated and reused.

The principals and approach discussed in this article are universal.

While there are many uses for Docker, this article will focus on using Docker as a local environment for frontend web development.

Although Craft CMS is referenced in this article, Docker works well for any kind of web development with any kind of CMS or dev stack (Laravel, NodeJS, Rails, whatevs).

The Docker config used here is used in both the devMode.fm GitHub repo, and in the nystudio107/craft boilerplate Composer project if you want to see some “in the wild” examples.

Why Docker?

If you’re doing frontend web development, you very likely already have some kind of a local development environment.

So why should you use Docker instead?

This is a very reasonable question to ask, because any kind of switch of tooling requires some upskilling, and some work.

I’ve long been using Homestead—which is really just a custom Vagrant box with some extras — as my local dev environment as discussed in the Local Development with Vagrant / Homestead article.

I’d chosen to use Homestead because I wanted a local dev environment that was deterministic, disposable, and separated my development environment from my actual computer.

Docker has all of these advantages, but also a much more lightweight approach. Here are the advantages of Docker for me:

Each application has exactly the environment it needs to run, including specific versions of any of the plumbing needed to get it to work (PHP, MySQL, Postgres, whatever)
Onboarding others becomes trivial, all they need to do is install Docker and type docker-compose up and away they go
Your development environment is entirely disposable; if something goes wrong, you just delete it and fire up a new one
Your local computer is separate from your development environment, so switching computers is trivial, and you won’t run into issues where you hose your computer or are stuck with conflicting versions of devops services
The cost of trying different versions of various services is low; just change a number in a .yaml file, docker-compose up, and away you go

There are other advantages as well, but these are the more important ones for me.

Additionally, containerizing your application in local development is a great first step to using a containerized deployment process, and running Docker in production as well.

Understanding Docker

This article is not a comprehensive tutorial on Docker, but I will attempt to explain some of the more important, broader concepts.

Docker has the notion of containers, each of which run one or more services. You can think of each container as a mini virtual machine (even though technically, they are not).

While you can run multiple services in a single Docker container, separating each service out into a separate container has many advantages.

If PHP, Apache, and MySQL are all in separate containers, they won’t affect each other, and also can be more easily swapped in and out.

If you decide you want to use Nginx or Postgres instead, the decoupling into separate containers makes it easy!

Docker containers are built from Docker images, which can be thought of as a recipe for building the container, with all of the files and code needed to make it happen.

If a Docker image is the recipe, a Docker container is the finished resulting meal.

Docker images almost always are layered on top of other existing images that they extend FROM. For instance, you might have a base image from Ubuntu or Alpine Linux which provide in the necessary operating system kernel layer for other processes like Nginx to run.

This layering works thanks to the Union file system, which handles composing all the layers of the cake together for you.

We said earlier that Docker is more lightweight than running a full Vagrant VM, and it is… but unfortunately, unless you’re running Linux there still is a virtualization layer running, which is HyperKit for the Mac, and Hyper‑V for Windows.

Docker for Mac & Windows still has a virtualization layer, it’s just relatively lightweight.

Fortunately, you don’t need to be concerned with any of this, but the performance implications do inform some of the decisions we’ve made in the Docker config presented here.

For more information on Docker, for that I’d highly recommend the Docker Mastery course (if it’s not on sale now, don’t worry, it will be at some point) and also the following devMode.fm episodes:

Containerize your Development with Docker!

…and there are tons of other excellent educational resources on Docker out there such as Matt Gray’s Craft in Docker: Everything I’ve Learnt presentation, and his excellent A Craft CMS Development Workflow With Dockerseries.

In our article, we will focus on annotating a real-world Docker config that’s used in production. We’ll discuss various Docker concepts as we go, but the primary goal here is documenting a working config.

This article is what I wished existed when I started learning Docker

I learn best by looking at a working example, and picking it apart. If you do, too, let’s get going!

My Docker Directory Structure

This Docker setup uses a directory structure that looks like this (don’t worry, it’s not as complex as it seems, many of the Docker images here are for reference only, and are actually pre-built):


├── cms
│ ├── composer.json
│ ├── config
│ ├── craft
│ ├── craft.bat
│ ├── example.env
│ ├── modules
│ ├── templates
│ └── web
├── docker-compose.yml
├── docker-config
│ ├── mariadb
│ │ └── Dockerfile
│ ├── nginx
│ │ ├── default.conf
│ │ └── Dockerfile
│ ├── php-dev-base
│ │ ├── Dockerfile
│ │ └── zzz-docker.conf
│ ├── php-dev-craft
│ │ └── Dockerfile
│ ├── postgres
│ │ └── Dockerfile
│ ├── redis
│ │ └── Dockerfile
│ ├── webpack-dev-base
│ │ └── Dockerfile
│ └── webpack-dev-craft
│ ├── Dockerfile
│ ├── package.json
│ ├── postcss.config.js
│ ├── tailwind.config.js
│ ├── webpack.common.js
│ ├── webpack.dev.js
│ ├── webpack.prod.js
│ └── webpack.settings.js
├── scripts
│ ├── common
│ ├── docker_pull_db.sh
│ ├── docker_restore_db.sh
│ ├── example.env.sh
│ └── seed_db.sql
└── src
    ├── conf
    ├── css
    ├── img
    ├── js
    ├── php
    ├── templates -> ../cms/templates
    └── vue

Here’s an explanation of what the top-level directories are:

cms — everything needed to run Craft CMS. The is the “app” of the project
docker-config — an individual directory for each service that the Docker setup uses, with a Dockerfile and other ancillary config files therein
scripts — helper shell scripts that do things like pull a remote or local database into the running Docker container. These are derived from the Craft-Scripts shell scripts
src — the frontend JavaScript, CSS, Vue, etc. source code that the project uses

Each service is referenced in the docker-compose.yaml file, and defined in the Dockerfile that is in the corresponding directory in the docker-config/ directory.

It isn’t strictly necessary to have a separate Dockerfile for each service, if they are just derived from a base image. But I like the consistency, and ease of future expansion should something custom be necessary down the road.

You’ll also notice that there are php-dev-base and php-dev-craft directories, as well as webpack-dev-base and webpack-dev-craft directories, and might be wondering why they aren’t consolidated.

The reason is that there’s a whole lot of the base setup in both that just never changes, so instead of rebuilding that each time, we can build it once and publish the images up on DockerHub.com as nystudio107/php-dev-base and nystudio107/webpack-dev-base.

Then we can layer anything specific about our project on top of these base images in the respective -craft services. This saves us significant building time, while keeping flexibility.

The docker-compose.yaml file

While a docker-compose.yaml file isn’t required when using Docker, from a practical point of view, you’ll almost always use it. The docker-compose.yaml file allows you to define multiple containers for running the services you need, and coordinate starting them up and shutting them down in unison.

Then all you need to do is run docker-compose up via the terminal in a directory that has a docker-compose.yaml file, and Docker will start up all of your containers for you!

Here’s an example of what that might look like, starting up your Docker containers:

Let’s have a look at our docker-compose.yaml file:


version: '3.7'

services:
  # nginx - web server
  nginx:
    build:
      context: .
      dockerfile: ./docker-config/nginx/Dockerfile
    env_file: &env
      - ./cms/.env
    links:
      - php
    ports:
      - "8000:80"
    volumes:
      - cpresources:/var/www/project/cms/web/cpresources
      - ./cms/web:/var/www/project/cms/web:cached
  # php - personal home page
  php:
    build:
      context: .
      dockerfile: ./docker-config/php-dev-craft/Dockerfile
    depends_on:
      - "mariadb"
      - "redis"
    env_file:
      *env
    expose:
      - "9000"
    links:
      - mariadb
      - redis
    volumes:
      - cpresources:/var/www/project/cms/web/cpresources
      - storage:/var/www/project/cms/storage
      - ./cms:/var/www/project/cms:cached
      - ./cms/vendor:/var/www/project/cms/vendor:delegated
      - ./cms/storage/logs:/var/www/project/cms/storage/logs:delegated
  # mariadb - database
  mariadb:
    build:
      context: .
      dockerfile: ./docker-config/mariadb/Dockerfile
    env_file:
      *env
    environment:
      MYSQL_ROOT_PASSWORD: secret
      MYSQL_DATABASE: project
      MYSQL_USER: project
      MYSQL_PASSWORD: project
    ports:
      - "3306:3306"
    volumes:
      - db-data:/var/lib/mysql
  # redis - key/value database for caching & php sessions
  redis:
    build:
      context: .
      dockerfile: ./docker-config/redis/Dockerfile
    expose:
      - "6379"
  # webpack - frontend build system
  webpack:
    build:
      context: .
      dockerfile: ./docker-config/webpack-dev-craft/Dockerfile
    env_file:
      *env
    ports:
      - "8080:8080"
    volumes:
      - ./docker-config/webpack-dev-craft:/var/www/project/docker-config/webpack-dev-craft:cached
      - ./docker-config/webpack-dev-craft/node_modules:/var/www/project/docker-config/webpack-dev-craft/node_modules:delegated
      - ./cms/web/dist:/var/www/project/cms/web/dist:delegated
      - ./src:/var/www/project/src:cached
      - ./cms/templates:/var/www/project/cms/templates:cached

volumes:
  db-data:
  cpresources:
  storage:

This .yaml file has 3 top-level keys:

version — the version number of the Docker Compose file, which corresponds to different capabilities offered by different versions of the Docker Engine
services — each service corresponds to a separate Docker container that is created using a separate Docker image
volumes — named volumes that are mounted and can be shared amongst your Docker containers (but not your host computer), for storing persistent data

We’ll detail each service below, but there are a few interesting tidbits to cover first.

BUILD

When you’re creating a Docker container, you can either base it on an existing image (either a local image or one pulled down from DockerHub.com), or you can build it locally via a Dockerfile.

As mentioned above, I chose the methodology that each service would be creating as a build from a Dockerfile (all of which extend FROM an image up on DockerHub.com) to keep things consistent.

This means that some of our Dockerfiles we use are nothing more than a single line, e.g.: FROM mariadb:10.3, but this setup does allow for expansion.

The two keys used for build are are:

context — this specifies where the working directory for the build should be, relative to the docker-compose.yaml file. We have this set to . (the current directory) for each service
dockerfile — this specifies a path to the Dockerfile to use to build the service Docker container. Think of the Dockerfile as a local Docker image

So the context is always the root directory of the project, with the Dockerfile and any supporting files for each service are off in a separate directory. We do it this way to keep the paths consistent (always relative to the project root) regardless of the service.

DEPENDS_ON

This just lets you specify what other services this particular service depends on; this allows you to ensure that other containers are up and running before this container starts up.

ENV_FILE

The env_file setting specifies a path to your .env file for key/value pairs that will be injected into a Docker container.

Docker does not allow for quotes in its .env file, which is contrary to how .env files work almost everywhere else… so remove any quotes you have in your .env file.

You’ll notice that for the nginx service, there’s a strange &env value in the env_file setting, and for the other services, the setting is *env. This is taking advantage of YAML aliases, so if we do change the .env file path, we only have to do it in one place.

Doing it this way also ensures that all of the .env environment variables are available in every container. For more on environment variables, check out the Flat Multi-Environment Config for Craft CMS 3 article.

Because it’s Docker that is injecting these .env environment variables, if you change your .env file, you’ll need to restart your Docker containers.

LINKS

Links in Docker allow you to define extra aliases by which a service is reachable from another service. They are not required to enable services to communicate, but I like being explicit about it.

The come into play when one container needs to talk to another. For example, if you normally would communicate with your database via the localhost socket, instead in our setup you’d use the socket named mariadb.

The key take-away is that when containers need to talk to each other, they are doing so over the internal Docker network, and refer to each other via their service or links name.

PORTS

This specifies the port that should be exposed outside of the container, followed by the port that the container uses internally. So for example, the nginx service has "8000:80", which means the externally accessible port for the Nginx webserver is 8000, and the internal port the service runs on is 80.

If this sounds confusing, understand that Docker uses its own internal network to allow containers to talk to each other, as well as the outside world.

VOLUMES

Docker containers run in their own little world, which is great for isolation purposes, but at some point you do need to share things from your “host” computer with the Docker container.

Docker volumes allow you to do this. You specify either a named volume or a path on your host, followed by the path where this volume should be bind mounted in the Docker container.

This is where performance problems can happen with Docker on the Mac and Windows. So we use some hints to help with performance:

consistent — perfect consistency (host and container have an identical view of the mount at all times)
cached — the host’s view is authoritative (permit delays before updates on the host appear in the container)
delegated — the container’s view is authoritative (permit delays before updates on the container appear in the host)

So for things like node_modules/ and vendor/ we mark them as :delegated because while we want them shared, the container is in control of modifying these volumes.

Some Docker setups I’ve seen put these directories into a named volume, which means they are visible only to the Docker containers.

But the problem is we lose out on our editor auto-completion, because our editor has nothing to index.

This is a non-negotiable for me

See the Auto-Complete Craft CMS 3 APIs in Twig with PhpStorm article for details.

Service: Nginx

Nginx is the web server of choice for me, both in local dev and in production.


FROM nginx:1.16

COPY ./docker-config/nginx/default.conf /etc/nginx/conf.d/default.conf

We’ve based the container on the nginx image, tagged at version 1.16

The only modification it makes is COPYing our default.conf file into place:


server {
    listen 80 default_server;
    root /var/www/project/cms/web;
    index index.html index.php;
    charset utf-8;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    access_log off;
    error_log /var/log/nginx/error.log error;

    sendfile off;
    ssi on;

    client_max_body_size 10m;

    gzip on;
    gzip_http_version 1.0;
    gzip_proxied any;
    gzip_min_length 500;
    gzip_disable "MSIE [1-6]\.";
    gzip_types text/plain text/xml text/css
                      text/comma-separated-values
                      text/javascript
                      application/x-javascript
                      application/javascript
                      application/atom+xml;

    location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass php:9000;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_intercept_errors off;
        fastcgi_buffer_size 16k;
        fastcgi_buffers 4 16k;
        fastcgi_read_timeout 300;
    }

    location ~ /\.ht {
        deny all;
    }
}

This is just a simple Nginx config that works well with Craft CMS. You can find more about Nginx configs for Craft CMS in the nginx-craft GitHub repo.

Service: MariaDB

MariaDB is a drop-in replacement for MySQL that I tend to use instead of MySQL itself. It was written by the original author of MySQL, and is binary compatible with MySQL.


FROM mariadb:10.3

We’ve based the container on the mariadb image, tagged at version 10.3

There’s no modification at all to the source image.

Service: Postgres

Postgres is a robust database that I am using more and more for Craft CMS projects. It’s not used in the docker-compose.yaml presented here, but I keep the configuration around in case I want to use it.

Postgres is used in local dev and in production on the devMode.fm GitHub repo, if you want to see it implemented.


FROM postgres:12.2

We’ve based the container on the postgres image, tagged at version 12.2

There’s no modification at all to the source image.

Service: Redis

Redis is a key/value pair database that I set all of my Craft CMS installs to use both as a caching method, and as a session handler for PHP.


FROM redis:5.0

We’ve based the container on the redis image, tagged at version 5.0

There’s no modification at all to the source image.

Service: PHP

PHP is the language that the Yii2 framework and Craft CMS itself is based on, so we need it in order to run our app.

This service is composed of a base image that contains all of the packages and PHP extensions we’ll always need to use, and then the project-specific image that contains whatever additional things are needed for our project.


FROM php:7.3-fpm

# Install packages
RUN apt-get update \
    && \
    # apt Debian packages
    apt-get install -y \
        apt-utils \
        autoconf \
        ca-certificates \
        curl \
        g++ \
        libbz2-dev \
        libfreetype6-dev \
        libjpeg62-turbo-dev \
        libpng-dev \
        libpq-dev \
        libssl-dev \
        libicu-dev \
        libmagickwand-dev \
        libzip-dev \
        unzip \
        zip \
    && \
    # pecl PHP extensions
    pecl install \
        imagick-3.4.4 \
        redis \
        xdebug-2.8.1 \
    && \
    # Configure PHP extensions
    docker-php-ext-configure \
        gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \
    && \
    # Install PHP extensions
    docker-php-ext-install \
        bcmath \
        bz2 \
        exif \
        ftp \
        gettext \
        gd \
        iconv \
        intl \
        mbstring \
        opcache \
        pdo \
        shmop \
        sockets \
        sysvmsg \
        sysvsem \
        sysvshm \
        zip \
    && \
    # Enable PHP extensions
    docker-php-ext-enable \
        imagick \
        redis \
        xdebug

# Append our php.ini overrides to the end of the file
RUN echo "upload_max_filesize = 10M" > /usr/local/etc/php/php.ini && \
    echo "post_max_size = 10M" >> /usr/local/etc/php/php.ini && \
    echo "max_execution_time = 300" >> /usr/local/etc/php/php.ini && \
    echo "memory_limit = 256M" >> /usr/local/etc/php/php.ini && \
    echo "opcache.revalidate_freq = 0" >> /usr/local/etc/php/php.ini && \
    echo "opcache.validate_timestamps = 1" >> /usr/local/etc/php/php.ini

# Copy the `zzz-docker.conf` file into place for php-fpm
COPY ./zzz-docker.conf /usr/local/etc/php-fpm.d/zzz-docker.conf

We’ve based the container on the php image, tagged at version 7.3

We’re then adding a bunch of Debian packages that we want available for our Ubuntu operating system base, some debugging tools, as well as some PHP extensions that Craft CMS requires.

Then we add some defaults to the php.ini, and copy into place the zzz-docker.conf file:


[www]
pm.max_children = 10
pm.process_idle_timeout = 10s
pm.max_requests = 1000

This just sets some defaults for php-fpm that make sense for local development.

By itself, this image won’t do much for us, and in fact we don’t even spin up this image. But we’ve built this image, and made it available as nystudio107/php-dev-base on DockerHub.

Since it’s pre-built, we don’t have to build it every time, and can layer on top of this image anything project-specific via the php-dev-craft container image:


FROM nystudio107/php-dev-base

# Install packages
RUN apt-get update \
    && \
    # apt Debian packages
    apt-get install -y \
        nano \
    && \
    # Install PHP extensions
    docker-php-ext-install \
        pdo_mysql \
    && \
    # Install Composer
    curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin/ --filename=composer

WORKDIR /var/www/project

# Copy over the directories/files php needs access to
COPY --chown=www-data:www-data ./cms/composer.* /var/www/project/cms/

# Create the storage directory and make it writeable by PHP
RUN mkdir -p /var/www/project/cms/storage && \
    mkdir -p /var/www/project/cms/storage/runtime && \
    chown -R www-data:www-data /var/www/project/cms/storage

# Create the cpresources directory and make it writeable by PHP
RUN mkdir -p /var/www/project/cms/web/cpresources && \
    chown -R www-data:www-data /var/www/project/cms/web/cpresources

WORKDIR /var/www/project/cms

# Do a `composer install` without running any Composer scripts
# - If `composer.lock` is present, it will install what is in the lock file
# - If `composer.lock` is missing, it will update to the latest dependencies
# and create the `composer.lock` file
# Force permissions, update Craft, and start php-fpm
CMD if [! -f "./composer.lock"]; then \
        composer install --no-scripts --optimize-autoloader --no-interaction; \
    fi \
    && \
    if [! -d ./vendor -o ! "$(ls -A ./vendor)"]; then \
        composer install --no-scripts --optimize-autoloader --no-interaction; \
    fi \
    && \
    chown -R www-data:www-data /var/www/project/cms/vendor \
    && \
    chown -R www-data:www-data /var/www/project/cms/storage \
    && \
    composer craft-update \
    && \
    php-fpm

This is the image that we actual build into a container, and use for our project. We install the nano editor because I find it handy sometimes, and we also install pdo_mysql so that PHP can connect to our MariaDB database.

We do it this way so that if we want to create a Craft CMS project that uses Postgres, we can just swap in the PDO extension needed here.

Then we make sure the various storage/ and cpresources/ directories are in place, with the right ownership so that Craft will run properly.

Then we do a bit of magic to do a composer install, but only if:

The composer.lock file doesn’t exist
The vendor/ directory doesn’t exist, or is empty

We have to do the composer install as part of the Docker image CMD because the file system mounts aren’t in place until the CMD is run.

This allows us to update our Composer dependencies just by deleting the composer.lock file or the vendor/ directory, and doing docker-compose up

Simple.

The alternative is doing a docker exec -it craft_php_1 /bin/bash to open up a shell in our container, and running the command manually. Which is fine, but a little convoluted for some.

Then we make sure that the ownership on important directories is correct, and we run the craft-update Composer script:


{
  "require": {
    "craftcms/cms": "^3.4.0",
    "vlucas/phpdotenv": "^3.4.0",
    "yiisoft/yii2-redis": "^2.0.6",
    "nystudio107/craft-imageoptimize": "^1.0.0",
    "nystudio107/craft-fastcgicachebust": "^1.0.0",
    "nystudio107/craft-minify": "^1.2.5",
    "nystudio107/craft-typogrify": "^1.1.4",
    "nystudio107/craft-retour": "^3.0.0",
    "nystudio107/craft-seomatic": "^3.2.0",
    "nystudio107/craft-webperf": "^1.0.0",
    "nystudio107/craft-twigpack": "^1.1.0",
    "nystudio107/dotenvy": "^1.1.0"
  },
  "autoload": {
    "psr-4": {
      "modules\\sitemodule\\": "modules/sitemodule/src/"
    }
  },
  "config": {
    "sort-packages": true,
    "optimize-autoloader": true,
    "platform": {
      "php": "7.0"
    }
  },
  "scripts": {
    "craft-update": [
      "@php craft migrate/all",
      "@php craft project-config/sync",
      "@php craft clear-caches/all"
    ],
    "post-root-package-install": [
      "@php -r \"file_exists('.env') || copy('example.env', '.env');\""
    ],
    "post-update-cmd": "@craft-update",
    "post-install-cmd": "@craft-update"
  }
}

So the craft-update script automatically does the following when our container starts up:

All migrations are run
Project Config is sync’d
All caches are cleared

Starting from a clean slate like this is so helpful in terms of avoiding silly problems like things being cached, not up to date, etc.

Service: webpack

webpack is the build tool that we use for building the CSS, JavaScript, and other parts of our application.

The setup used here is entirely based on the An Annotated webpack 4 Config for Frontend Web Development article, just with some settings tweaked.

That means our webpack build process runs entirely inside of a Docker container, but we still get all of the Hot Module Replacement goodness for local development.

This service is composed of a base image that contains node itself, all of the Debian packages needed for headless Chrome, the npm packages we’ll always need to use, and then the project-specific image that contains whatever additional things are needed for our project.


FROM node:11

# Install packages for headless chrome
# https://medium.com/@ssmak/how-to-fix-puppetteer-error-while-loading-shared-libraries-libx11-xcb-so-1-c1918b75acc3
RUN apt-get update \
    && \
    # apt Debian packages
    apt-get install -y \
        ca-certificates \
        fonts-liberation \
        gconf-service \
        libgl1-mesa-glx \
        libasound2 \
        libatk1.0-0 \
        libc6 \
        libcairo2 \
        libcups2 \
        libdbus-1-3 \
        libexpat1 \
        libfontconfig1 \
        libgcc1 \
        libgconf-2-4 \
        libgdk-pixbuf2.0-0 \
        libglib2.0-0 \
        libgtk-3-0 \
        libnspr4 \
        libpango-1.0-0 \
        libpangocairo-1.0-0 \
        libstdc++6 \
        libx11-6 \
        libx11-xcb1 \
        libxcb1 \
        libxcomposite1 \
        libxcursor1 \
        libxdamage1 \
        libxext6 \
        libxfixes3 \
        libxi6 \
        libxrandr2 \
        libxrender1 \
        libxss1 \
        libxtst6 \
        libappindicator1 \
        libnss3 \
        lsb-release \
        wget \
        xdg-utils

We’ve based the container on the node image, tagged at version 11

We’re then adding a bunch of Debian packages that we need in order to get headless Chrome working (needed for Critical CSS generation), as well as other libraries for the Sharp image library to work effectively.

By itself, this image won’t do much for us, and in fact we don’t even spin up this image. But we’ve built this image, and made it available as nystudio107/webpack-dev-base on DockerHub.

Since it’s pre-built, we don’t have to build it every time, and can layer on top of this image anything project-specific via the webpack-dev-craft container image:


FROM nystudio107/webpack-dev-base

WORKDIR /var/www/project/docker-config/webpack-dev-craft/

# We'd normally use `npm ci` here, but by using `install`:
# - If `package-lock.json` is present, it will install what is in the lock file
# - If `package-lock.json` is missing, it will update to the latest dependencies
# and create the `package-lock-json` file
# Run our webpack build in debug mode
CMD if [! -f "./package-lock.json"]; then \
        npm install; \
    fi \
    && \
    if [! -d "./node_modules" -o ! "$(ls -A ./node_modules)"]; then \
        npm install; \
    fi \
    && \
    npm run debug

Then, just like with the php-dev-craft image, we do a bit of magic to do a npm install, but only if:

The package-lock.json file doesn’t exist
The node_modules/ directory doesn’t exist, or is empty

We have to do the npm install as part of the Docker image CMD because the file system mounts aren’t in place until the CMD is run.

This allows us to update our npm dependencies just by deleting the package-lock.json file or the node_modules/ directory, and doing docker-compose up

The alternative is doing a docker exec -it craft_webpack_1 /bin/bash to open up a shell in our container, and running the command manually.

All Aboard!

Hopefully this annotated Docker config has been useful to you. If you use Craft CMS, you can dive in and start using it yourself; if you use something else entirely, the concepts here should still be very salient for your project.

I think that Docker — or some other conceptually similar containerization strategy — is going to be an important technology going forward. So it’s time to jump on board.

As mentioned earlier, the Docker config used here is used in both the devMode.fm GitHub repo, and in the nystudio107/craft boilerplate Composer project if you want to see some “in the wild” examples.

Happy containerizing!

Post-Mortem: Outbreak Database

Andrew Welch — Tue, 03 Mar 2020 05:00:00 +0000

Post-Mortem: Outbreak Database

Modernizing an aging custom PHP website with Craft CMS for content management, and a hybrid Twig/Vue.js + Vuex + Axios + GraphQL on the frontend

Andrew Welch / nystudio107

I was contacted to do overflow work for a freelancer who found himself in the enviable position of having too much work booked.

The project was something that will be familiar to most web developers, which was to take an old website OutbreakDatabase.com and modernize it.

This article describes the higher-level decisions made while working on the project; if you want to get into the technical implementation, check out the Using the Craft CMS “headless” with the GraphQL API article.

N.B.: While my role on the project is finished, the project may or may not be live at the time of this writing.

The custom-built Cake PHP website was starting to show its age, both visually and technologically.

The client wanted a website that was easier for content authors to maintain the hygiene of the data in the outbreak database, and the site just needed an overall refresh to carry it forward for the next 10 years.

The website describes itself thusly:

Outbreak Database is a resource that provides access to food poisoning outbreak data in one easy to search place, dating back to 1993.

They just didn’t want the website to look like it dated back to 1993.

The Initial Handoff

The design for the website was already done, and the less interesting (to me anyway) work of data migration to Craft CMS was done already as well.

Bonus for me.

I was given access to the existing site, a CSS file that was being used to style this project and several other “mini-site” projects for the client, and some Twig templates that showed the mocked out design.

The clients goals were:

Make the outbreak database easier to maintain for the content authors
Make the frontend easier to use by researchers and journalists
Modernize the website underpinnings
Potentially provide an API to allow other parties to access the database directly

Other than that, I was given pretty much free rein to do whatever I thought was best. Which is a level of trust I really enjoy in my relationship with the original freelance developer.

Luckily for me, using Craft CMS as a backend ensures that the first two bullet points are already taken care of by Craft CMS’s excellent content modeling & authoring capabilities.

As I do for any project I work on, I spend a bit of time upfront learning about the client, their goals, etc. The normal stuff.

Then I sit down to think about what technologies and techniques I could apply to help them reach their goals.

GraphQL as an API

While the actual design of the website was not in my control, the technological underpinnings of the website and the user experience definitely was.

I wanted to use GraphQL over the Element API not just because it was less work, but because it provided a self-documented, strictly typed API for us automatically. GraphQL is a documented, widely embraced standard, so plenty of learning materials are available.

Since the client had a stated intention of wanting to be able to provide others access to the database, I immediately thought of GraphQL.

It was a nice, clean, modern way to present standardized access to data, that allows researchers to query for just the data that they are looking for. Since Pixel & Tonic had recently released a first-party GraphQL implementation for Craft CMS 3.3, it seemed like a lock.

There was a rub, however.

At the time, the GraphQL implementation didn’t support querying based on custom fields, which we needed for the faceted search. So we were left with the prospect of:

Writing a custom Element API implementation
Using Mark Huot’s CraftQL plugin
???

So like any responsible developer, I went with ???. Which in this case meant filing some issues for the Craft CMS developers to see if the concerns could be addressed.

Fortunately, we weren’t the only developers wanting this functionality, so Andris rolled his sleeves up and got it implemented in Craft CMS 3.4.

We were in business.

Adopting Vue + Vuex + Axios

Since we’d already decided on GraphQL as an API, I thought the best way to ensure we were building out an API others could access would be to consume that API ourselves.

So instead of using Craft’s built-in Element Queries for accessing data via Twig, I adopted Vue.js and Axios.

We’d use Vue to help make writing the interactive UI easier to do, and Axios to send along our GraphQL queries to the Craft CMS backend.

Vuex is a global data store that we’d leverage to stash the data fetched via Axios, and make it available to all of our Vue.js components.

Here’s what the original website UX was like for searching:

So pretty typical for an older website design: a form where you blindly enter search criteria, click the Search button, and a results page shows up.

If you make a mistake, or don’t find what you want, you hit the back button, and try again.

The new design and UX handed off to me looked visually nicer:

While this looks better, it operated much the same: enter your search criteria, click a button, go to a search results page. Hit the back button to try again if you don’t get what you want.

I thought we could do better, and Vue.js + Vuex + Axios + GraphQL would make doing that easier.

Doing Better

A great part of my satisfaction working on renovating older sites is the goal of making the world just a little bit better. We don’t always hit the mark dead-on, but striving to improve things is what motivates me.

So here’s what we ended up with:

First I eliminated the “search results page”; instead, the search results would be displayed interactively right below the query. As soon as you start typing, it starts searching (debounced of course), and a little spinner shows you so (thanks, vue-simple-spinner).

Clicking on the Search button or hitting the Return/Enter key would smoothly autoscroll (thanks, vue2-smooth-scroll) to view the search results.

I think the UI should be reworked a bit to make this a little less bulky so we can see more of the search results, but already I think we have a nice improvement.

People can interactively see the results of their search query, and make adjustments as needed without hopping back and forth between pages.

But we didn’t want to lose the ability of being able to copy a search result from the address bar, and send it to colleagues. So a little magic was done to update the address bar with a proper search?keywords= URL.

Next up was to eliminate some of the “I don’t know what to search for” problem. Instead of providing just an empty box where you type what criteria you want, we’d provide an auto-complete lookup of available choices (thanks, @trevoreyre/autocomplete-vue):

I think this helps greatly with the UX, because researchers can just start typing, and they’ll see a list of possible things they can choose from.

This also adds some transparency to the database hygiene, and allows the content authors to easily see duplicated data.

The CSS Problem

Whenever I start on a new project, I greatly look forward to refactoring the site to use Tailwind CSS. If you’re not on-board the Tailwind express yet, do give it a look, I’ve yet to know of anyone who has used it, and moved back to a more traditional BEM approach.

I’d be willing to use some pro-bono hours to do the refactoring myself if it isn’t included in the project. But in this case, the CSS was being used on a number of sites to give them all a similar look.

So even if I did the CSS refactoring to Tailwind CSS on my own time, it wouldn’t mesh well with their goals of having one CSS file for multiple sites.

So I decided to roll their CSS in as legacy/styles.css and use my normal Tailwind CSS + PurgeCSS setup to to override styles or add new styles:


/**
 * app.css
 *
 * The entry point for the css.
 *
 */

/**
 * This injects Tailwind's base styles, which is a combination of
 * Normalize.css and some additional base styles.
 */
 @import 'tailwindcss/base';

/**
 * This injects any component classes registered by plugins.
 *
 */
@import 'tailwindcss/components';

/**
 * Here we add custom component classes; stuff we want loaded
 * *before* the utilities so that the utilities can still
 * override them.
 *
 */
@import './components/global.pcss';
@import './components/typography.pcss';
@import './components/webfonts.pcss';

/**
 * Legacy CSS used for the project, rather than rewriting it in Tailwind
 */
@import './legacy/styles.css';

/**
 * Include styles for individual pages
 */
@import './pages/homepage.pcss';

/**
 * Include vendor css.
 */
@import './vendor.pcss';

/**
 * This injects all of Tailwind's utility classes, generated based on your
 * config file.
 */
@import 'tailwindcss/utilities';

/**
 * Forced overrides of the legacy CSS
 */
@import './components/overrides.pcss';

This gives me the best of both worlds:

I can use Tailwind CSS’s utility classes for additional styling or to override the base CSS as needed
The existing legacy styles.css is imported wholesale, so they can update it as they see fit

Hybrid Website

This website is what I’d term a “hybrid” website, in that it uses both Twig and Vue to render content.

It was done this way for practical reasons. The project was already using Twig to render pages, and the budget wasn’t there to redo the tooling to use JAMstack with something like Gridsome. The benefits of doing so were also dubious in this case.

So instead we dropped Vue.js into the mix just for the dynamic components on the page. For example, this is what the homepage looks like:


{% extends "_layouts/generic-page-layout.twig" %}

{% block headLinks %}
    {{ parent() }}
{% endblock headLinks %}

{% block content %}
    <div class="section--grey-pattern section--grey-pattern-solid section--mobile-gutter-none"
         style="min-height: 648px;"
    >
        <div id="component-container">
        </div>
    </div><!-- /.section-/-grey-pattern -->
{% endblock %}

{% block subcontent %}
{% endblock %}

{# -- Any JavaScript that should be included before </body> -- #}
{% block bodyJs %}
    {{ parent() }}
    {{ craft.twigpack.includeJsModule("home.js", true) }}
{% endblock bodyJs %}

This is using the Twig template setup described in the An Effective Twig Base Templating Setup article, and the <div id="component-container"> is where the Vue instance mounts:


// Home page
import { OutbreakMixins } from '../mixins/outbreak.js';
import { createStore } from '../store/store.js';
import '@trevoreyre/autocomplete-vue/dist/style.css';

// App main
const main = async() => {
    // Async load the vue module
    const [Vue, VueSmoothScroll] = await Promise.all([
        import(/* webpackChunkName: "vue" */ 'vue'),
        import(/* webpackChunkName: "vue" */ 'vue2-smooth-scroll'),
    ]);
    const store = await createStore(Vue.default);
    Vue.default.use(VueSmoothScroll.default);
    // Create our vue instance
    const vm = new Vue.default({
        render: (h) => {
            return h('search-form');
        },
        mixins: [OutbreakMixins],
        store,
        components: {
            'search-form': () => import(/* webpackChunkName: "searchform" */ '../../vue/SearchForm.vue'),
        },
    });

    return vm;
};

// Execute async function
main().then((vm) => {
});

// Accept HMR as per: https://webpack.js.org/api/hot-module-replacement#accept
if (module.hot) {
    module.hot.accept();
}

This means that our Vue components are not rendered until Vue & our components are loaded, executed, and mounted. However the resulting website still performs nicely:

So it was done this way in a nod to practicality, but should the client wish to jump to a full JAMstack setup in the future, we’re more than halfway home already.

This technique was described in the Using VueJS 2.0 with Craft CMS and Using VueJS + GraphQL to make Practical Magic articles if you want to learn more.

Final Thoughts

No project is ever perfect, especially software development projects. But I feel like the higher level decisions made helped to improve this project overall.

It’s a good example of how picking the right bits of technology can enable you to create an improved end result.

Flat Multi-Environment Config for Craft CMS 3

Andrew Welch — Sat, 29 Feb 2020 15:29:00 +0000

Flat Multi-Environment Config for Craft CMS 3

Multi-environment configs for Craft CMS are a mix of aliases, environment variables, and config files. This article sorts it all out, and presents a flat config file approach

Andrew Welch / nystudio107

Multi-environment configuration is a way to have your website or webapp do different things depending on where it is being served from. For instance, a typical setup might have the following environments:

dev — your local development environment
staging — a staging or User Acceptance Testing (UAT) server allowing stakeholders to test
production — the live production server

In each environment, you might want your project working differently. For example:

Debugging — in local dev you might want debugging tools enabled, but not in live production
Credentials — things like database credentials, API keys, etc. may be different per environment
Tracking — you probably don’t want Google Analytics data in local dev, but you probably do in live production

There are many other behaviors of settings that you might need or want to be different depending on where your project is being served from.

Additionally, you may have “secrets” that you don’t want stored in version control, and you also don’t want stored in your database.

Multi-environment configuration is for all of these things.

Enter the .ENV file

Craft CMS and a number of other systems have adopted the concept of a .env file which for storing environment variables and secrets.

This .env file is:

Never checked into source code control such as Git
Created manually in each environment where the project will run
Stores both environment variables and “secrets”

It’s a simple key/value that looks something like this:


# Craft database settings
DB_DRIVER=pgsql
DB_SERVER=localhost
DB_USER=project
DB_PASSWORD=XXXX
DB_DATABASE=project
DB_SCHEMA=public
DB_TABLE_PREFIX=
DB_PORT=5432

The values can be quoted or not (and indeed need to be quoted if they contain spaces), but keep in mind that if you used Docker, it doesn’t allow for quoted values.

You can also add comments to your .env files by proceeding a line with a # character.

Adding comments to your .env file is being nice to future-you

While there is some debate over the efficacy of storing secrets in this way, it’s become a commonly accepted practice that is “good enough” for non-critical purposes.

Additionally, this separation of environment variables & secrets from code — and from the database — allows for the natural use of more sophisticated measures should they be needed.

Heroku, Docker, Buddy.works, Forge, and many other tools work directly with .env files.

Environment variables can also be injected directly into the environment via the webserver and other tools, check out Dotenvy for details on automating that.

It’s a good practice to provide an example.env file with each of your projects that containers the boilerplate for the environment variables your project uses, as well as default values:


# Craft database settings
DB_DRIVER=pgsql
DB_SERVER=localhost
DB_USER=project
DB_PASSWORD=REPLACE_ME
DB_DATABASE=project
DB_SCHEMA=public
DB_TABLE_PREFIX=
DB_PORT=5432

The example.env file can and should be checked into Git, just make sure it has nothing sensitive in it such as passwords.

This gives you a nice starting point that you can rename to .env when configuring the project for a new environment. I use the screaming snake case constant REPLACE_ME to indicate non-default values that need to be filled in on a per-environment basis.

You’ll thank yourself the next time you go to set up the project, and so will others on your team.

Environment Variables in Craft CMS

In the context of Craft CMS, Pixel & Tonic has the canonical configuration information in their Environmental Configuration guide. However, we’re going to go into it in-depth, and provide a flexible reference implementation.

Craft CMS uses the vlucas/phpdotenv library for .env file handling. In fact, in the web/index.php we can see it being used thusly:


// Load dotenv?
if (class_exists('Dotenv\Dotenv') && file_exists(CRAFT_BASE_PATH.'/.env')) {
    Dotenv\Dotenv::create(CRAFT_BASE_PATH)->load();
}

If the Dotenv class exists, will look for a .env file in the project directory (set by the constant CRAFT_BASE_PATH) and try to load it.

What this actually does is it calls the PHP function putenv() for each key/value pair in your .env file, which sets those variables in PHP’s $_ENV superglobal.

The $_ENV superglobal contains variables from the PHP runtime environment, and the $_SERVER superglobal contains variables from the server environment. The PHP function getenv() reads variables from both of them of these superglobals, and is how you can access your .env environment variables.

“Superglobal” just means it’s a global variable defined by PHP, and available in every script. It isn’t faster than a speeding bullet or anything.

So if our .env file looked like this:


# Craft database settings
DB_DRIVER=pgsql
DB_SERVER=localhost
DB_USER=project
DB_PASSWORD=XXXX
DB_DATABASE=project
DB_SCHEMA=public
DB_TABLE_PREFIX=
DB_PORT=5432

Here’s what the the auto-complete dropdown looks like in the Craft CMS CP for the environment variables:

We could get a value from PHP like this:


$database = getenv('DB_DATABASE');

And we could get the same value from Twig like this:


{% set database = getenv('DB_DATABASE') %}

Aliases in Craft CMS

Craft CMS also has the concept of aliases, which are actually inherited from Yii2 aliases.

Yii2 is the webapp framework that Craft CMS is built on

Aliases can sometimes be confused with environment variables, but they really serve a different purpose. You’ll use an alias when:

The setting in question is a path
The setting in question is a URL

That’s it.

Could you use environment variables in these cases? Sure. But with aliases you can do things like have it resolve a path or URL that has a partial path in it (see below).

You define aliases in your config/general.php file in the aliases key, e.g.:


<?php
/**
 * General Configuration
 *
 * All of your system's general configuration settings go in here. You can see a
 * list of the available settings in vendor/craftcms/cms/src/config/GeneralConfig.php.
 *
 * @see craft\config\GeneralConfig
 */

return [
    // Craft config settings from .env variables
    'aliases' => [
        '@cloudfrontUrl' => getenv('CLOUDFRONT_URL'),
        '@web' => getenv('SITE_URL'),
        '@webroot' => getenv('WEB_ROOT_PATH'),
    ],
];

Note that we’re actually setting aliases from environment variables! They actually compliment each other.

Both @web and @webroot are aliases that Yii2 tries to set automatically for you. However, you should always set them explicitly (as shown above) to avoid potential cache poisoning.

Here’s how we can resolve an alias in PHP:


$path = Craft::getAlias('@webroot/assets');

To resolve an alias from Twig:


{% set path = alias('@webreoot/assets') %}

This demonstrates what you can do with aliases that you cannot do with environment variables, which is pass in a partial path and have the alias resolve with that path added to it.

You cannot do this with environment variables:


{% set path = getenv('WEB_ROOT_PATH/assets') %}

Similarly, you cannot put this in a CP setting in Craft:


$WEB_ROOT_PATH/assets

Here’s what the the auto-complete dropdown looks like in the Craft CMS CP for aliases:

parseEnv() does both

Since it’s commonplace that settings could be either aliases or environment variables (especially in CP settings), Craft CMS 3.1.0 introduced the convenience function parseEnv() that:

Fetches any environment variables in the passed string
Resolves any aliases in the passed string

So you can happily use it as a universal way to resolve both aliases and environment variables.

Here’s what it looks like in Twig:


{% set path = parseEnv(someVariable) %}
{# This is equivalent to #}
{% set path = alias(getenv(someVariable)) %}

Here’s what it looks like using parseEnv() via PHP:


$path = Craft::parseEnv($someVariable);
// This is equivalent to:
$path = Craft::getAlias(getenv($someVariable));

The parseEnv() function is a nice shortcut when you’re dealing with CP settings that could be aliases, environment variables, or both.

Config files in Craft CMS

Craft CMS also has the concept of config files, stored in the config/directory. These can either be “flat” config files that always return the same values regardless of environment:


// -- config/general.php --
return [
    'omitScriptNameInUrls' => true,
    'devMode' => true,
    'cpTrigger' => 'secret-word',
];

Or config files can be multi-environment:


// -- config/general.php --
return [
    // Global settings
    '*' => [
        'omitScriptNameInUrls' => true,
    ],

    // Dev environment settings
    'dev' => [
        'devMode' => true,
    ],

    // Production environment settings
    'production' => [
        'cpTrigger' => 'secret-word',
    ],
];

The key is **required* for a config file to be parsed as a multi-environment config file. If the * key is present, any settings in that sub-array are considered global settings.

Other keys in the array correspond with the CRAFT_ENVIRONMENT constant, which is set by:

The ENVIRONMENT variable in your .env, if present
The incoming URL’s hostname otherwise

Multi-environment config files are a carry-over from Craft 2, and continue to be quite useful.

Flat is beautiful

However, we’ve moved towards flat config files combined with .env files. Let’s have a look.

A real-world example

For a real-world example of using flat config files combined with environment variables and aliases, we’ll use the OSS’d devMode.fm website.

The reason we’ve moved away from using multi-environment config files is simplicity. It takes less mental space to know that any environment-specific settings or secrets are always coming from one place: the .env file.

Using flat config files with environment variables keeps all the per-environment settings in one place

This will save you time having to try to track down where a particular config setting is stored in each environment. It’s all in one place.

Here’s what the example.env file looks like for devMode.fm:


# Craft general settings
ALLOW_UPDATES=1
ALLOW_ADMIN_CHANGES=1
BACKUP_ON_UPDATE=0
DEV_MODE=1
ENABLE_TEMPLATE_CACHING=0
ENVIRONMENT=local
IS_SYSTEM_LIVE=1
RUN_QUEUE_AUTOMATICALLY=1
SECURITY_KEY=FnKtqveecwgMavLwQnX2I-dqYjpwZMR6

# Craft database settings
DB_DRIVER=pgsql
DB_SERVER=postgres
DB_USER=project
DB_PASSWORD=REPLACE_ME
DB_DATABASE=project
DB_SCHEMA=public
DB_TABLE_PREFIX=
DB_PORT=5432

# URL & path settings
ASSETS_URL=http://localhost:8000/
SITE_URL=http://localhost:8000/
WEB_ROOT_PATH=/var/www/project/cms/web

# Craft & Plugin Licenses
LICENSE_KEY=
PLUGIN_IMAGEOPTIMIZE_LICENSE=
PLUGIN_RETOUR_LICENSE=
PLUGIN_SEOMATIC_LICENSE=
PLUGIN_TRANSCODER_LICENSE=
PLUGIN_WEBPERF_LICENSE=

# S3 settings
S3_KEY_ID=REPLACE_ME
S3_SECRET=REPLACE_ME
S3_BUCKET=devmode-bucket
S3_REGION=us-east-2
S3_SUBFOLDER=

# CloudFront settings
CLOUDFRONT_URL=https://dnzwsrj1eic0g.cloudfront.net
CLOUDFRONT_DISTRIBUTION_ID=E17SKV1U1OTZKW
CLOUDFRONT_PATH_PREFIX=

# Redis settings
REDIS_HOSTNAME=redis
REDIS_PORT=6379
REDIS_DEFAULT_DB=0
REDIS_CRAFT_DB=3

# webpack settings
PUBLIC_PATH=/dist/
DEVSERVER_PUBLIC=http://localhost:8080
DEVSERVER_HOST=0.0.0.0
DEVSERVER_POLL=0
DEVSERVER_PORT=8080
DEVSERVER_HTTPS=0

# Twigpack settings
TWIGPACK_DEV_SERVER_MANIFEST_PATH=http://webpack:8080/
TWIGPACK_DEV_SERVER_PUBLIC_PATH=http://webpack:8080/

# Disqus settings
DISQUS_PUBLIC_KEY=
DISQUS_SECRET_KEY=

# Google Analytics settings
GA_TRACKING_ID=UA-69117511-5

# FastCGI Cache Bust settings
FAST_CGI_CACHE_PATH=

Because we’re using Project Config to allow us to easily deploy site changes across environments, we have to be mindful to put things like our Craft license key, plugin license keys, and other secrets into our .env file

Otherwise we’d end up with secrets checked into our git repo, which is not ideal from a security point of view.

While this .env file may look long, remember that it’s consolidating all of the environment variables in one place

Note also that the .env settings are logically grouped, with comments.

Let’s have a look at how we utilize these environment variables in our config/general.php file:


<?php
/**
 * General Configuration
 *
 * All of your system's general configuration settings go in here. You can see a
 * list of the available settings in vendor/craftcms/cms/src/config/GeneralConfig.php.
 *
 * @see craft\config\GeneralConfig
 */

return [
    // Craft config settings from .env variables
    'aliases' => [
        '@assetsUrl' => getenv('ASSETS_URL'),
        '@cloudfrontUrl' => getenv('CLOUDFRONT_URL'),
        '@web' => getenv('SITE_URL'),
        '@webroot' => getenv('WEB_ROOT_PATH'),
    ],
    'allowUpdates' => (bool)getenv('ALLOW_UPDATES'),
    'allowAdminChanges' => (bool)getenv('ALLOW_ADMIN_CHANGES'),
    'backupOnUpdate' => (bool)getenv('BACKUP_ON_UPDATE'),
    'devMode' => (bool)getenv('DEV_MODE'),
    'enableTemplateCaching' => (bool)getenv('ENABLE_TEMPLATE_CACHING'),
    'isSystemLive' => (bool)getenv('IS_SYSTEM_LIVE'),
    'resourceBasePath' => getenv('WEB_ROOT_PATH').'/cpresources',
    'runQueueAutomatically' => (bool)getenv('RUN_QUEUE_AUTOMATICALLY'),
    'securityKey' => getenv('SECURITY_KEY'),
    'siteUrl' => getenv('SITE_URL'),
    // Craft config settings from constants
    'cacheDuration' => false,
    'defaultSearchTermOptions' => [
        'subLeft' => true,
        'subRight' => true,
    ],
    'defaultTokenDuration' => 'P2W',
    'enableCsrfProtection' => true,
    'errorTemplatePrefix' => 'errors/',
    'generateTransformsBeforePageLoad' => true,
    'maxCachedCloudImageSize' => 3000,
    'maxUploadFileSize' => '100M',
    'omitScriptNameInUrls' => true,
    'useEmailAsUsername' => true,
    'usePathInfo' => true,
    'useProjectConfigFile' => true,
];

// Craft config settings from .env variables

The settings under this comment, including the aliases, are all set from .env environment variables via getenv().

Note that we’re explicitly typecasting the boolean values with (bool) because they are set with either 0 (false) or 1 (true) in the .env file, because true and false are both strings. Normally this isn’t a problem, but there can be edge cases with weakly typed languages like PHP.

// Craft config settings from constants

The settings under this comment are settings that we typically want to adjust from their default, but we don’t need them to be different on a per-environment basis.

You can look up what the various config settings are on the Craft CMS General Config Settings page.

Let’s have a look at the config/db.php file:


<?php
/**
 * Database Configuration
 *
 * All of your system's database connection settings go in here. You can see a
 * list of the available settings in vendor/craftcms/cms/src/config/DbConfig.php.
 *
 * @see craft\config\DbConfig
 */

return [
    'driver' => getenv('DB_DRIVER'),
    'server' => getenv('DB_SERVER'),
    'user' => getenv('DB_USER'),
    'password' => getenv('DB_PASSWORD'),
    'database' => getenv('DB_DATABASE'),
    'schema' => getenv('DB_SCHEMA'),
    'tablePrefix' => getenv('DB_TABLE_PREFIX'),
    'port' => getenv('DB_PORT')
];

These settings are all pretty straightforward, we’re just reading in secrets or settings that may be different per environment from .env environment variables via getenv().

Finally, let’s have a look at the config/app.php file that lets you configure just about any aspect of the Craft CMS webapp:


<?php
/**
 * Yii Application Config
 *
 * Edit this file at your own risk!
 *
 * The array returned by this file will get merged with
 * vendor/craftcms/cms/src/config/app/main.php and [web|console].php, when
 * Craft's bootstrap script is defining the configuration for the entire
 * application.
 *
 * You can define custom modules and system components, and even override the
 * built-in system components.
 */

return [
    'modules' => [
        'site-module' => [
            'class' => \modules\sitemodule\SiteModule::class,
        ],
    ],
    'bootstrap' => ['site-module'],
    'components' => [
        'deprecator' => [
            'throwExceptions' => YII_DEBUG,
        ],
        'redis' => [
            'class' => yii\redis\Connection::class,
            'hostname' => getenv('REDIS_HOSTNAME'),
            'port' => getenv('REDIS_PORT'),
            'database' => getenv('REDIS_DEFAULT_DB'),
        ],
        'cache' => [
            'class' => yii\redis\Cache::class,
            'redis' => [
                'hostname' => getenv('REDIS_HOSTNAME'),
                'port' => getenv('REDIS_PORT'),
                'database' => getenv('REDIS_CRAFT_DB'),
            ],
        ],
        'session' => [
            'class' => \yii\redis\Session::class,
            'redis' => [
                'hostname' => getenv('REDIS_HOSTNAME'),
                'port' => getenv('REDIS_PORT'),
                'database' => getenv('REDIS_CRAFT_DB'),
            ],
            'as session' => [
                'class' => \craft\behaviors\SessionBehavior::class,
            ],
        ],
    ],
];

Here we’re bootstrapping our Site Module as per the Enhancing a Craft CMS 3 Website with a Custom Module article.

Then we’re configuring the deprecator component so that if devMode is enabled, deprecation errors that would normally be logged instead cause an exception to be thrown.

This is playing Craft in “hard” mode

This can be really useful for tracking down and fixing deprecation errors as they happen.

Finally, we configure Redis, and use it as the Yii2 caching method, and more importantly for PHP sessions. You can read more about setting up Redis in Matt Gray’s excellent Adding Redis to Craft CMS article.

Multi-site Multi-Environment in Craft CMS

Craft CMS has powerful multi-site baked in that allows you to create localizations of existing sites, or sister-sites all managed under one umbrella.

In the context of a multi-environment config the siteUrl in your config/general.php changes from a string to an array:


    'siteUrl' => [
        'en' => getenv('EN_SITE_URL'),
        'fr' => getenv('FR_SITE_URL'),
    ],

The key in the array is the language handle, and the value is the siteUrl for that site.

And your .env would have the corresponding URLs in it:


# Site URLs
EN_SITE_URL=https://english-example.com/
FR_SITE_URL=https://french-example.com/

You can have a separate .env environment variable for each site as shown above, or if your sites will all have the same base URL, you can define an alias:


    'aliases' => [
        '@baseSiteUrl' => getenv('SITE_URL'),
    ],

And then your siteUrl array would just look like this:


    'siteUrl' => [
        'en' => '@baseSiteUrl/en',
        'fr' => '@baseSiteUrl/fr',
    ],

This makes it a little cleaner to set up and maintain, and it’s fewer environment variables you need to change.

Winding Down

That about wraps it up our spelunking into the world of multi-environment configs in Craft CMS 3.

Hopefully this in-depth exploration of how environment variables work combined with real-world examples have helped to give you a better understanding of how you can create a solid multi-environment configuration for Craft CMS 3.

If you adopt some of the methodologies discussed here, you will reap the benefits of a proven setup.

The approach presented here is also used in the nystudio107 Craft 3 CMS scaffolding project. Enjoy!

Setting Up AWS S3 Buckets + CloudFront CDN for your Assets

Andrew Welch — Thu, 23 Jan 2020 23:01:00 +0000

Setting Up AWS S3 Buckets + CloudFront CDN for your Assets

Using a cloud storage system like AWS S3 with a CDN distribution can be a convenient and inexpensive way to store your assets. Here’s how to set it up right.

Andrew Welch / nystudio107

Assets like images, PDFs, and other files are often an important part of the “content” that a Content Management System handles.

Although this article was written with Craft CMS in mind, the vast majority of the article applies generically to any CMS or website.

Craft CMS has some fantastic native handling of said assets, which by default are stored in folders on your server.

However, it can be convenient to use a cloud-based storage system like Amazon Web Services (AWS) Simple Storage Service (S3):

You’ll never run out of disk space
Your assets are inherently backed up & stored off-site
Your assets can go to long-term backup in AWS Glacier easily
You can leverage a Content Delivery Network (CDN) in the form of Cloudfront
Eliminates the need to sync assets between local dev, staging, and production environments
It’s cheap (or even free) & scalable

There are other advantages as well, but we’ll just assume you’re on-board, and get right into how to set it up.

Setting up S3

S3 stores files in “buckets”, and while you can serve assets directly from an S3 bucket, I’d strongly recommend against it.

Don’t serve assets directly from S3 buckets

S3 buckets are intended for asset storage, not serving assets. It’ll work, but you’ll only be serving them out of one geographic region (wherever your S3 bucket was created), and it really wasn’t designed to for that.

Instead, use AWS CloudFront as your Content Delivery Network (CDN) that actually serves up your assets.

Here’s what it looks like conceptually:

The idea is that when a person loads one of your web pages with an image on it, the image will point to a CloudFront URL.

If that image is in the cache, it’ll return it from a CDN Edge server that is geographically near the person loading the page. This makes it quick, with low latency.

If the image isn’t in the cache, CloudFront pulls it from your S3 bucket, returns it to the person, and propagates the image to the CDN Edge locations.

S3 Buckets shouldn’t be publicly accessible

The rule of thumb here is that S3 buckets aren’t publicly accessible. Instead, CloudFront is given permission to pull assets from the S3 buckets.

That said, let’s get into setting it all up.

Step 1: Have an AWS account

If you don’t already have an AWS account, you’re going to need one. Go to your AWS Console and either log in to your account, or create a new account.

If you want the billing to go directly to your clients, you can either create an AWS account for them, or use their existing account.

If you factor in the relatively low cost of S3 into your maintenance contracts or the like, you can just your account for all of your clients.

If you don’t have one already, you should also set up an admin account for AWS now.

Step 2: Create an S3 Bucket

Now that we have an AWS account, the first thing we’re going to do is set up our S3 bucket. We’re going to use kebab-case for all of our AWS entity names, and we’ll use the format:

project-name + - + descriptor

We’ll be setting up S3 + CloudFront for devMode.fm, so the project name is devmode, and the bucket name is devmode-bucket.

In your AWS Console, click on Services → S3 → + Create Bucket. Fill in the name of the bucket, and click through to the end (we won’t be changing any settings from the default other than the name):

Note that for Permissions, it’s set to Block all public access. As discussed above, our S3 bucket will be private, and the CloudFront distribution will be public.

For most projects, I create one bucket, and use sub-folders in the bucket for different Asset Volumes. You can certainly also create more than one bucket if that’ll work better for you.

Step 3: Create a Custom Policy

Next we’ll set up a custom Identity and Access Management (IAM) policy to control access. IAM policies can be attached to any AWS object, and they control who can access what, with some fairly fine-grained permissions.

We’re using a custom policy because we want to grant as little access as possible, but still have it work correctly.

This isn’t a fancy watch. We don’t add complications just because they look cool

From your AWS Console, click on Services → IAM → Policies → Create Policy.

Click on the JSON tab, and then paste this in:


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "acm:ListCertificates",
                "cloudfront:GetDistribution",
                "cloudfront:GetStreamingDistribution",
                "cloudfront:GetDistributionConfig",
                "cloudfront:ListDistributions",
                "cloudfront:ListCloudFrontOriginAccessIdentities",
                "cloudfront:CreateInvalidation",
                "cloudfront:GetInvalidation",
                "cloudfront:ListInvalidations",
                "elasticloadbalancing:DescribeLoadBalancers",
                "iam:ListServerCertificates",
                "sns:ListSubscriptionsByTopic",
                "sns:ListTopics",
                "waf:GetWebACL",
                "waf:ListWebACLs"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListAllMyBuckets"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::REPLACE-WITH-BUCKET-NAME"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::REPLACE-WITH-BUCKET-NAME/*"
            ]
        }
    ]
}

Make sure you replace REPLACE-WITH-BUCKET-NAME with the name of the bucket we created in Step 2 (in our case devmode-bucket), in both places.

Then click on Review Policy , and give your policy a name (in our case devmode-policy) and description, then click on Create Policy :

Step 4: Create a Group

Instead of attaching the IAM policy we created to a bucket or a user, we’re going to create a group, and attach it to the group.

The IAM policy gets attached to the group

We’re doing this because it makes it trivial to move users in and out of the group, and the group is what is controlling access permissions.

You also then won’t be out of luck if you somehow lose the user credentials, you can just create a new user and assign it to the group.

From your AWS Console, click on Services → IAM → Groups → Create New Group , and give your group a name (in our case devmode-group).

At the Attach Policy screen, search for the policy we created in Step 3, and check it and click Next Step :

Then click on Create Group to create your new group.

Step 5: Create a User

Next we’re going to create a user, and assign it to the group we just created. The user we create will be a generic project user, rather than an actual person.

From your AWS Console, click on Services → IAM → Users → Add user , and give your user a name (in our case devmode-user). Also check only the Programatic access checkbox under Access type:

This ensures that using these credentials, there’s AWS Management Console access. For that, you’ll use your regular admin account & credentials.

Click on Next: Permissions , then add the user to the group we created in Step 4:

Then click on Next: Tags (we don’t set any tags here), then click on Next: Review :

Then click on Create User. You’ll be taken to a screen where you can see your Access key ID and Secret access key :

Click on Download .csv to download a CSV file that has your credentials in it. You will need these credentials to access your S3 bucket, and this is the only time you’ll be able to retrieve them.

Step 6: Create a CloudFront Distribution

Now we need to create a CloudFront Distribution that will be acting as our CDN, and actually delivering our assets to the users who request them.

From your AWS Console, click on Services → CloudFront → Create Distribution , and click on the Get Started button below the Web heading.

Alter the following settings:

Origin Domain Name — choose the origin for the S3 bucket we created in Step 2
Restrict Bucket Access — Yes
Origin Access Identity — Create a New Identity
Grant Read Permissions on Bucket — Yes, Update Bucket Policy
Viewer Protocol Policy — Redirect HTTP to HTTPS
Allowed HTTP Methods — GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE
Object Caching — Customize
Minimum TTL — 1296000
Maximum TTL — 31536000
Default TTL — 1296000
Compress Objects Automatically — Yes

Here’s a full screenshot for the settings we’re using for devMode.fm:

Then click on Create Distribution.

You’ll be taken to a strange “CloudFront Private Content Getting Started” page; this isn’t an error page, and there aren’t any more steps to take.

We just need to grab a couple of settings from our newly created CloudFront distribution.

From your AWS Console, click on Services → CloudFront → then click on the CloudFront distribution we just created:

We’re going to need the Distribution ID and Domain Name settings, so copy them down somewhere.

We’re all done with the AWS S3 + CloudFront setup!

Step 7: Configure your Asset Volumes in Craft CMS

Next we need to configure Craft CMS to use our new S3 bucket setup. If you’re using something other than Craft CMS, you’ll need to fill in the analogous settings.

The key point to remember is that the public URL will be our CloudFront distribution URL (in our case https://dnzwsrj1eic0g.cloudfront.net); make sure you add the https:// protocol to it.

First, we’ll need to install the first-party Amazon S3 plugin.

Next we’ll need to set up our Asset Volumes. I use Environment Variables in my .env file to store all of my secrets, so they aren’t in the database, and the don’t end up in Git via Project Config:


# S3 settings
S3_KEY_ID=XXXXXXXXXX
S3_SECRET=XXXXXXXXXX
S3_BUCKET=devmode-bucket
S3_REGION=us-east-2

# CloudFront settings
CLOUDFRONT_URL=https://dnzwsrj1eic0g.cloudfront.net
CLOUDFRONT_DISTRIBUTION_ID=E17SKV1U1OTZKW
CLOUDFRONT_PATH_PREFIX=

In the Craft CMS CP, go to Settings → Assets → New volume , and fill in the volume settings as shown:

Note that Bucket has been set to Manual so we can use our environment variables, and we’ve manually added episodes to Subfolder.

Also note that we have turned Make Uploads Public OFF. If you enable it, it will set a manual ACL on your S3 assets to allow public access, which isn’t what we want. We want private S3 assets with public access through CloudFront only

If you’re using Project Config, your project.yaml will end up looking like the following:


  e69c8edb-d562-4367-9a05-91a6fd2c7d99:
    name: 'Devmode Episodes'
    handle: devmodeEpisodes
    type: craft\awss3\Volume
    hasUrls: true
    url: $CLOUDFRONT_URL
    settings:
      subfolder: episodes
      keyId: $S3_KEY_ID
      secret: $S3_SECRET
      bucketSelectionMode: manual
      bucket: $S3_BUCKET
      region: $S3_REGION
      expires: '3 months'
      makeUploadsPublic: ''
      storageClass: ''
      cfDistributionId: $CLOUDFRONT_DISTRIBUTION_ID
      cfPrefix: $CLOUDFRONT_PATH_PREFIX
      autoFocalPoint: ''
    sortOrder: 4

Look, mah, no secrets!

Then we can upload an image to our new Asset Volume:

And we can verify that the image is indeed coming from our CloudFront distribution URL:

Fill your Buckets

That’s all you need to start enjoying the benefits of cloud storage in S3, and CloudFront as a global Content Delivery Network.

This whole manual setup can be automated via an AWS CloudFormation stack… but that’s left as an exercise for the reader (or maybe another article).

Thanks to all around great guy Jonathan Melville of CodeMDD.io for this help with this article.

Happy uploading!

Using the Craft CMS "headless" with the GraphQL API

Andrew Welch — Mon, 20 Jan 2020 05:00:00 +0000

Using the Craft CMS “headless” with the GraphQL API

Craft CMS 3.3 added a GraphQL layer that gives your website a formalized, structured API out of the box. Here’s how to use GraphQL + Craft CMS as a “headless” CMS

Andrew Welch / nystudio107

GraphQL & Craft CMS

This article explores coding patterns you might find useful when working with Craft CMS’s GraphQL API via real-world examples, and also why you might want to use GraphQL to begin with.

GraphQL describes itself with the following pithy tagline:

A query language for your API

And that’s exactly what it is. It’s a neutral layer that sits on top of whatever API layer you have, from databases like MySQL, Postgres, Mongo, etc. to custom built solutions.

In August 2019, Pixel & Tonic added GraphQL to Craft CMS 3.3, and vastly improved it in Craft CMS 3.4.

Prior to this, we relied on Mark Huot’s CraftQL plugin for GraphQL, as discussed in the Using VueJS + GraphQL to make Practical Magic article.

Should you use Mark’s CraftQL plugin or should you use Craft CMS’s first-party GraphQL implementation?

Unless you need mutations (the ability to change data in the Craft CMS backend via GraphQL), using the Craft CMS first-party GraphQL implementation in Craft 3.3 or later is the way to go.

Craft CMS’s first-party GraphQL API doesn’t currently support mutations, but it does offer a robust, performant data source for headless Craft CMS.

Why use GraphQL?

As a web developer, you’d use GraphQL if you were writing a frontend that is separate from the backend. You’d be using Craft CMS as a “headless” CMS for its excellent content authoring experience.

Perhaps you’re writing the frontend in a framework like React, Vue.js, Svelte, or one of the frameworks that layers on top of them like Next.js, Nuxt.js, Gatsby, Gridsome, or Sapper.

Or perhaps you’re jumping on the JAMstack bandwagon, and just need a self-hosted CMS like Craft CMS to manage your content.

If you need a way to get data out of Craft CMS, GraphQL is a great way to do it

Or maybe the project has an iPhone app that needs to communicate with a content management system on the backend for data.

In those cases, you don’t have access to Twig or the Element Queries to interact with Craft CMS and your data, but you still want to reap the benefits of Craft CMS’s wonderful & flexible content authoring experience.

You could write a custom API for Craft CMS using the Element API, but that can be a significant amount of work, and you’ll end up with something boutique, rather than an industry standard.

Enter the GraphQL

This is where GraphQL for Craft CMS excels. By virtue of creating your content models in Craft CMS, you automatically get a GraphQL API to access it, with no extra work on your part.

It’s also self-documenting, and strictly typed. You don’t need to define the GraphQL schemas, because you’ve already implicitly done that in Craft.

It just works.

You just need to be using Craft CMS “Pro” edition, and you’ll see GraphQL in your CP, and you can immediately start exploring the GraphQL API using the explorer (which is provided by a frontend tool called GraphiQL):

While it’s beyond the scope of this article to teach you GraphQL (there are many great resources online for that), there are a few things to note here:

In the upper-left pane, is the GraphQL query specifying what we’re searching
In the lower-left pane are variables we pass into the query, in JSON format
The middle pane is the data returned from the GraphQL endpoint as a result of our query
To the right pane is a searchable, documented schema reference

The fun thing about using this GraphQL explorer is that you can learn by just poking around.

You can press Option-Spacebar at any time for a list of auto-completed items that can appear in any given context. You can also use the Documentation Explorer to see your entire schema.

Combined with the official Craft CMS GraphQL API documentation, you can get pretty far just by exploring around in the GraphQL Explorer.

The Way of the GraphQL

Before we dive into some queries in JavaScript, I have a few tips I’d like to pass along to you to make your life with GraphQL easier.

One nice feature of the Craft CMS implementation of GraphQL is that it has a caching layer for your queries. While this is normally great for performance reasons, when you’re developing a site, you really want it off to avoid potentially dealing with cached results.

You can do this via the enableGraphQlCaching setting in your config/general.php file:


<?php
/**
 * General Configuration
 *
 * All of your system's general configuration settings go in here. You can see a
 * list of the available settings in vendor/craftcms/cms/src/config/GeneralConfig.php.
 *
 * @see \craft\config\GeneralConfig
 */

return [
    // Craft config settings from .env variables
    'enableGraphQlCaching' => (bool)getenv('ENABLE_GQL_CACHING'),
];

Here I set it to an Environment Variable, but you can set it to true or false directly if you prefer.

Craft CMS also has a headlessMode setting. This is intended for situations where Craft CMS will never be used to render any content on the frontend.

Due to limitations in how this works in Craft CMS 3.3, I recommend leaving it set to false, since elements lack URIs in the 3.3 implementation of headlessMode. This causes many plugins to not work properly, and other undesired behavior.

However, Pixel & Tonic has made improvements to how the headlessMode setting works in Craft CMS 3.4, so you can set it to true if you want Craft CMS to never render content, and simply be a data/API provider via GraphQL.

Something many people aren’t aware of is that you can use GraphQL in Twig if you want to:


{# Set the GraphQL query #}
{% set query %}
query organismsQuery($needle: String!)
{
    entries(section: "organisms", search: $needle, orderBy: "title") {
        title,
        slug,
        id
    }
}
{% endset %}

{# Set the variables to pass into the query #}
{% set variables = {
    'needle': 'Botulism'
} %}

{# Query the data via Twig! #}
{% set data = gql(query, variables) %}

{% dd data %}

The gql() Twig filter was added in Craft CMS 3.3.12. Why would you ever want to do such a thing?

Usually, you wouldn’t. You’d just use Element Queries instead… but it can be a nice way to experiment and play around with queries in a familiar/comfortable environment.

In fact, it hammers home a very important conceptual point with GraphQL in Craft CMS:

In Craft CMS, GraphQL queries map to Element Queries

GraphQL is a thin-ish layer in Craft CMS that handles mapping from a GraphQL queries to Element Queries. It works in both directions:

The GraphQL schema is derived from the underlying Craft CMS Sections, Element content models, etc.
GraphQL queries are internally mapped to Element Queries, which are then executed

The reason this is an important concept is that if you want to figure out how to do something in GraphQL, figure out how to do it via an Element Query first, and then work your way backwards.

So it can then be handy to use gql() in Twig, because you can write your Element Query as you normally would. Get it working the way you want it to, and then construct & test the analogous GraphQL query in the same place.

Finally, if you’re using PhpStorm, check out the GraphQL Schema Auto-Completion with PhpStorm article for blissful auto-complete query writing in your code.

Controller of Fury

I mentioned earlier that you get a GraphQL API without having to do any work with Craft CMS. While this is true, having a little custom controller as part of your site module can be helpful.

I always have a site module as part of my Craft CMS setups, as discussed in the Enhancing a Craft CMS 3 Website with a Custom Module article.

You always start with no custom code, but inevitably you end up needing a little piece here or there. And if you have a module in place already, it makes adding this in really easy.

In our case, we want a custom controller that our frontend can ping to:

actionGetCsrf() — get the current CSRF token to validate POST request submissions
actionGetGqlToken() — get a particular GraphQL token for access permissions
actionGetFieldOptions() — get the options in a Dropdown field in Craft CMS

POST requests are used to send data from the website along to an endpoint. While we could use GET for simple requests like actionGetGqlToken(), we need to send data to other endpoints like actionGetFieldOptions(), so we might as well use POST for all requests.

Here’s what our custom controller looks like:


<?php
/**
 * Site module for Craft CMS 3.x
 *
 * An example module for Craft CMS 3 that lets you enhance your websites with a
 * custom site module
 *
 * @link https://nystudio107.com/
 * @copyright Copyright (c) 2019 nystudio107
 */

namespace modules\sitemodule\controllers;

use Craft;
use craft\web\Controller;

use yii\web\Response;

/**
 * @author nystudio107
 * @package SiteModule
 * @since 1.0.0
 */
class SiteController extends Controller
{
    // Constants
    // =========================================================================

    const GQL_TOKEN_NAME = 'Site GQL Token';

    // Protected Properties
    // =========================================================================

    protected $allowAnonymous = [
        'get-csrf',
        'get-gql-token',
        'get-field-options',
    ];

    // Public Methods
    // =========================================================================

    /**
     * @inheritdoc
     */
    public function beforeAction($action): bool
    {
        // Disable CSRF validation for get-csrf POST requests
        if ($action->id === 'get-csrf') {
            $this->enableCsrfValidation = false;
        }

        return parent::beforeAction($action);
    }

    /**
     * @return Response
     */
    public function actionGetCsrf(): Response
    {
        return $this->asJson([
            'name' => Craft::$app->getConfig()->getGeneral()->csrfTokenName,
            'value' => Craft::$app->getRequest()->getCsrfToken(),
        ]);
    }

    /**
     * @return Response
     */
    public function actionGetGqlToken(): Response
    {
        $result = null;
        $tokens = Craft::$app->getGql()->getTokens();
        foreach ($tokens as $token) {
            if ($token->name === self::GQL_TOKEN_NAME) {
                $result = $token->accessToken;
            }
        }

        return $this->asJson([
            'token' => $result,
        ]);
    }

    /**
     * Return all of the field options from the passed in array of $fieldHandles
     *
     * @return Response
     */
    public function actionGetFieldOptions(): Response
    {
        $result = [];
        $request = Craft::$app->getRequest();
        $fieldHandles = $request->getBodyParam('fieldHandles');
        foreach ($fieldHandles as $fieldHandle) {
            $field = Craft::$app->getFields()->getFieldByHandle($fieldHandle);
            if ($field) {
                $result[$fieldHandle] = $field->options;
            }
        }

        return $this->asJson($result);
    }
}

While this custom controller isn’t necessary for using your GraphQL endpoint, it does make the experience a bit nicer. We grab the CSRF token, and send that CSRF token along with further requests, to allow Yii2 to validate the data submissions.

Schemas & Tokens

GraphQL in Craft CMS has a concept of Schemas and Tokens:

Schemas — schemas you can think of these as permissions, in a way, in that they define what parts of the underlying CraftCMS content you want exposed

Tokens — tokens are they “keys” that you pass along with your GraphQL queries, and they link to a schema

So what I do is:

Get the CSRF token for the current session
Use that CSRF to obtain a specific GraphQL token used for API access
Use that GraphQL token in all GraphQL request to the endpoint

In many cases, you won’t need to do this because you’ll just have one Public Schema that defines your GraphQL API. But if you want to potentially have varying levels of access, you’d created multiple tokens and use them to what schema you can use.

If you just want to use the special Public Schema, you don’t need any token at all. Just don’t send a bearer token down with your GraphQL queries, and it’ll just use whatever schema is defined in Public Schema.

N.B.: In our case, we want public access to do the data. The CRSF and GraphQL tokens are just for lightweight auth, it is not treated as a locked down “secret” for auth’ing. For that you’d want something like JSON Web Tokens (JWT), check out the Craft JWT Auth plugin.

Controller & GraphQL XHRs

Let’s get into some JavaScript that we use to communicate with our custom controller endpoints, and also the Craft CMS GraphQL endpoint.

First here is a little xhr.js helper to configure and execute generic XHRs:


// Configure the XHR api endpoint
export const configureXhrApi = (url) => ({
    baseURL: url,
    headers: {
        'X-Requested-With': 'XMLHttpRequest'
    }
});

// Execute an XHR to our api endpoint
export const executeXhr = async(api, variables, callback) => {
    // Execute the XHR
    try {
        const response = await api.post('', variables);
        if (response.data) {
            callback(response.data);
        }
    } catch (error) {
        console.error(error);
    }
};

The X-Requested-With: XMLHttpRequest header lets Craft CMS know that this is an “AJAX” request, which can affect how the request is processed.

Here’s how we get the CSRF from our custom controller endpoint:


import axios from 'axios';
import { configureXhrApi, executeXhr } from '../utils/xhr.js';

const CSRF_ENDPOINT = '/actions/site-module/site/get-csrf';

// Fetch & commit the CSRF token
export const getCsrf = async({commit, state}) => {
    const api = axios.create(configureXhrApi(CSRF_ENDPOINT));
    let variables = {
    };
    // Execute the XHR
    await executeXhr(api, variables, (data) => {
        commit('setCsrf', data);
    });
};

The above function is a Vuex Action, but Vuex is just used as a convenient place to store data. It’s not important that you use Vuex yourself to get something out of the examples presented here.

The code is just creating an Axios instance, executing the XHR, and stashing the returned CSRF data for future use.

We just use Axios here as opposed to something like Apollo Client just because all we really need is a lightweight way to communicate without GraphQL endpoint. We could also use the built-in Fetch API, but Axios handles some edge-case situations and browser compatibility for us.

Then this is how we get the GraphQL token:


import axios from 'axios';
import { configureGqlApi, executeGqlQuery } from '../utils/gql.js';

// Fetch & commit the GraphQL token
export const getGqlToken = async({commit, state}) => {
    const api = axios.create(configureXhrApi(TOKEN_ENDPOINT));
    let variables = {
        ...(state.csrf && { [state.csrf.name]: state.csrf.value }),
    };
    // Execute the XHR
    await executeXhr(api, variables, (data) => {
        commit('setGqlToken', data);
    });
};

Once again, this is a Vuex Action, but it doesn’t have to be. The important part is that it’s using the CSRF we obtained earlier, and sending that along with a request for our GraphQL token, which it then saves for future use.

The funky ...(state.csrf && { [state.csrf.name]: state.csrf.value }), line deserves some explanation. It’s using the JavaScript Spread Syntax to add a key/value pair to the variables object.

But it’s doing it in a conditional way, using the behavior of the JavaScript && Logical Operator to only spread the { [state.csrf.name]: state.csrf.value } object into variables only if state.csrf is truthy.

This works because JavaScript will return the object if state.csrf is truthy, and if not, it’ll return state.csrf itself, and the … spread syntax is smart enough to know not to spread null, so it spread nothing into variables.

Finally we have some helper JavaScript in gql.js (very similar to the xhr.js above) which configures and creates an Axios instance for querying Craft’s GraphQL endpoint:


// Configure the GraphQL api endpoint
export const configureGqlApi = (url, token) => ({
    baseURL: url,
    headers: {
        'X-Requested-With': 'XMLHttpRequest',
        ...(token && { 'Authorization': `Bearer ${token}` }),
    }
});

// Execute a GraphQL query by sending an XHR to our api endpoint
export const executeGqlQuery = async(api, query, variables, callback) => {
    // Execute the GQL query
    try {
        const response = await api.post('', {
            query: query,
            variables: variables
        });
        if (callback && response.data.data) {
            callback(response.data.data);
        }
        // Log any errors
        if (response.data.errors) {
            console.error(response.data.errors);
        }
    } catch (error) {
        console.error(error);
    }
};

This just adds an additional a Bearer Token to the header which Craft CMS will use to link to the schema that the request will have permission to access, and it adds some error logging specific to the GraphQL implementation.

A Simple Query

Let’s look at a simple query that uses the infrastructure we’ve described above:


export const organismsQuery =
    `
query organismsQuery($needle: String!)
{
    entries(section: "organisms", search: $needle, orderBy: "title") {
        title,
        slug,
        id
    }
}
`;

This is the same query we looked at in the GraphQL Explorer previously, which is executed as follows:


import axios from 'axios';
import { configureGqlApi, executeGqlQuery } from '../utils/gql.js';

const GRAPHQL_ENDPOINT = '/api';

// Fetch & commit the Organisms array from a GraphQL query
export const getOrganisms = async({commit, state}) => {
    const token = state.gqlToken ? state.gqlToken.token : null;
    const api = axios.create(configureGqlApi(GRAPHQL_ENDPOINT, token));
    let variables = {
        needle: 'useInSearchForm:*'
    };
    // Execute the GQL query
    await executeGqlQuery(api, organismsQuery, variables, (data) => {
        if (data.entries) {
            commit('setOrganisms', data.entries);
        }
    });
};

This is once again a Vuex Action, which uses the GraphQL token we’ve already obtained, and passes in useInSearchForm:* as a search parameter in needle.

This is looking for a lightswitch field in Craft CMS with the handle useInSearchForm, so that it’ll only return entries from the organisms section that have that lightswitch on.

This returns data that looks like this:


[
    {
        "title": "Bacillus cereus",
        "slug": "bacillus-cereus",
        "id": "10226"
    },
    {
        "title": "Bacterial toxin",
        "slug": "bacterial-toxin",
        "id": "14472"
    },
]

Querying a Single Thing

Normally all of your GraphQL queries will return an array of data, even if you’re only asking for one thing. While this consistency is nice, there are times when you really only ever want one thing returned.

In Craft CMS 3.4, singular versions of all of the queries were made available:

assets() → asset()
categories() → category()
entries() → entry()
globalSets() → globalSet()
tags() → tag()
users() → user()

This is analogous to the .all() → .one() methods used on Element Queries.

So we can leverage this for our queries where we only ever want one thing returned, so we don’t have to do ugly [0] array syntax:


export const outbreakDetailQuery =
    `
query outbreakDetailQuery($slug: [String])
{
    entry(section: "outbreaks", slug: $slug) {
        title,
        url,
        slug,
        ...on outbreaks_outbreaks_Entry {
            summary,
            beginningDate,
            states,
            country,
            hasTest,
            testResults,
            productSubjectToRecall,
            totalIll,
            numberIllByCaseDefinitionKnown,
            probableIll,
            possibleIll,
            confirmedIll,
            hospitalized,
            numberHospitalized,
            anyDeaths,
            numberOfDeaths,
            recallLinks {
                col1
                url
            },
            reportReferenceLinks {
                col1
                url
            },
            brands {
                title
            },
            locations {
                title
            },
            organisms {
                title
            },
            tags {
                title
            },
            vehicles {
                title
            }
        }
    }
}
`;

This rather large query exemplifies the GraphQL mindset, which is that you need to specify all of the data that you want returned. This is unlike Element Queries, where by default you get all of the data back.

Here’s what it looks like executing this query:


import axios from 'axios';
import { configureGqlApi, executeGqlQuery } from '../utils/gql.js';

const GRAPHQL_ENDPOINT = '/api';

// Fetch & commit an Outbreak detail from a GraphQL query
export const getOutbreakDetail = async({commit, state}) => {
    const token = state.gqlToken ? state.gqlToken.token : null;
    const outbreakSlug = state.outbreakSlug || null;
    const api = axios.create(configureGqlApi(GRAPHQL_ENDPOINT, token));
    let variables = {
        slug: outbreakSlug
    };
    // Execute the GQL query
    await executeGqlQuery(api, outbreakDetailQuery, variables, (data) => {
        if (data.entry) {
            commit('setOutbreakDetail', data.entry);
        }
    });
};

This code is almost exactly identical to the code we used to for getOrganisms(), except that we pass in a slug to look for in our entry() query rather than a search parameter.

Here’s what the data returned from this query looks like:


{
  "title": "2017 Multistate Outbreak of Salmonella Infantis Linked Mangoes (Suspected)",
  "url": "http://outbreakdatabase.test/outbreaks/2017-multistate-outbreak-of-salmonella-infantis-linked-mangoes-suspected",
  "slug": "2017-multistate-outbreak-of-salmonella-infantis-linked-mangoes-suspected",
  "summary": "In the summer of 2017 federal, state and local health officials investigated an outbreak of Salmonella Infantis. The suspected vehicle of transmission was mangoes. Forty eight outbreak associated cases were reported by 14 states. Reported and estimated illness onset dates ranged from June 30, 2017 to August 31, 2017. Among 42 cases with available information, there were 15 reported hospitalizations. No deaths were reported. \nThe cluster investigation was assigned CDC 1708MLJFX-1.",
  "beginningDate": "2017-06-01T07:00:00+00:00",
  "states": [
    "CA",
    "IL",
    "IN",
    "MI",
    "MN",
    "NJ",
    "NY",
    "NC",
    "OH",
    "OK",
    "TX",
    "WA",
    "WI"
  ],
  "country": "us",
  "hasTest": "yes",
  "testResults": "JFXX01.0010",
  "productSubjectToRecall": "yes",
  "totalIll": 48,
  "numberIllByCaseDefinitionKnown": "yes",
  "probableIll": null,
  "possibleIll": null,
  "confirmedIll": 48,
  "hospitalized": "yes",
  "numberHospitalized": 15,
  "anyDeaths": "",
  "numberOfDeaths": null,
  "recallLinks": [
    {
      "col1": "",
      "url": ""
    }
  ],
  "reportReferenceLinks": [
    {
      "col1": "",
      "url": ""
    }
  ],
  "brands": [],
  "locations": [
    {
      "title": "Retail"
    }
  ],
  "organisms": [
    {
      "title": "Salmonella"
    }
  ],
  "tags": [
    {
      "title": "salmonellosis"
    },
    {
      "title": "salmonella"
    },
    {
      "title": "mangoes"
    },
    {
      "title": "infantis"
    }
  ],
  "vehicles": [
    {
      "title": "Mangoes"
    }
  ]
}

Dynamic Parameter Queries

But what if we’re doing something more complicated, like a faceted search, where we want to narrow down the search criteria based on a series of optional search parameters from the user such as shown in this video:

Prior to Craft CMS 3.4, you couldn’t do this via the GraphQL API. But Craft CMS 3.4 added two key features to make it possible

It’s now possible to query for elements by their custom field values via GraphQL. (#5208)
It’s now possible to filter element query results by their related elements using relational fields’ element query params (e.g. publisher(100) rather than relatedTo({targetElement: 100, field: 'publisher'})). (#5200)

These are both huge in terms of making the GraphQL API flexible, so hat’s off to Andris for making it happen.

This allows us to construct a dynamic query like this:


export const outbreaksQuery = (additionalParams) => {
    let queryAdditionalParams = '';
    let entryAdditionalParams = '';
    additionalParams.forEach((item) => {
        queryAdditionalParams += `, $${item.fieldName}: [QueryArgument]`;
        entryAdditionalParams += `, ${item.fieldName}: $${item.fieldName}`;
    });
    return `
    query outbreaksQuery($needle: String!${queryAdditionalParams})
    {
        entries(section: "outbreaks", search: $needle${entryAdditionalParams}) {
            title,
            url,
            slug,
            ...on outbreaks_outbreaks_Entry {
                summary,
                beginningDate,
                vehicles {
                    title
                },
                organisms {
                    title
                },
                tags {
                    title
                }
            }
        }
    }
    `;
};

Note that this is a function, not a simple returned template literal as in the previous examples.

This allows us to dynamically add parameters to our query based on whatever has been pushed into the additionalParams array.

So the base query looks like this:


query outbreaksQuery($needle: String!)
{
    entries(section: "outbreaks", search: $needle) {
        title,
        url,
        slug,
        ...on outbreaks_outbreaks_Entry {
            summary,
            beginningDate,
            vehicles {
                title
            },
            organisms {
                title
            },
            tags {
                title
            }
        }
    }
}

…and the variables we pass in look like this:


{
    needle: "vomit"
}

If the user then also adds in an organism to search on (which in our case is a related entry in the Craft CMS backend), the query will look like this:


query outbreaksQuery($needle: String!, $organisms: [QueryArgument])
{
    entries(section: "outbreaks", search: $needle, organisms: $organisms) {
        title,
        url,
        slug,
        ...on outbreaks_outbreaks_Entry {
            summary,
            beginningDate,
            vehicles {
                title
            },
            organisms {
                title
            },
            tags {
                title
            }
        }
    }
}

…and the variables we pass in look like this:


{
    needle: "vomit",
    organisms: "4425"
}

The 4425 number is the element ID of the related organism entry in the Organisms section. It could also be an array of IDs, or anything else you’d normally pass in as a relational field’s element query params.

This technique works for an arbitrary number of additional faceted search criteria, which gives the user quite a bit of power in the search form.

Here’s the Vuex Action code that implements it:


import axios from 'axios';
import { configureGqlApi, executeGqlQuery } from '../utils/gql.js';

const GRAPHQL_ENDPOINT = '/api';

// Push additional params
const pushAdditionalParam = (state, dataFieldName, dbFieldName, additionalParams) => {
    let fieldValue = state.searchForm ? state.searchForm[dataFieldName] || '' : '';
    if (fieldValue.length) {
        // Special case for fields
        if (dbFieldName === 'states') {
            // As per https://docs.craftcms.com/v3/checkboxes-fields.html#querying-elements-with-checkboxes-fields
            fieldValue = `*"${fieldValue}"*`
        }
        additionalParams.push({
            fieldName: dbFieldName,
            fieldValue: fieldValue,
        });
    }
};

// Fetch & commit the Outbreaks array from a GraphQL query
export const getOutbreaks = async({commit, state}) => {
    const token = state.gqlToken ? state.gqlToken.token : null;
    const keywords = state.searchForm ? state.searchForm.keywords || '*' : '*';
    // Construct the additional parameters
    let additionalParams = [];
    for (const [key, value] of Object.entries(additionalParamFields)) {
        pushAdditionalParam(state, key, value, additionalParams);
    }
    // Configure out API endpoint
    const api = axios.create(configureGqlApi(GRAPHQL_ENDPOINT, token))
    // Construct the variables object
    let variables = {
        needle: keywords,
    };
    additionalParams.forEach((item) => {
        variables[item.fieldName] = item.fieldValue;
    });
    // Execute the GQL query
    await executeGqlQuery(api, outbreaksQuery(additionalParams), variables, (data) => {
        if (data.entries) {
            commit('setOutbreaks', data.entries);
        }
    });
};

This pushAdditionalParam() function just does a little magic to map from the name of a property in our Vue component to the name of the field in the database.

Then it special-cases for the states field, which is Checkboxes field in Craft CMS. To query for an item in a Checkboxes field, you need to use the format '"foo"'.

As per what we discussed previously, we figured this out by looking up how to do it in an Element Query, and then using that in our GraphQL query.

Then the getOutbreaks() function then just dynamically builds the additionalParams array and variables object, which then is passed along to our query.

The data returned from this query looks like this:


[
  {
    "title": "1997 Botulism After Consumption of Home-Pickled Eggs, Illinois",
    "url": "http://outbreakdatabase.test/outbreaks/1997-botulism-after-consumption-of-home-pickled-eggs-illinois",
    "slug": "1997-botulism-after-consumption-of-home-pickled-eggs-illinois",
    "summary": "On November 23, 1997, a previously healthy man became nauseated, vomited, and complained of abdominal pain. During the next 2 days, he developed double vision, difficult movement of joints, and respiratory impairment. He was hospitalized and placed on mechanical ventilation. His physical examination confirmed multiple cranial nerve abnormalities, including extraocular motor palsy and diffuse flaccid paralysis. Botulism was diagnosed, and antibotulinum toxin was administered. His blood serum demonstrated the presence of type B botulinum toxin. A food history revealed no exposures to home-canned products; however, the patient had eaten pickled eggs that he had prepared seven days before his symptoms began. The patient recovered after a prolonged period of supportive care. \n\nThe pickled eggs were prepared using a recipe that consisted of hard-boiled eggs, commercially prepared beets and hot peppers, and vinegar. The intact hard-boiled eggs were peeled and punctured with toothpicks then combined with the other ingredients in a glass jar that closed with a metal screw-on lid. The mixture was stored at room temperature and occasionally was exposed to sunlight. \n\nCultures revealed Clostridium botulinum type B, and type B toxin was detected in samples of the pickled egg mixture, the pickling liquid, beets, and egg yolk. The commercially sold peppers contained no detectable toxin or Clostridium botulinum bacteria. Beets from the original commercial containers were not available. The pH of the pickling liquid was 3.5 (i.e., adequate to prevent C. botulinum germination and toxin formation, however, the pH of egg yolk, although not tested for the investigation, is normally 6.8., conducive for bacteria growth and toxin production. Inadequate acidification and lack of refrigeration enhanced the risk of bacterial contamination of this home-prepared food.",
    "beginningDate": "1997-11-01T08:00:00+00:00",
    "vehicles": [
      {
        "title": "Eggs, Pickled eggs"
      }
    ],
    "organisms": [
      {
        "title": "Botulism"
      }
    ],
    "tags": [
      {
        "title": "c.+botulinum"
      },
      {
        "title": "clostridium+botulinum"
      },
      {
        "title": "c.+bot."
      }
    ]
  }
]

Tapping Out

Hopefully these real-world examples of using Craft CMS’s GraphQL API in “headless” CMS setups has been helpful to you!

Craft CMS is an excellent choice if you want to combine a great content authoring experience on the backend with a “headless” CMS that serves up data to a frontend via a GraphQL API.

If you’re a plugin or custom module developer, there are also some other nice new features in Craft CMS 3.4 for GraphQL:

Plugins can now modify the GraphQL schema via craft\gql\TypeManager::EVENT_DEFINE_GQL_TYPE_FIELDS
Plugins can now modify the GraphQL permissions via craft\services\Gql::EVENT_REGISTER_GQL_PERMISSIONS

This allows for great flexibility in terms of extending the existing Craft CMS GraphQL API.

Happy querying!

An Effective Twig Base Templating Setup

Andrew Welch — Thu, 21 Nov 2019 01:29:00 +0000

An Effective Twig Base Templating Setup

A good base templating setup for your Craft CMS Twig templates provides a stable, solid foundation on which to build your projects

Andrew Welch / nystudio107

Twig is a fantastic templating language that features multiple inheritance of layout templates, and is optimized to be an easy to use presentation layer.

This article discusses an effective Twig base templating setup that I have found to work extremely well for me in my Craft CMS websites.

However, even if you use another CMS that uses Twig like Drupal or Grav, or you use another templating language entirely like Blade or Antlers, the principles discussed here still apply.

Think of a templating language as the thin layer of frosting that covers the layer cake that is your website, and makes it presentable.

The key thing to note here is that Twig is a templating language, and as such it should not be used for complicated business or intensive calculations.

Not that it can’t handle either (it can) but rather that it shouldn’t.

If you’re unclear as to why, read about why Twig was created to begin with in the Templating Engines in PHP article.

It’s all about that base

Over the years, on a vast array of software projects of all shapes and sizes, I’ve seen developers chasing the holy grail of code reusability.

Often times it ends up being that they spend an inordinate amount of time creating “the one high level framework to rule them all”, only to be confused when reality butts in its ugly head.

Many high level frameworks are either over-engineered & unwieldy or so specific they aren’t that reusable in the end anyway.

I try to be more practical about which things I will actually re-use (and some would say less ambitious).

Websites created in Craft CMS tend to be more on the bespoke side of things, otherwise they might be better done in a more cookie-cutter system anyway.

So what I re-use are very fundamental things like the build system (discussed in the An Annotated webpack 4 Config for Frontend Web Development article), and a base templating system.

If you’re going to build anything of substance, it’s crucial that the base it’s built on is robust.

So here’s what I want out of a base templating system:

The ability to use it unmodified on a wide variety of projects
One template that can be used both as a web page, and as popup modals via AJAX / XHR
Implement core features for me, without restricting me in terms of flexibility
Allow for creating Google AMP pages, if the project warrants it

Often I see developers making templates that inherit from just one layout, or if they use multiple layouts, it’s still a single inheritance chain.

Twig allows for more than that. So let’s see how one approach to leverage this might work.

SEO & popup modals

Many of the points mentioned in the previous section are largely self-explanatory, but point 2 deserves more explanation. It’s all about templates working both as web pages and popup modals loaded in via AJAX / XHR.

I frequently work with Jonathan Melville of Code MDD on projects, and he often does designs that have content in popup modals.

For example, if you go to the Seaside Events page you’ll see a number of events listed, and if you click on an event, you’ll see the event details in a popup modal:

This is great, and gives it a nice app-ish feel, allowing the user to view multiple events without leaving the original page.

But for SEO reasons, as well as for canonical page linking reasons, the same content can also be found on its own unique page: Seaside Farmer’s Market — Saturdays in November:

This is ideally what we want to be able to do automatically: have the same core content be displayable both with and without the web page “chrome” around it.

And this is one of the things that the base templating setup does.

The overall structure

Here’s an overview of what this base templating system looks like. It may seem involved, but we’ll break it down:

The orange rounded rectangles represent templates that will be in your templates/_layouts/ directory, and may vary from project to project.

The blue rectangles represent boilerplate templates that will be in your templates/_boilerplate/_layouts/ directory, and won’t change from project to project.

You do not have to use the exact Twig base templating setup we use; but you may benefit from taking away and using the concepts presented

If at this point you’re someone who learns better by real-world examples, the exact base templating system described here is used in the MIT-licensed devMode.fm website Github repo.

Feel free to check it out; it’s also used in the nystudio107/craft boilerplate setup.

Meanwhile, everyone else, read on! We’re going to break down each template.

PROJECT: will prefix each template that may vary from project to project

BOILERPLATE: will prefix each template that stays the same from project to project

Here we go…

PROJECT: global-variables.twig

Due to Twig’s Processing Order & Scope, if we want to have global variables that are always available in all of our templates, they need to be defined in the root template that all others extends from.

Since these globals can vary from project to project, they are not part of the boilerplate, but they are required for the setup.


{# -- Root global variables that all templates inherit from -- #}
{# -- This allows for defining site-wide Twig variables as needed -- #}
{% spaceless %}

{# -- Prefetch & preconnect headers and links -- #}
{% set prefetchUrls = [
    alias("@assetsUrl"),
] %}
{# -- General global variables -- #}
{% set baseUrl = alias('@assetsUrl') ~ '/' %}
{% set gaTrackingId = getenv('GA_TRACKING_ID') %}

{# -- Twig output from the render; this must be in a block -- #}
{% block htmlPage %}
{% endblock %}

{% endspaceless %}

The global-variables.twig template has the following blocks that can be overridden by its children:

htmlPage — a block that encompasses the entire rendered HTML page

BOILERPLATE: base-web-layout.twig

Every webpage, whether a regular web page or a Google AMP page inherits from this template. The setup may look a little weird, but it’s done this way so that child templates can override bits like the opening <html> tag if they need to:


{# -- Base web layout template that all web requests inherit from -- #}
{% extends "_layouts/global-variables.twig" %}

{%- block htmlPage -%}
    {% minify %}
    <!DOCTYPE html>
        {% block htmlTag %}
            <html lang="{{ craft.app.language |slice(0,2) }}">
        {% endblock htmlTag %}
        {% block headTag %}
            <head>
        {% endblock headTag %}
            {% include "_boilerplate/_partials/head-meta.twig" %}
            {# -- Page content that should be included in the <head> -- #}
            {% block headContent %}
            {% endblock headContent %}
            </head>

            {% block bodyTag %}
            <body>
            {% endblock bodyTag %}
                {# -- Page content that should be included in the <body> -- #}
                {% block bodyContent %}
                {% endblock bodyContent %}
            </body>
        </html>
    {% endminify %}
{%- endblock htmlPage -%}

Since this is a base template that all other web pages inherit from, if we wanted to do full page caching using the Craft Cache tag, we could wrap that around the {% minify %} tags here.

The base-web-layout.twig template has the following blocks that can be overridden by its children:

htmlTag — the <html> tag, which child templates might need to override
headTag — the <head> tag, which child templates might need to override
headContent — whatever tags need to go into the <head>
bodyTag — the <body> tag, which child templates might need to override
bodyContent — whatever tags need to go in the <body>

In addition, the _boilerplate/_partials/head-meta.twig partial that contains boilerplate tags put into the <head> is included here as well.

BOILERPLATE: base-ajax-layout.twig

If the request is an AJAX / XHR request, we want to return just the page’s {% content %} block, without any of the web page “chrome” around it.

This is exactly what this template does:


{# -- Base layout template that all AJAX requests inherit from -- #}
{% extends "_layouts/global-variables.twig" %}

{% block htmlPage %}
    {% minify %}
        {# -- Primary content block -- #}
        {% block content %}
            <code>No content block defined.</code>
        {% endblock content %}
    {% endminify %}
{% endblock htmlPage %}

The base-ajax-layout.twig template has the following blocks that can be overridden by its children:

content — the core content that is represented on the page

BOILERPLATE: base-html-layout.twig

This is the base HTML layout that all HTML requests inherit from:


{# -- Base HTML layout template that all HTML requests inherit from -- #}
{% extends craft.app.request.isAjax() and not craft.app.request.getIsPreview()
    ? "_boilerplate/_layouts/base-ajax-layout.twig"
    : "_boilerplate/_layouts/base-web-layout.twig"
%}

{% block htmlTag %}
    <html class="fonts-loaded" lang="{{ craft.app.language |slice(0,2) }}" prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb#">
{% endblock htmlTag %}

{# -- Page content that should be included in the <head> -- #}
{% block headContent %}
    {# -- Any <meta> tags that should be included in the <head> #}
    {% block headMeta %}
    {% endblock headMeta %}

    {# -- Any <link> tags that should be included in the <head> #}
    {% block headLinks %}
    {% endblock headLinks %}

    {# -- Inline and polyfill JS #}
    {% include "_boilerplate/_partials/head-js.twig" %}

    {# -- Any JavaScript that should be included before </head> -- #}
    {% block headJs %}
    {% endblock headJs %}

    {# -- Inline and critical CSS #}
    <style>
        [v-cloak] {display: none !important;}
        {# -- Any CSS that should be included before </head> -- #}
        {% block headCss %}
        {% endblock headCss %}
    </style>
    {% include "_boilerplate/_partials/critical-css.twig" %}

{% endblock headContent %}

{# -- Page content that should be included in the <body> -- #}
{% block bodyContent %}
    {# -- Page content that should be included in the <body> -- #}
    {% block bodyHtml %}
    {% endblock bodyHtml %}

    {#-- Site-wide JavaScript --#}
    {{ craft.twigpack.includeSafariNomoduleFix() }}
    {{ craft.twigpack.includeJsModule("app.js", true) }}
    {{ craft.twigpack.includeJsModule("styles.js", true) }}

    {# -- Any JavaScript that should be included before </body> -- #}
    {% block bodyJs %}
    {% endblock bodyJs %}
{% endblock bodyContent %}

The base-html-layout.twig template has the following blocks that can be overridden by its children:

headMeta — Any <meta> tags that should be included in the <head>
headLinks — Any <link> tags that should be included in the <head>
headJs — Any JavaScript that should be included before </head>
headCss — Any CSS that should be included before </head>
bodyHtml — Page content that should be included in the <body>
bodyJs — Any JavaScript that should be included before </body>

In addition, the _boilerplate/_partials/head-js.twig & _boilerplate/_partials/critical-css.twig boilerplate partials are included here as well.

BOILERPLATE: amp-base-html-layout.twig

This is the base AMP HTML layout that all AMP HTML requests inherit from:


{# -- Base AMP HTML layout template that AMP web requests inherit from -- #}
{% extends craft.app.request.isAjax() and not craft.app.request.getIsPreview()
    ? "_boilerplate/_layouts/base-ajax-layout.twig"
    : "_boilerplate/_layouts/base-web-layout.twig"
%}

{% do seomatic.script.container().include(false) %}
{% do craft.webperf.includeBeacon(false) %}

{% block htmlTag %}
    <html ⚡ lang="{{ craft.app.language |slice(0,2) }}" class="fonts-loaded">
{% endblock htmlTag %}

{# -- Page content that should be included in the <head> -- #}
{% block headContent %}
    {# -- Any <meta> tags that should be included in the <head> #}
    {% block headMeta %}
    {% endblock headMeta %}

    {# -- Any <link> tags that should be included in the <head> #}
    {% block headLinks %}
    {% endblock headLinks %}

    {# -- Google AMP JavaScripts #}
    {% include "_boilerplate/_partials/amp-head-js.twig" %}

    {# -- Any JavaScript that should be included before </head> -- #}
    {% block headJs %}
    {% endblock headJs %}

    {# -- Boilerplate & custom AMP CSS #}
    {% include "_boilerplate/_partials/amp-boilerplate-css.twig" %}
    <style amp-custom>
    {# -- Any CSS that should be included before </head> -- #}
    {% block headCss %}
    {% endblock headCss %}
    </style>
{% endblock headContent %}

{# -- Page content that should be included in the <body> -- #}
{% block bodyContent %}
    {# -- Page content that should be included in the <body> -- #}
    {% block bodyHtml %}
    {% endblock bodyHtml %}

    {# -- AMP Analytics --#}
    {% include "_boilerplate/_partials/amp-analytics.twig" %}

    {# -- Any JavaScript that should be included before </body> -- #}
    {% block bodyJs %}
    {% endblock bodyJs %}
{% endblock bodyContent %}

The amp-base-html-layout.twig template has the following blocks that can be overridden by its children:

headMeta — Any <meta> tags that should be included in the <head>
headLinks — Any <link> tags that should be included in the <head>
headJs — Any JavaScript that should be included before </head>
headCss — Any CSS that should be included before </head>
bodyHtml — Page content that should be included in the <body>
bodyJs — Any JavaScript that should be included before </body>

N.B.: these blocks are all purposefully the same as the ones used in the base-html-layout.twig template.

In addition, the _boilerplate/_partials/amp-head-js.twig, _boilerplate/_partials/amp-boilerplate-css.twig & amp-analytics.twig boilerplate partials are included here as well.

PROJECT: generic-page-layout.twig

This is a generic page layout that I’ve found suits most of the projects I build, and my other templates extends it.

For similar pages, I can even extend this layout to get everything it offers, plus what I need for another subset of pages. For example, see the generic-page-layout.twig below.

However, if I have other pages that require radically different layouts, I’ll just create another layout template that extends _boilerplate/_layouts/base-html-layout.twig and away we go!


{# -- Layout template for HTML pages -- #}
{% extends "_boilerplate/_layouts/base-html-layout.twig" %}

{# -- Any <meta> tags that should be included in the <head> #}
{% block headMeta %}
{% endblock headMeta %}

{# -- Any <link> tags that should be included in the <head> #}
{% block headLinks %}
{% endblock headLinks %}

{# -- Any CSS that should be included before </head> -- #}
{% block headCss %}
    {% include "_inline-css/site-fonts.css" %}
{% endblock headCss %}

{# -- Page body -- #}
{% block bodyHtml %}
    <div id="page-container" class="overflow-hidden leading-tight">
        <confetti></confetti>
        <div id="content-container" class="bg-repeat header-background">

            {# -- Info header, including _navbar.twig -- #}
            {% include "_partials/info-header.twig" %}

            <main>
                <div class="container mx-auto pb-8">
                    {# -- Primary content block -- #}
                    {% block content %}
                    {% endblock %}
                </div>
            </main>
        </div>

        {# -- Content that appears below the primary content block -- #}
        {% block subcontent %}
        {% endblock %}

        {# -- Info footer -- #}
        {% include "_partials/info-footer.twig" %}

        {# -- HTML Footer -- #}
        {% include "_partials/global-footer.twig" %}
    </div>
{% endblock bodyHtml %}

The generic-page-layout.twig template has the following blocks that can be overridden by its children:

content — Primary content block
subContent — Content that appears below the primary content block

Markup in the subContent block will appear on web pages, but not on pages loaded in via AJAX / XHR.

In addition, it includes a few partials for the header, footer, etc., but you can have it do whatever makes the most sense to you.

PROJECT: error-page-layout.twig

Here we further extends the generic-page-layout.twig with another layout template that’s specifically intended for error pages.

Because we have a number of different error pages that display different content, but have the same basic layout, this is the perfect opportunity to consolidate them in another layout template.

Instead of replicating the content for each error page, we can have the error pages extends error-page-layout.twig and have very lightweight error pages.

The same idea of an inheritance chain can be used in similar situations.


{# -- Layout template for error pages -- #}
{% extends "_layouts/generic-page-layout.twig" %}

{% block content %}
{% endblock %}

{% block subcontent %}
    <section>
        <div class="container mx-auto py-8">
            <div class="text-center p-8 mb-8">
                <h1 class="font-mono italic font-bold text-5xl pt-4">
                    {{ entry.errorHeadline ?? 'Error' }}
                </h1>
                <p class="font-sans text-xl pt-4">
                    {{ (entry.errorText ?? 'An error has occurred.') |nl2br }}
                </p>
            </div>
        </div>
    </section>

{% endblock %}

{# -- Any JavaScript that should be included before </body> -- #}
{% block bodyJs %}
{% endblock bodyJs %}

The error-page-layout.twig template has the following blocks that can be overridden by its children:

content — Primary content block
subContent — Content that appears below the primary content block

Markup in the subContent block will appear on web pages, but not on pages loaded in via AJAX / XHR.

PROJECT: amp-generic-page-layout.twig

This is the Google AMP generic page template, which mirrors the blocks and methodology from the generic-page-layout.twig template, but is separated out to allow for the unique tags that Google AMP requires:


{# -- Layout template for AMP HTML pages -- #}
{% extends "_boilerplate/_layouts/amp-base-html-layout.twig" %}

{# -- Any <meta> tags that should be included in the <head> #}
{% block headMeta %}
{% endblock headMeta %}

{# -- Any <link> tags that should be included in the <head> #}
{% block headLinks %}
{% endblock headLinks %}

{# -- Any JavaScript that should be included before </head> -- #}
{% block headJs %}
{% endblock headJs %}

{# -- Any CSS that should be included before </head> -- #}
{% block headCss %}
    {% include "_partials/amp-inline-css.css" %}
    {% include "_inline-css/site-fonts.css" %}
{% endblock %}

{# -- Page body -- #}
{% block bodyHtml %}
    {% include "_partials/amp-navbar.twig" %}
    <div id="page-container" class="overflow-hidden leading-tight">
        <div id="content-container" class="bg-repeat header-background">
            {# -- Info header, including _navbar.twig -- #}
            {% include "_partials/amp-info-header.twig" %}

            <main>
                <div class="container mx-auto pb-8">
                    {# -- Primary content block -- #}
                    {% block content %}
                    {% endblock %}
                </div>
            </main>
        </div>

        {# -- Content that appears below the primary content block -- #}
        {% block subcontent %}
        {% endblock %}

        {# -- Info footer -- #}
        {% include "_partials/amp-info-footer.twig" %}

        {# -- HTML Footer -- #}
        {% include "_partials/global-footer.twig" %}
    </div>
{% endblock bodyHtml %}

The amp-generic-page-layout.twig template has the following blocks that can be overridden by its children:

content — Primary content block
subContent — Content that appears below the primary content block

Markup in the subContent block will appear on web pages, but not on pages loaded in via AJAX / XHR.

devMode.fm page: a real world example

So how does this all look with a real world example? Why, I’m glad you asked! Let’s have a look at the devMode.fm home page, which extends generic-page-layout.twig:


{% extends "_layouts/generic-page-layout.twig" %}

{% set includeAudioMeta = false %}

{% block headLinks %}
    {{ parent() }}
    <link rel="amphtml" href="{{ siteUrl('/amp') }}">
{% endblock headLinks %}

{% block content %}
    {% include "_partials/_meta-schema-radio-series.twig" with {
        "showInfo": showInfo,
    } only %}
    <section>
        <div>
            {% for episode in craft.entries.section("episodes").limit(1).all() %}
                <div class="flex flex-wrap">
                    {% include "episodes/_partials/_display_episode.twig" with {
                        "episode": episode,
                        "showInfo": showInfo,
                        "includeAudioMeta": includeAudioMeta,
                        "autoPlay": false,
                    } only %}
                </div>
            {% endfor %}
        </div>
    </section>
{% endblock %}

{% block subcontent %}
    {% include "episodes/_partials/_display_recent_episodes.twig" with {
        "showInfo": showInfo,
    } only %}
{% endblock %}

{# -- Any JavaScript that should be included before </body> -- #}
{% block bodyJs %}
    {{ craft.twigpack.includeJsModule("player.js", true) }}
    {{ craft.twigpack.includeJsModule("episodes.js", true) }}
{% endblock bodyJs %}

You can compare this to the devMode.fm rendered home page.

The index.twig template overrides just four blocks:

headLinks — Here we add a link to point browsers at the Google AMP version of this page
content — The content of this page, in this case the current episode summary & audio player
subContent — The episodes listing component, displayed under the content
bodyJs — Adds some JavaScript to handle the player & episodes listing to the page, courtesy of Twigpack

You can see that this makes the actual templates that we write pretty clean. And if this page was ever requested via AJAX / XHR, it’d return just the content block.

The Google AMP version of the homepage template is very similar:


{% extends "_layouts/amp-generic-page-layout.twig" %}

{% if entry is not defined %}
    {% set entry = craft.entries({
        "uri": " __home__",
    }).one() %}
{% endif %}

{% do seomatic.helper.loadMetadataForUri(entry.uri) %}
{% do seomatic.script.container().include(false) %}

{% block headCss %}
    {{ parent() }}
    {{ craft.twigpack.includeFile("@webroot/dist/criticalcss/amp_index_critical.min.css") }}
{% endblock headCss %}

{% block content %}
    <section>
        <div>
            {% for episode in craft.entries.section("episodes").limit(1).all() %}
                <div class="flex flex-wrap">
                    {% include "episodes/_partials/_amp_display_episode.twig" with {
                        "episode": episode,
                        "showInfo": showInfo,
                    } only %}
                </div>
            {% endfor %}
        </div>
    </section>
{% endblock %}

{% block subcontent %}
    {% include "episodes/_partials/_amp_display_recent_episodes.twig" with {
        "showInfo": showInfo,
    } only %}
{% endblock %}

{% block bodyJs %}
{% endblock bodyJs %}

It’s explicitly loading the appropriate entry, because it won’t be auto-injected for us by Craft, and then it loads the appropriate metadata for the route via seomatic.helper.loadMetadataForUri() and excludes all scripts via seomatic.script.container().include(false) because Google AMP doesn’t allow for them.

It’s also using Twigpack to include the full CSS for the page inline (as per Google AMP spec) but other than that… it’s the same as the regular web page example.

All about that Bass

While you certainly could just start using my boilerplate, odds are good you’ll want to customize some things to suit your tastes.

That’s totally fine. What’s important is the structure and methodology, not the specific implementation details.

The point of a modularized system like this is that if you wanted to add, say, a way to output the same content in JSON format, you could. Just slap in another layout in the right place, and away you go.

Enjoy the obligatory “All About That Bass” and have an excellent day!

Links:

Using Tailwind CSS with Gatsby, React & Emotion Styled Components

Andrew Welch — Sun, 20 Oct 2019 14:08:00 +0000

Using Tailwind CSS with Gatsby, React & Emotion Styled Components

Learn how to use the utility-first Tailwind CSS with Emotion “CSS-in-JS” Styled Components in a Gatsby JS + React project.

Andrew Welch / nystudio107

Tailwind CSS is a utility-first CSS framework that allows for rapidly building custom designs, and it is something I’ve adopted as a standard part of my workflow.

If you want to hear the why’s and how’s of Tailwind CSS, check out the Tailwind CSS Utility-First CSS devMode.fm podcast episode. For the purposes of this article, we’ll just assume you’re all on-board the Tailwind Express like I am.

Since I’m a fan of Tailwind CSS, when I started working with Gatsby & React, I wanted a way to bring Tailwind with me. I ended up deciding to go with the Emotion CSS-in-JS approach, specifically using Styled Components.

Again, we’ll just assume you’re on-board with Emotion CSS-in-JS; if not, check out the CSS in JS, an Emotional Topic devMode.fm podcast episode.

This article discusses how to make all of these technologies play nice together, and explores some of the fun things the resulting stack enables you to do.

If you want a head start on this setup, check out Paulo Elias’s gatsby-tailwind-emotion-starter to get you going. Otherwise, buckle up and let’s dive right in!

Why are we doing this?

If you haven’t worked with CSS-in-JS or Styled Components before, there are some nice advantages:

You can use full-blown JavaScript
CSS is scoped to just the component, and doesn’t “bleed out”
You automatically get just the CSS used on each page
You automatically get Critical CSS
The end result is just CSS

You may or may not be convinced that CSS-in-JS is a good idea, but these are tangible benefits that I’ve found, and thus my desire to use CSS-in-JS and Styled Components.

Adding Tailwind CSS to the mix brings all of the wonderful benefits of a utilty-first CSS framework like Tailwind CSS to our Styled Components.

So with that said, let’s get to it!

A Hybrid Approach

The excellent Gatsby documentation has two approaches listed for using Tailwind CSS with Gatsby:

We’re actually going to use a hybrid approach, using both the tailwind.macro Babel Macro and PostCSS.

We’re going to use the tailwind.macro to translate Tailwind CSS classes to Emotion Styled Components for us, generating just the CSS selectors we actually use.

Then we’ll use PostCSS for building the Tailwind CSS base styles (and any base global styles we want to use) that will be applied globally to every page. This gives us things like the Normalize CSS reset as a base.

This is the reason for the hybrid approach: if we just used the tailwind.macro, we wouldn’t get any of the Tailwind CSS base styles.

You could also use the gatsby-theme-tailwindcss Gatsby Theme as a way to scaffold things, but I think it makes sense to understand how all of the pieces fit together first.

So let’s get going by installing Tailwind CSS:


# Using npm
npm install --save tailwindcss

# Using Yarn
yarn add tailwindcss

Next up, let’s get tailwind.macro set up!

Setting up tailwind.macro

The tailwind.macro is a Babel Macro that allows you to use Tailwind CSS with any CSS-in-JS library. We’ve chosen to use Emotion, so let’s get it all installed:


# Using npm
npm install --save @emotion/core @emotion/styled gatsby-plugin-emotion tailwind.macro@next

# Using Yarn
yarn add @emotion/core @emotion/styled gatsby-plugin-emotion tailwind.macro@next

Then we need to add the gatsby-plugin-emotion to our gatsby-config.js (in the project root):


module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-emotion`,
      options: {
        // Accepts all options defined by `babel-plugin-emotion` plugin.
      },
    },
  ],
}

Then we need to create a babel-plugin-macros-config.js to tell it that we want to use Emotion, and where our tailwind.config.js file lives:


module.exports = {
    tailwind: {
        styled: '@emotion/styled',
        config: './tailwind.config.js',
        format: 'auto',
    }
};

Finally, there’s a small issue we need to work around, which appears to be due to Tailwind CSS’s use of reduce-css-calc starting with Tailwind ^1.1.0, which apparently depends on it being run via Node.

We can work around this by adding the following to our gatsby-node.js file (in the project root):


exports.onCreateWebpackConfig = ({actions, getConfig}) => {
    // Hack due to Tailwind ^1.1.0 using `reduce-css-calc` which assumes node
    // https://github.com/bradlc/babel-plugin-tailwind-components/issues/39#issuecomment-526892633
    const config = getConfig();
    config.node = {
        fs: 'empty'
    };
};

Using tailwind.macro

Now that we’ve got tailwind.macro installed, let’s have a look at how we can use it. It works very similar to Emotion Styled Components:


import React from 'react';
import tw from 'tailwind.macro';

const PageContainer = tw.div`
    bg-gray-200 text-xl w-1/2
`;

const Layout = ({children}) => (
    <PageContainer>
        {children}
    </PageContainer>
);

export default Layout;

This will create a Styled Component that is composed of the styles from the Tailwind CSS classes listed in the template literal. There’s a Github issue that shows a good example of how this works.

In development :


import tw from 'tailwind.macro'
let styles = tw`w-1/2`

// ↓↓↓↓↓↓↓↓

import _tailwind from './path/to/your/tailwind.js'
let styles = {
  width: _tailwind.widths['1/2']
}

In production (NODE_ENV=production):


import tw from 'tailwind.macro'
let styles = tw`w-1/2`

// ↓↓↓↓↓↓↓↓

let styles = {
  width: '50%'
}

You can see how it directly uses the JavaScript Tailwind CSS config to extract styles. The only real downside to this approach is that it will not work with Tailwind Plugins.

Building on the initial example above, we can mix and match Emotion Styled Components custom CSS with Tailwind classes easily:


import React from 'react';
import styled from '@emotion/styled';
import tw from 'tailwind.macro';

import background from '../../static/fabric_plaid@2x.png';

const PageContainer = styled.div`
    ${tw`
        bg-gray-200 text-xl w-1/2
    `}
    background-image: url(${background});
    padding: 10px;
`;

const Layout = ({children}) => (
    <PageContainer>
        {children}
    </PageContainer>
);

export default Layout;

So it actually combines the CSS styles from the Tailwind CSS classes in the ${tw` template literal with the custom styled component CSS below it. So you can do any of the fun patterns you do with Emotion Styled Components, too!

In either case, only the CSS that is actually used on a page will be extracted (no need for PurgeCSS), and the styles used by components on a page will be inlined, appearing something like this:

.css-14xcvvf-PageContainer {
    background-color:#edf2f7;
    font-size:1.25rem;
    width:50%;
    background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAGQBAMAAABykSv/AAAAG1BMVEXq6urr6+vp6eno6Ojn5+fs7Ozt7e3m5ubu7u7dMibkAAAf6ElEQVR4AdyX4W3jSgyEp4WvhWmBLaiFaWFbeC2k7AfNSoICHHC4vyISO8vlfOQAdGxL2PaMx);
    padding:10px;
}

The hashed CSS class name ensures that the styles are scoped to just the component they are applied to, and will not leak out and affect anything else.

Sweet!

Setting up PostCSS

If we just left things as-is, using only the tailwind.macro, everything would still work, but we wouldn’t get the Tailwind CSS base styles to do things like apply a Normalize CSS style reset, and other global styles.

Because these base styles are super useful, we’ll configure PostCSS to generate them for us. So let’s install the PostCSS packages we’re going to use:

Using npm

npm install --save gatsby-plugin-postcss postcss-import postcss-preset-env stylelint

Using Yarn

yarn add gatsby-plugin-postcss postcss-import postcss-preset-env stylelint

Then we need to add the gatsby-plugin-postcss to our gatsby-config.js (in the project root):

module.exports = {
plugins: [
{
resolve: gatsby-plugin-emotion,
options: {
// Accepts all options defined by babel-plugin-emotion plugin.
},
},
{
resolve: gatsby-plugin-postcss,
options: {
// Accepts all options defined by gatsby-plugin-postcss plugin.
},
},
],
}

We’ll also need to create a postcss.config.js file (in the project root) to tell it what PostCSS plugins we want to use (including tailwindcss):

module.exports = {
plugins: [
require('postcss-import')({
plugins: [
require('stylelint')
]
}),
require('tailwindcss')('./tailwind.config.js'),
require('postcss-preset-env')({
autoprefixer: { grid: true },
features: {
'nesting-rules': true
},
browsers: [
'> 1%',
'last 2 versions',
'Firefox ESR',
]
})
]
};

The postcss.config.js file is used by PostCSS to configure the plugins and settings that PostCSS will use. The important bit here is that we’re doing a require('tailwindcss') to include Tailwind CSS.

The other PostCSS plugins we’re including here are just things that I find useful. You can read more about them in-depth in the An Annotated webpack 4 Config for Frontend Web Development article.

Then we need to create a file for the Tailwind CSS base styles, as well as any global styles we might want to add in src/utils/globals.css:

@tailwind base;

// Add any global styles here

Finally, we need to load these styles in the gatsby-browser.js (in the project root):

import "./src/utils/globals.css"

That’s it! When we do a build, it’ll now generate the Tailwind CSS base styles, and any of our global styles, and include them on the page. Here’s a truncated version of what that looks like:

/*! normalize.css v8.0.1 | MIT License | github.com/necolas/normalize.css */

html {
line-height: 1.15;
-webkit-text-size-adjust: 100%
}

body {
margin: 0
}

main {
display: block
}

/* Truncated here */

Wrapping Up

That’s all she wrote! I’ve found that being able to intertwine the familiar Tailwind CSS styles that I’m used to with Emotion Styled Components had created a really compelling way to work with CSS.

In the process of actually using CSS-in-JS and Styled Components, I found that many of my uncertain feelings about it disappeared. It turned my frown upside down.

Part of the reason is that all of this is just a really sophisticated way to generate and scope CSS in a way that allows for an excellent developer experience.

And since we’re using Gatsby to render everything out to static pages, it results in an excellent user experience too.

The fact that it ends up scoping the CSS to each component, and automatically inlining just the CSS that we use on each page is pretty fantastic.

It makes processes like PurgeCSS and Critical CSS as described in the Implementing Critical CSS on your website article unnecessary.

Give it a whirl, and you might just second that emotion.

DEV Community: Andrew Welch

Speeding Up Tailwind CSS Builds

Speeding Up Tailwind CSS Builds

Learn how to opti­mize your Tail­wind CSS PostC­SS build times to make local devel­op­ment with Hot Mod­ule Replace­ment or Live Reload orders of mag­ni­tude faster!

Fram­ing the Problem

The Prob­lem Setup

The Prob­lem

The Solu­tion

The Solu­tion Setup

Bench­mark­ing Prob­lem vs. Solution

Further Reading

Running Node.js in Docker for local development

Running Node.js in Docker for local development

You don’t need to know Dock­er to ben­e­fit from run­ning local dev Node.js build­chains & apps inside of Dock­er con­tain­ers. You get easy onboard­ing, and less hassle.

Why in the world would I use Docker?

Dock­er set­up overview

Dock­er set­up detail

Tak­ing Dock­er for a spin

Con­tainer­iza­tion as a way forward

Further Reading

Atomic Deployments Without Tears

Atomic Deployments Without Tears

Learn how to use atom­ic deploy­ments to auto­mat­i­cal­ly, deter­min­is­ti­cal­ly, and safe­ly deploy changes to your web­site using Con­tin­u­ous Inte­gra­tion (CI) tools

Anato­my of a web project

Non-Atom­ic Deploy­ment Flow

Atom­ic Deploy­ment Flow

Atom­ic Deploy­ments Under the Hood

Step 1: Cre­at­ing a new project

Step 2: Set­ting Variables

Step 3: Exe­cute: web­pack build

Step 4: Exe­cute: com­pos­er install

Step 3: Rsync files to production

Step 4: Atom­ic Deploy

Step 5: Prep Craft CMS

Step #6: Send noti­fi­ca­tion to nystudio107 channel

The Gold­en Road (to unlim­it­ed deployment)

One more deploy for the road

Further Reading

A taste of Vue.js 3: API Changes, Async Components, and Plugins

A taste of Vue.js 3: API Changes, Async Components, and Plugins

This arti­cle takes you through changes that have to be made when mov­ing to Vue.js 3, cov­er­ing API changes, async com­po­nents, and adapt­ing exist­ing plugins

Overview of the Changes

Package.json Changes

web­pack con­fig changes

app.js Changes

Confetti.vue changes

Should you use Vue.js 3 now?

Wrap­ping up

Further Reading

Extending Craft CMS with Validation Rules and Behaviors

Extending Craft CMS with Validation Rules and Behaviors

Craft CMS is a web appli­ca­tion that is amaz­ing­ly flex­i­ble & cus­tomiz­able using the built-in func­tion­al­i­ty that the plat­form offers. Use the platform!

Mod­els & Rules

Mod­els & Behaviors

Wrap­ping Up

Further Reading

An Annotated Docker Config for Frontend Web Development

An Annotated Docker Config for Frontend Web Development

A local devel­op­ment envi­ron­ment with Dock­er allows you to shrink-wrap the devops your project needs as con­fig, mak­ing onboard­ing frictionless

Why Dock­er?

Under­stand­ing Docker

My Dock­er Direc­to­ry Structure

The docker-compose.yaml file

Ser­vice: Nginx

Ser­vice: MariaDB

Ser­vice: Postgres

Ser­vice: Redis

Ser­vice: PHP

Ser­vice: webpack

All Aboard!

Further Reading

Post-Mortem: Outbreak Database

Post-Mortem: Outbreak Database

Mod­ern­iz­ing an aging cus­tom PHP web­site with Craft CMS for con­tent man­age­ment, and a hybrid Twig/Vue.js + Vuex + Axios + GraphQL on the frontend

The Ini­tial Handoff

GraphQL as an API

Adopt­ing Vue + Vuex + Axios

Doing Bet­ter

The CSS Problem

Hybrid Web­site

Learn how to optimize your Tailwind CSS PostCSS build times to make local development with Hot Module Replacement or Live Reload orders of magnitude faster!

Framing the Problem

The Problem Setup

The Problem

The Solution

The Solution Setup

Benchmarking Problem vs. Solution

You don’t need to know Docker to benefit from running local dev Node.js buildchains & apps inside of Docker containers. You get easy onboarding, and less hassle.

Docker setup overview

Docker setup detail

Taking Docker for a spin

Containerization as a way forward

Learn how to use atomic deployments to automatically, deterministically, and safely deploy changes to your website using Continuous Integration (CI) tools

Anatomy of a web project

Non-Atomic Deployment Flow

Atomic Deployment Flow

Atomic Deployments Under the Hood

Step 1: Creating a new project

Step 2: Setting Variables

Step 3: Execute: webpack build

Step 4: Execute: composer install

Step 4: Atomic Deploy

Step #6: Send notification to nystudio107 channel

The Golden Road (to unlimited deployment)

This article takes you through changes that have to be made when moving to Vue.js 3, covering API changes, async components, and adapting existing plugins

webpack config changes

Wrapping up

Craft CMS is a web application that is amazingly flexible & customizable using the built-in functionality that the platform offers. Use the platform!

Models & Rules

Models & Behaviors

Wrapping Up

A local development environment with Docker allows you to shrink-wrap the devops your project needs as config, making onboarding frictionless

Why Docker?

Understanding Docker

My Docker Directory Structure

Service: Nginx

Service: MariaDB

Service: Postgres

Service: Redis

Service: PHP

Service: webpack

Modernizing an aging custom PHP website with Craft CMS for content management, and a hybrid Twig/Vue.js + Vuex + Axios + GraphQL on the frontend

The Initial Handoff

Adopting Vue + Vuex + Axios

Doing Better

Hybrid Website

Multi-environment configs for Craft CMS are a mix of aliases, environment variables, and config files. This article sorts it all out, and presents a flat config file approach

Environment Variables in Craft CMS

Aliases in Craft CMS

Config files in Craft CMS

Multi-site Multi-Environment in Craft CMS

Winding Down

Using a cloud storage system like AWS S3 with a CDN distribution can be a convenient and inexpensive way to store your assets. Here’s how to set it up right.

Setting up S3

Step 2: Create an S3 Bucket

Step 3: Create a Custom Policy

Step 4: Create a Group

Step 5: Create a User

Step 6: Create a CloudFront Distribution

Step 7: Configure your Asset Volumes in Craft CMS

Craft CMS 3.3 added a GraphQL layer that gives your website a formalized, structured API out of the box. Here’s how to use GraphQL + Craft CMS as a “headless” CMS

Controller of Fury

Controller & GraphQL XHRs

A Simple Query

Querying a Single Thing

Dynamic Parameter Queries

Tapping Out

A good base templating setup for your Craft CMS Twig templates provides a stable, solid foundation on which to build your projects

SEO & popup modals

The overall structure

BOILERPLATE: base-web-layout.twig

BOILERPLATE: base-ajax-layout.twig

BOILERPLATE: base-html-layout.twig

BOILERPLATE: amp-base-html-layout.twig

devMode.fm page: a real world example

Learn how to use the utility-first Tailwind CSS with Emotion “CSS-in-JS” Styled Components in a Gatsby JS + React project.

Setting up tailwind.macro

Setting up PostCSS

Wrapping Up