DEV Community: mattbloomfield

CraftCMS & Docksal: A Local Dev Match Made in Heaven

mattbloomfield — Fri, 02 Apr 2021 15:25:02 +0000

If you've ever joined a new team you know the pain it can be to get a project successfully running locally on your machine. Without Docker there will be an untold number of non-documented configurations and it will inevitably take days to complete your setup. With docker, future updates can be complex and difficult with a myriad of options you don't need.

Having recently found myself in this conundrum I decided to find ways to make local development easy to setup and easy to maintain. After a search across various solutions, Docksal was the clear winner.

If you're not familiar with it, Docksal allows you to simply and easily setup a dockerized, orchestrated environment without ever having to deal with docker compose. In addition, their CLI container comes pre-installed with many of the tools you will need to run common CMSs on popular hosting platforms.

In my case, I am using platform.sh for hosting. Platform.sh is a non-opinionated linux hosting solution that essentially gives you yaml-based configuration to spin up new environments and works seamlessly with git to spin up new environments for PRs and branches. Docksal has platform, Platform.sh's CLI, pre-installed so it works great out of the box.

Docksal also has a CLI, fin, which is easily customizable to suit your needs.

Basic Setup

Our setup uses MySQL 5.7, Redis, & PHP 7.4.

To get started using docksal create a .docksal directory at the root of your CraftCMS repo. Within it you'll need a docksal.yml and docksal.env.

Docksal comes standard with three containers: A database, a web server, and a CLI. You can specify which versions of PHP, MySQL, etc. within your docksal.env. To add new services, use your docksal.yml.

Here is a simple docksal.env. Any variable created here will be accessible within your cli container and within any custom commands you write.

# .docksal/docksal.env

# Run `fin restart` anytime you change these values

PROJECT="project_name"

# Docksal Defaults
DOCKSAL_STACK=default
DOCROOT="web"
DB_IMAGE="docksal/mysql:5.7"
CLI_IMAGE="docksal/cli:php7.4"
COMPOSER_DEFAULT_VERSION=2

# Platform.sh
HOSTING_PLATFORM=platformsh
HOSTING_SITE=xxxxxxxxxx
HOSTING_ENV=production

# MySQL
MYSQL_DATABASE='db_name'
MYSQL_PORT_MAPPING='33061:3306'
MYSQL_ROOT_PASSWORD="root"

And here is a simple docksal.yml that brings in redis as a service.

# .docksal/docksal.yml

version: "2.1"

services:
  redis:
    hostname: redis
    image: wodby/redis:4.0
    environment:
      - REDIS_MAXMEMORY=256m

In my case we use S3 for asset storage, so I'm leveraging the AWS CLI in my cli container. To pass the environment variables into the cli container I add this to my docksal.yml under services

services:
  ...
  cli:
    environment:
      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY

You probably also have secrets you don't want committed to your repository because they are either sensitive or personalized. For that you can use a docksal-local.env. Add this to your .gitignore and use it just as you use the previous file.

# .docksal/docksal-local.env

# Run `fin restart` anytime you change these values
SECRET_PLATFORMSH_CLI_TOKEN=""
DEV_USER=""
INTERNAL_API_KEY=""
DB_PW_STAGING=""
DB_PW_PROD=""
AWS_ACCESS_KEY_ID="" # note how these are used in the yml above
AWS_SECRET_ACCESS_KEY="" # note how these are used in the yml above

Once you have your configuration intact, go ahead and run fin project start. fin is the Docksal CLI and is quite powerful.

If you are going my route of using AWS for asset storage, you'll also want to add a .docksal/services/cli/startup.sh script to install the AWS CLI.

# .docksal/services/cli/startup.sh

sudo apt-get update
sudo apt-get install awscli -y
sudo apt-get install groff -y
aws configure set region us-east-1 --profile default

Anytime you make changes to your configuration files you can run fin restart to restart your containers. If you make changes to the images referenced you may need to run fin project reset <container> e.g. fin project reset cli.

Custom Commands

This is all fine and dandy, but the real magic comes in the form of custom commands. We use these ALL THE TIME.

Go ahead and create a commands directory under .docksal. Any file you place in this directory will be a new command that fin can run.

You can pretty much use any language to write these commands, but I have stuck with bash for now.

One example could be to have an init command that gets docksal running, then installs composer and npm dependencies within the CLI container. Note that these commands will not be dependent on your local npm/composer installs but on what exists in the cli container.

# .docksal/commands/init

#!/bin/sh

BLUE=\\x1B[34m
YELLOW=\\x1B[33m
GREEN=\\x1B[32m
RED=\\x1B[31m
RESET=\\x1b[0m

## Removes any previous containers this preject may have created, then starts and executes npm install/composer install
## Usage: fin init

# Forces whole job to error on failure
set -e

echo "${YELLOW}REMOVING ANY OLD CONTAINERS${RESET}"
fin project rm -f

echo "${YELLOW}STARTING PROJECT CONTAINERS${RESET}"
fin project start

echo "${YELLOW}RUNNING NPM INSTALL${RESET}"
fin exec npm install

echo "${YELLOW}RUNNING COMPOSER INSTALL${RESET}"
fin exec composer install

The most useful of these commands is probably the setup command which allows new devs to get up and running within minutes. It leverages a lot of the jobs below.

Other custom commands I have written and use often:

We use a lambda function to store our local .env file for CraftCMS since it can securely hold secrets. We then request it using a custom command and secure key - basically minimizing the number of tokens we are passing around in Slack/Email. This custom command is found here:

Refresh Local .env

And the node.js-powered Lambda Function is here.

Accessing the CLI container

Sometimes you need to ssh into the CLI container to run certain things, e.g. craft commands like php craft db/migrate

This can be done using the fin bash command.

Platform.sh Token confusion.

It's always best to run platform commands within the CLI container. This is done by prepending the command with fin. This is because the CLI container comes pre-installed with the platform CLI. If it prompts you to log in, generate an API token and store it in your .docksal/docksal-local.env

And there you have it! You're all ready to start developing on CraftCMS using Docksal to supercharge your local setup.

If you like this approach let me know in the comments! If you've found something better, let me know that as well.

Automating CraftCMS Database Backups on Platform.sh

mattbloomfield — Fri, 02 Apr 2021 13:10:05 +0000

Congrats, you just released your very first CraftCMS site. Great! You're done!

Wait... you forgot to setup backups! Of course, some hosting providers take their own database snapshots; but often they are inaccessible except in case of catastrophic issues. What if you just want to jump back a day or week and know what's going on.

Well fortunately for us, it's not difficult to setup database backups on our own.

To setup backups there are three things you'll need: 1) Some type of CRON system; 2) A computing server that will run shell scripts; 3) A storage location.

In my case I'm going to use Rundeck for CRON, a standard Linux server for compute, and Amazon S3 for storage. You can adapt from there, however.

Storage Location

We're going to start at the end and work backwards here. If you're using S3, jump into the AWS Console and create a new bucket for CraftCMS backups.

Be sure to keep it as a private bucket or else anyone could access your database backups of potentially private information.

You may also want to enable Glacier as well to limit your costs. I typically push backups to glacier after a week.

Compute Server

Now let's write your script! I've posted mine here for you to copy if you'd like. The main point here is to SSH into the location and export the database, save it locally, then push it to S3.

If you're using my script - you'll need to update some variables to work for you.

This script leverages the AWS CLI, which greatly simplifies this process. You could do this via API as well, but if you can use the CLI I highly recommend it. You could also store your backups on a mounted network drive wherever your compute server lives as well.

The SSH User & Host will be easy to find in the Platform.sh GUI. Once you SSH in you can echo $PLATFORM_RELATIONSHIPS | base64 --decode | json_pp to see your database credentials.

CRON Setup

Finally, let's trigger the backups at the frequency you choose. I would recommend once a day as a default, increasing or decreasing depending on publication frequency.

In my case I used Rundeck because I already use it for heaps of other things. However, if you don't have Rundeck installed you could just use the crontab. Either way, just set it up to hit your backup shell script:

## In crontab, each night at midnight: 
0 0 * * * sh /var/scripts/craft_backups/my_script.sh

And that's it! You've got stable backups stored in S3 for easy access anytime you need.

Automate Pull Request Testing using Cypress Dashboard, Github Actions, and Platform.sh

mattbloomfield — Fri, 26 Mar 2021 14:31:42 +0000

I work on a small team and we manage dozens of websites. With a host of requests coming in from Product Managers it's important that we work quickly without introducing new bugs into the code.

Of course, if you have a QA team you can rely on them to double-check issues, but we don't have that luxury. And even if you do, no QA engineer has ever complained about a layer of automated tests.

Enter Cypress Testing

If you're not familiar with Cypress, it's one of the newest and most popular testing frameworks to hit the scene recently. It's a lot like using selenium, with, so far, none of the annoyances. It even lets you record your own tests by clicking around your app and then later adding assertions! It literally writes the code!

You can run it locally while developing - which is great! - but I wanted to be able to ensure that before each PR is merged we could see test results to ensure it wouldn't bork our main development branch.

This guide will not walk you through how to setup Cypress, but check out the Getting Started pages.

You'll also need to create a Cypress Dashboard account for this to work, since we'll be using the Record Key provided by that service.

Setup

It's worth noting that I'm working with CraftCMS sites that use npm/node for preprocessing CSS/JS. So in addition to NPM dependencies I've also got Composer and all its PHP dependencies.

I'm also running everything in Docker containers locally, which are basically mirrored by Platform.sh's deployment containers.

We are using Platform.sh for hosting, which is a great, docker-based solution for hosting fullstack applications where you need a database, runtime environment, and a server.

Triggers, Environments, and Integrations

There are probably lots of posts about setting up Cypress. But it took a lot of work and a lot of coordination with the Cypress DevRel team (who are fantastic) and Platform.sh support to be able to get this whole thing hooked up. Hopefully this post helps some other poor sap tasked with setting this up. The process is fairly specific to Platform.sh for hosting, but other hosting providers may provide similar tools.

Essentially we have three separate systems that all need to communicate with each other. The PR is created by a developer in Github. Upon creation, we need to provision a new docker environment in Platform.sh with a dedicated URL. This takes a few minutes to build and propagate. Once it's ready, Cypress tests need to be triggered, but they also need the URL of the Platform.sh environment. Cypress also needs a runtime environment with Node.js installed to run the tests. And once those are complete, it needs to talk back to the Github PR.

So how do we make this work?

Platform.sh Integrations

The first step is pretty native to Platform.sh, but your hosting provider probably offers this as well. Using the Platfrom CLI you can create an integration that triggers a new environment on creation of every Pull Request in Github.

Platform.sh has great docs on this integration, so I'll just say this is the basic command:

platform integration:add --type=github --project=PLATFORMSH_PROJECT_ID --token=GITHUB-USER-TOKEN --repository=USER/REPOSITORY

And this is what my settings look like (found by running platform integration:get <integration_id>):

+---------------------------------+-------------------------------------------------------------------------------------+
| Property                        | Value                                                                               |
+---------------------------------+-------------------------------------------------------------------------------------+
| id                              | xxxxxxxx                                                                       |
| type                            | github                                                                              |
| base_url                        |                                                                                     |
| repository                      | <GH_Org_Name/GH_Repo_Name>                                                             |
| fetch_branches                  | true                                                                                |
| prune_branches                  | true                                                                                |
| build_pull_requests             | true                                                                                |
| build_draft_pull_requests       | false                                                                               |
| build_pull_requests_post_merge  | true                                                                                |
| pull_requests_clone_parent_data | true                                                                                |
| hook_url                        | <Platform_Generated_URL> |
+---------------------------------+-------------------------------------------------------------------------------------+

Once you have that built out - you're halfway there! And it's pretty easy...

Github Actions (and API)

Now that we have an environment created, let's create a Github Action to spin up a new linux environment with Node.js installed and run our Cypress tests from there.

I had never used Github Actions before so this took a bit work. And truthfully, there are a lot of Cypress examples out there that all point you in different directions and don't seem up to date with each other. So I'll provide exactly what worked for me.

In your repository, create a new directory called .github (if you don't already have one) and then within that directory another called workflows, and within that a file named cypress.yml. This file is where we will program our Action. Go ahead and paste in the following, being sure to modify your inputs.

This job will create a new environment using ubuntu 16.04, then npm install your dependencies, then cypress run all your tests.

# .github/workflows/cypress.yml 

name: Manually Triggered Cypress Tests with installation job

on:
  workflow_dispatch:
    inputs:
      baseUrl:
        description: 'Testing Domain'
        required: true
        default: 'https://www.uschamber.com'
      title:
        description: 'Merge Comment'
        required: true
        default: 'Running Cypress Tests'

jobs:
  cypress-run:
    runs-on: ubuntu-16.04
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      # Install NPM dependencies, cache them correctly
      # and run all Cypress tests
      - name: Cypress run
        uses: cypress-io/github-action@v2
        with:
          record: true
        env:
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
          CYPRESS_baseUrl: ${{ github.event.inputs.baseUrl }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          COMMIT_INFO_MESSAGE: ${{ github.event.inputs.title }}

(see the gist here)

Notice the on section - this seems to be a little-known fact about Actions - you can actually pass variables in from a remote source. This is perfect for us because we want to give it the URL of the Pull Request environment, not our production URL!

Truthfully, I'm not sure we need the COMMIT_INFO_MESSAGE, but if I figure that out I'll update it later.

Also note that we are using some secrets. Secrets can be added to your repository under the Settings > Secrets UI. Grab your Cypress Record Key from Cypress Dashboard (log in at cypress.io for a free account), then drop it in as a secret called CYPRESS_RECORD_KEY. The GITHUB_TOKEN is automatically created by Github, so don't worry about that one.

Now that we've gotten this far, go ahead and trigger your script! You can hit it manually under the Actions Tab of your repository:

If it errors out, follow the error messages to hopefully debug your way out of the issue.

Platform.sh Activity Scripts

Now that we have a manual action set up we need a trigger. And of course, we don't want it triggered when the Pull Request is created, but instead once the environment is created.

Enter Activity Scripts. Platform.sh allows you to basically hook into the deploy process of its containers and run any JavaScript after the fact.

Quick note on these: The scripts only run ES5, and they are incredibly difficult to debug. I really hope Platform.sh updates them in the future to work with promises and ES6+. It would also be really nice to trigger them without pushing arbitrary code.

Rather than pasting the code here - I'll link to the gist that you can copy.

In order for this script to work you'll need to create a Github Token and store it in your Platform.sh environment variables as GITHUB_AUTH.

Github Token permissions for your reference:

To understand the activity script, you'll need to understand that Platform.sh has a different git strategy than Github.

This script gets the base URL from Platform.sh, grabs the Platform.sh branch name (pr-x) and then hits the Github Pull Request API to get the source branch name feature-x, then hits the Github Actions API and hits our cypress.yml action with the branch name and environment URL.

If you want to debug the script you can use the platform integration:log <integration_id> to get the latest run. It will show you anything that was printed to the stdout.

Cypress Dashboard

The final step in the whole strategy is to post the results of the test right back to the PR. Luckily, Cypress Dashboard handles this for us. Remember how we grabbed the RecordID earlier in the Action script?

Navigate to cypress.io and check your Project Settings. Fill out the Github Integrations section, and you're DONE.

Testing

Now give it a whirl! Open a new PR, watch the environment automatically build in Platform.sh, then check the log for the script to see it trigger Github Actions, then watch the actions run, then see it post a status and comment to your Github Pull Request.

This has helped our team go much faster! Hopefully it helps your team too.

Shout out to Kevin Old @ Cypress who proactively reached out to me on Twitter when I was at wits end and dealt with all my questions on making continuous testing a reality for us.