DEV Community: Mike Street

Running Playwright in Gitlab CI

Mike Street — Fri, 09 May 2025 00:00:00 +0000

After setting up my smoke tests with Playwright, I wanted to run them in Gitlab CI.

Following the Playwright documentation, I set up my YAML file to run the tests after deployment

test:playwright:smoke:
  cache:
    key:
      prefix: npm
      files:
      - package.json
      - package-lock.json
    paths:
    - node_modules/
    policy: pull
    untracked: true
  stage: monitor
  image: mcr.microsoft.com/playwright:v1.52.0-noble
  variables:
    PLAYWRIGHT_ENV: production
  script:
    - npm run test -- app/*/*/smoke.spec.ts
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_DEPLOY_FREEZE == null && $CI_PIPELINE_SOURCE

Note there are some extra bits in the YAML above which I won't repeat but are always good to see

cache: In a previous step, I've run an npm install and cached the node_modules folder to speed up the job
My npm run test looks like playwright test --grep-invert @vr to exclude visual regression tests by default (although we are then specifying to run only the smoke tests)
The PLAYWRIGHT_ENV is specified as we are using our custom Playwright framework for dynamic hosts
The rules prevent the smoke test from running on anything except main when there isn't a deployment freeze

The issue we ran into was with a version mis-match. The image version we were using was different to that in the package-lock.json. We were presented with this error:

    ╔══════════════════════════════════════════════════════════════════════╗
    ║ Looks like Playwright Test or Playwright was just updated to 1.50.1. ║
    ║ Please update docker image as well.                                  ║
    ║ -  current: mcr.microsoft.com/playwright:v1.52.0-noble               ║
    ║ - required: mcr.microsoft.com/playwright:v1.50.1-noble               ║
    ║                                                                      ║
    ║ <3 Playwright Team                                                   ║
    ╚══════════════════════════════════════════════════════════════════════╝

We could update the image to match the package-lock version, however this CI config is shared between projects and it would be a pain to keep them all in sync - especially as the CI fails if this error crops up.

Why?

Why does this happen? Playwright installs a specific version of the browsers to match the code version. The issue happens when you have a different Playwright version installed to the browsers set up.

The Playwright Docker image has the browsers set up, so running the local Playwright code with the global browsers throws the version mis-match.

We needed to run the local npm run test instead of a global npx playwright test as we needed the Playwright framework to be loaded to parse the custom code.

The solution

In the end, the solution was a one-line change (always the way) - adding npx playwright install before running the tests (this is actually done in the Github action docs for Playwright)

test:playwright:smoke:
  #...
  script:
    - npx playwright install
    - npm run test -- app/*/*/smoke.spec.ts

By running npx playwright install before the test, Node syncs the browser versions with the code to allow them to run. We still use the Playwright docker images as they have all the right tech set-up.

Other ways of solving it

The other ways I thought of solving the issue

Keep everything in sync

The first option I considered was keeping everything in sync. Pin the playwright version in the package.json file to prevent anything updating and then using the corresponding Docker image. This would be an absolute pain with maintenance as the Gitlab CI YAML is shared. We would have to have a release day where we bump all the projects and change the YAML to prevent it blocking a deployment.

No custom framework

If we weren't using the custom Playwright framework, we could have run npx playwright test without the local node_modules folder. This would have used the globally installed Playwright which matched the cached browsers.

The danger with this, though, is if your test is (for some reason) incompatible with the version you are using (say, if you were using a new version locally than your Docker image) it could cause the tests to fail

Dynamic image usage

I considered extracting the current version during the npm install job and caching it as a variable to use later. I didn't get this far as I found the solution above, but for reference:

I would have run npm run test -- --version and stored the output to use later.

Read time: 3 mins

Run unit tests in Playwright

Mike Street — Wed, 08 May 2024 00:00:00 +0000

I'm a big believer in using the right tool for the job - but I'm a bigger believer in the best tool for the job is the one you've got.

While you might not install Playwright to just do unit tests, if you already have it installed, there is no need to install an additional dependency (such as Jest) to check your functions & methods.

In simple terms, unit tests are the idea of testing a specific function - the classic example always given is passing two parameters into a function that adds them together and produces a single integer of which you can test the result.

For some of our projects we already have Playwright installed and wanted a way to unit test some utility classes without using a separate framework (and thus, loading a whole new set of dependencies & files).

This post assumes you already have Playwright installed and configured

The Code

For our examples, we are going to be using a function which limits an array to the first X number of items

export default function(array, limit) {
    return array.slice(0, limit);
};

This lives in a file called limit.js - we'll be making our test next to it.

The Test

Our conventions state we would make a file next to the JS file as limit.spec.ts - however if you have Playwright setup, you probably have standards/conventions of your own.

// Import the playwright functions
import { test, expect } from '@playwright/test';

// Import our JS file
import limit from './limit.js';

// Create a new test
test('Limit', async () => {
    // Create a dummy array
    let data = ['a', 'b', 'c', 'd', 'e', 'f', 'g'];

    // Check our array length matches our expected length
    expect(limit(data, 3).length).toBe(3);
});

With this test code, we import the JS file and fire the function - we then check the output. We don't use the page or request object.

Watch out: If you have multiple projects pointing to the same directories (e.g. a Desktop and Mobile) this test will run multiple times.

You can still use test.describe to group tests & share data - for example, if I wished to test these Mean, Median and Mode functions from Ale House Rock, I could do something like:

import { test, expect } from '@playwright/test';
import meanMedianMode from './meanMedianMode.js';

// Verified with
// https://www.calculatorsoup.com/calculators/statistics/mean-median-mode.php
test.describe('Calculations', async () => {
    const data = [9, 10, 12, 13, 13, 13, 15, 15, 16, 16, 18, 22, 23, 24, 24, 25];

    const output = meanMedianMode(data);

    // Mean - average
    test('Mean', () => {
        expect(output.mean).toStrictEqual(16.75);
    })

    // Median - middle value
    test('Median', () => {
        expect(output.median).toStrictEqual(15.5);
    })

    // Mode - most frequent value
    test('Mode', () => {
        expect(output.mode).toStrictEqual(["13"]);
    })
});

View the original post: https://www.mikestreety.co.uk/blog/run-unit-tests-in-playwright/

Login with Puppeteer and re-use cookies for another window

Mike Street — Thu, 23 Nov 2023 00:00:00 +0000

FFor a recent project I needed to automate something which was only available in the CMS via a login. To help speed to process up, I created a script which can login with supplied credentials and store the cookies in a local file. The main process can then use these cookies to carry out the task rather than needing to login each time.

A working example of this code can be found in this git repository.

What is Puppeteer?

Puppeteer is a Node/NPM package which allows you to create & control a headless Chrome instance, allowing you to do front-end/UI based tasks programmatically. It is hugely powerful and worth investigating if that is your thing. One of the most common examples is opening a page and taking a screenshot or submitting a form for testing.

In this instance, we are going to login to the CMS and then store the cookies in a file.

Login to your site

Below is the code to login to the site and store the cookies - there is some explanation text afterwards with some more details.

To make this work, you need to install both puppeteer to carry out the work and fs to write the file. If you are using the cookies later in the same file, this bit isn't required.

Install the dependencies

npm i puppeteer fs --save

Creating a `login.js`

Save this code in a file (e.g. login.js) and then run it via command line (e.g. node login.js).

I would recommend changing the headless value to false while you are testing, as this opens the browser and allows you to watch the code execute and spot any issues

// Require packages
const puppeteer = require('puppeteer');
const fs = require('fs');

// Login credentials
const url = '',
    username = '',
    password = '';

// Create a login function
const login = async () => {
    // Create a new puppeteer browser
    const browser = await puppeteer.launch({
        // Change to `false` if you want to open the window
        headless: 'new',
    });

    // Create a new browser page
    const page = await browser.newPage();

    // Go to the URL
    await page.goto(url);

    // Input username (selector may need updating)
    await page.type('input[type=text]', username);
    // Input password (selector may need updating)
    await page.type('input[type=password]', password);
    // Click the submit button
    await page.click('button[type=submit]');

    // Wait for a selector to be loaded on the page -
    // this helps make sure the page is fully loaded so you capture all the cookies
    await page.waitForSelector('main');

    const cookies = JSON.stringify(await page.cookies());
    await fs.writeFileSync('./cookies.json', cookies);

    // Optional - sessions & local storage
    // const sessionStorage = await page.evaluate(() => JSON.stringify(sessionStorage));
    // await fs.writeFileSync('./sessionStorage.json', cookies);

    // const localStorage = await page.evaluate(() => JSON.stringify(localStorage));
    // await fs.writeFileSync('./localStorage.json', cookies);

    // Close the browser once you have finished
    browser.close();
};

// Fire the function
login();

Read through the comments as they should help guide you where things may need altering - the main thing to watch out for this the field selectors when entering a username & password and the selector for when the page has loaded.

The other thing to watch out for (that this does not cater for) is 2FA. It may be you need to open the browser window and enter it yourself before proceeding.

You can also choose to store the session and local storage, should your application use this for authentication.

Using the cookies

Once the above script as run, you should have a cookies.json file sitting alongside your login script. If you opted to also collect the localStorage and sessionStorage then these files will also exist.

Once again you will need puppeteer and fs as dependencies so you can load the cookie file.

Create your secondary script which will utilise the cookies with the following code as a base:

// Load dependencies
const puppeteer = require('puppeteer');
const fs = require('fs');

// Load the cookies into the page passed in
const loadCookie = async (page) => {
    // Load the cookie JSON file
    const cookieJson = await fs.readFileSync('./cookies.json');

    // Parse the text file as JSON
    const cookies = JSON.parse(cookieJson);

    // Set the cookies on the page
    await page.setCookie(...cookies);
}

// Our main function
const run = async () => {
    // Create a new puppeteer browser
    const browser = await puppeteer.launch({
        // Change to `false` if you want to open the window
        headless: 'new',
    });

    // Create a new page in the browser
    const page = await browser.newPage();

    // Load the cookies
    await loadCookie(page);

    // Load your super secure URL
    // await page.goto(https://super.secure/url);
    // Do more work
    // Profit

    // Close the browser once you have finished
    browser.close();
}

// Run it all
run();

From there you can navigate through your system as if you were logged in.

Read time: 5 mins

Tags: Node, NPM

Adding Tina CMS to 11ty

Mike Street — Sat, 14 Jan 2023 00:00:00 +0000

This website is powered by 11ty. I've written about 11ty before and will talk the ear off anyone who even mentions Static Site Generators (or just websites) about 11ty. I love it, it's great.

The only disadvantage with using Markdown and a static site is editing content. It's great having your content with the code as you can leverage the power of your IDE and do programmatic changes if required, however, sometimes you just want to fix a typo or write a quick note - normally from your mobile.

What I wanted was a CMS which would work the existing ecosystem. Something which will commit to the repository so I can still clone down my site and edit it, but also so I can make small changes on the go without battling the Gitlab/Github IDEs on mobile.

Enter Netlify CMS and something I've written about before. I tried to add it to this site but got in a muddle and the mobile experience isn't the best, so I went on the hunt for something else.

I eventually came across Forestry CMS. My banner blindness kicked in and it was only once I had signed up, configured and added it to my site did I notice the red banner at the top mentioned they were closing down in March - not good for longevity.

However, it did mention it was being replaced with TinaCMS - they have a nice little free tier and seemed to do what I wanted, so I thought I would give it ago.

After signing up I noticed my first hurdle - they only support Github. It was a bit of a shame coming from Forestry, which support Gitlab but I migrated my repository to trial the CMS.

As I had set up with Forestry already, the TinaCMS setup was able to migrate some of the config to the .tina folder. On further inspection, the config.js file it generates contains a JSON config object, which is able to be manipulated for the fields I wanted - including changing the order and setting some defaults.

The noteworthy changes I did were for the Notes section of the config. I wanted the least amount of friction between me wanting to post and pressing the save button, so I have the filename generated from the title and the date to be now.

The full config can be found in the repo, however, the snippet below should help identify the solution.

export default defineConfig({
    schema: {
        collections: [
            {
                label: "Notes",
                name: "notes",
                path: "app/content/notes",
                format: "md",
                ui: {
                    filename: {
                        slugify: (values) => {
                            return slugIt(values?.title);
                        },
                    }
                },
                defaultItem: () => {
                    return {
                        date: new Date().toISOString(),
                    }
                },
                fields: [
                    {
                        type: "string",
                        name: "title",
                        label: "title",
                    },
                    {
                        type: "string",
                        name: "link",
                        label: "link",
                    },
                    {
                        type: "datetime",
                        name: "date",
                        label: "date",
                        ui: {
                            dateFormat: 'YYYY-MM-DD',
                            timeFormat: 'HH:MM:SS'
                        }
                    },
                    {
                        type: "rich-text",
                        name: "body",
                        label: "Body of Document",
                        description: "This is the markdown body",
                        isBody: true,
                    },
                ],
            },
        ],
    },
});

To set a default date and time in TinaCMS

The date field has the dateFormat and timeFormat specified, but has a default set in the defaultItem function at the top.

To have an auto-generated filename

This is done in the ui section of the config. I don't quite understand how it works, but a copy and paste from StackOverflow and a tweak using my own slugify function solved it.

const slugIt = function (str) {
    if (str) {
        str = str.replace(/^\s+|\s+$/g, ''); // trim
        str = str.toLowerCase();

        // remove accents, swap ñ for n, etc
        var from = "àáäâèéëêìíïîòóöôùúüûñç·/_,:;";
        var to = "aaaaeeeeiiiioooouuuunc------";
        for (var i = 0, l = from.length; i < l; i++) {
            str = str.replace(new RegExp(from.charAt(i), 'g'), to.charAt(i));
        }

        str = str.replace(/[^a-z0-9 -]/g, '') // remove invalid chars
            .replace(/\s+/g, '-') // collapse whitespace and replace by -
            .replace(/-+/g, '-'); // collapse dashes
    }

    return str;
};

I now have TinaCMS running on this site, with notes auto-posting to Mastodon thanks to this article from James Montemagno.

Be Code Agnostic

Mike Street — Thu, 05 Jan 2023 00:00:00 +0000

I wrote a blog about being device agnostic which is an idea about ensuring your data is not tied to one kind of device, or one manufacturer. I believe you should be able to export your data regularly (in a generic format) or it should exist in a more device-fluid state. Services and devices shouldn't try to lock users in by holding their data hostage. The idea is you can move between devices (or lose a phone) and not be ground to a halt (or be worried about where your data has gone).

While converting my website from Craft CMS to 11ty, a similar concept approached me: when creating websites, we should try and make our content as code agnostic as possible. This not only applies to our personal websites, but should apply to our clients sites as well.

Code should just be the transportation for the content and should not influence how it is crafted. If the package/library/CMS you are using or the framework that powers your website disappeared, can you honestly say that you could easily access your content to migrate to a new platform?

With the move to 11ty, a lot of my content was already written in Markdown in the Craft CMS database. Despite them falling out of fashion with the cool kids, database powered websites are often a great measure of code agnosticism. These frameworks and content management systems rarely stuff too much opinionated code in there. Although, saying that, we use TYPO3 at work which has <a href="t3://page?uid=713"></a> style links. These would be difficult to parse/transpose into complete URLs if you were migrating, but I digress.

Craft CMS is great example of keeping content, content. I was able to download all my posts as a JSON and didn't have to do any processing on the content itself. With the export, I was able to write a small PHP script that turned the big array into individual Markdown files. As a side-note, JSON, like Markdown, is a great framework-free file format. They seem to have become a pretty solid and reliable standard that the web industry has embraced.

Sure, 11ty requires some front matter, but that seems to be moving into a standard when writing content with Markdown. It does make me feel slightly "off" that framework specific content is in my files, but I feel confident that, if 11ty disappeared tomorrow and my computer died, I could extract, process and get my content extracted again - it's about finding a compromise between framework standards and keeping it clean.

Throughout my site, I have <div class="info"> boxes and such. I considered turning these into 11ty shortcodes, but this tied me deeper into the 11ty ecosystem. I love 11ty, but if the shortcode syntax changed, or I needed/wanted to migrate my content elsewhere, I would need to do a lot of find and replacing. I figured it best to stick to plain ol' HTML.

I suppose that is the crux of it really - it's not necessarily about whether your content does or does not have framework specific code in it, it's whether you feel confident in getting it out. Can the framework specific code be extracted and parsed? Are you tied in or can you get your words out without too much trouble?

Perhaps you need a headless CMS? A headless CMS is a content management system without the front-end. You use APIs and to access the data for a website which means you can use any technology you want to build your website or app.

This blog seems to be turning into a massive advocacy into custom APIs serving up data. The nature of a good API ensures the content stays code agnostic so it can be used by many apps. I've never used GraphQL, but I can see the appeal as the data is just data, and the framework determines how it gets that content and how it uses it.

So be more code agnostic, let your content be content and your code be code.

Read time: 3 mins

Tags: Ramblings

How to delete a Git branch

Mike Street — Sat, 19 Nov 2022 00:00:00 +0000

Deleting branches is important to ensure old code doesn't hang around or the wrong thing doesn't get merged

Note: In the examples below pow will be used for the branch name

Delete a local branch

If you have a local branch you want to delete you can run

git branch -d pow

If your branch has not been merged into your current branch you need to change the -d to a capital:

git branch -D pow

Delete a remote branch

If you wish to delete the branch via command line, you have to "push" it with a colon (:) preceding the name

git push origin :pow

Updating your local branch data

If you deleted your remote branch from another computer (or via the website if on Github/Gitlab), you might find it is still listed when running a git branch -a. This means your local Git repository thinks the branch still exits and could cause conflicts if you try to create a branch of the same name.

To remove these, you can fetch with an additional --prune parameter

git fetch origin --prune

Tip: If you want it to always prune when you do a git fetch origin, you can set this as a global setting:

git config --global fetch.prune true

Read time: 1 mins

Tags: Git

How I improved the speed of Docker builds in Gitlab CI

Mike Street — Wed, 08 Jun 2022 00:00:00 +0000

Our Gitlab CI powered Docker build is now 500% faster (I think?) - taking it more than 15 minutes to around 3 minutes to build and push to a registry.

Unfortunately, there isn't one thing to copy and paste - this blog post is more a collection of concepts and steps I took to drop the build time. 15 minutes wasn't the end of the world, but I wanted to see how far I could take it.

This blog post features some pseudo dockerfile & Gitlab CI yaml code - it might not be able to be copied but should give you some idea as to what was happening

Application Background

The application is a website which runs on TYPO3 - a composer-based open-source CMS and the client provided a Docker environment to host the website. I'm a big fan of Gitlab CI, so took the challenge to completely compile and build a Docker image via Gitlab, rather than building assets and installing dependencies locally (which I had done in the past).

The site also features NPM dependencies - nothing over the top, but our SCSS and Webpack JS is compiled using Gulp, along with optimising images and building a sprite.

Note: I'll be using "website" and "application" interchangeably - they ultimately mean the same thing.

Requirements

The build needed to:

Clone the repository
composer install - to get the backend dependencies
npm ci - this is "clean install"
gulp compile - this generates all the assets
Copy the compiled assets into an apache-ready Docker container
Push the container to the registry

Docker multi-stage build

The first attempt at building the application was to use a multi-stage Docker build (this is actually the project that was written for). This used a custom image which had NPM and Composer preloaded to install and compile the dependencies and the final assets we passed to the Web image for pushing.

The Docker file looked something like

FROM custom-node-and-composer-image as build

# COPY package.json, composer.json & app files
composer i
npm ci
gulp compile

FROM custom-web-app

COPY --from=build /app/dist /var/www/

With the Gitlab CI file having a script block like:

script:
  - docker build -t registry:my-image .
  - docker push registry:my-image

This meant the Gitlab file was extremely lightweight, which was great as it meant you could compile the full Docker image locally, if required. It left Docker to do most of the heavy lifting which seemed to make the most sense.

This built the Docker image as expected, but took a whopping 15 minutes from start to finish. Granted, it is all done on Gitlab so you can push and carry on with some other work if required, but it still needed you to go back 15 minutes later to test everything worked as expected.

Installing with Gitlab CI

In a bid to try and speed up the process I took a blind punt and seeing what would happen if I took the npm and composer install steps and moved them outside of the Docker build. Let Gitlab do the installing and then copy the resulting files to Docker.

Docker:

FROM custom-web-app

COPY /app/dist /var/www/

Gitlab CI:

image: custom-node-and-composer-image-with-docker
script:
  - composer i
  - npm ci
  - gulp compile
  - docker build -t registry:my-image .
  - docker push registry:my-image

The premise behind this is to run all our build steps outside the Docker file and then copy the resulting files in.

We use the base Docker image used in the original multi-stage build to run the Gitlab CI step - the only difference being that Docker is installed inside the Docker image so it can build the final… image (Docker and image were mentioned far too many times).

This obviously comes with the big disadvantage of suddenly not being able to build the Docker image locally - however we use ddev for local development so this wasn't an issue for us.

This dropped the build times from 15 to 7 minutes - halving the time it takes just by moving the build step out of Docker.

Caching

With the time down to 7 minutes for a fresh build, it was time to start looking at caching. Using the next few steps, I was able to get the build down to around 3 minutes. However, if all the caches got invalidated then there is likelihood the next build would go back up to 7 minutes again.

Docker Caching

Docker has caching build in to the process - that is why a fresh build always takes longer. Each step is made into a mini image that then gets "loaded" into the main build. More steps means more HDD space but a speedier build. Less steps means smaller space, but a bigger chance of one of the steps' cache being invalidated.

During this exercise I learnt a lot about Dockerfiles and optimising them to get the best performance. The big thing I learnt was:

If anything changes in a step, it invalidates the cache for every step after that

Knowing that really helps order the copy steps - make sure you have the files which change the least being copied first (base config files etc.). The last COPY (or other step) should be the files that change the most - for example if you have FE assets with cache-busting file names.

Gitlab CI Caching

The final piece to the puzzle was to use Gitlab CI caching. You can use the cache in the runner itself, however this isn't very reliable - especially if you have several runners.

Gitlab runners don't retain for projects or even jobs in a pipeline, so you can have a different runner picking up different stages - if you use the runner cache there is a high likelihood it would be out-of-date or (dangerously) wrong.

The most efficient (and safest) method is to use an external service like an Amazon S3 bucket or a self hosted solution, such as Minio - this has S3 compatible APIs so can be used as drop in replacement.

Our Gitlab CI file features a global cache of the node_modules and vendor folders, which is invalidated if the respective lockfiles change

cache:
  - key:
      files:
        - composer.lock
    paths:
      - vendor/
  - key:
      files:
        - package-lock.json
    paths:
      - node_modules/

There is plenty of documentation about Gitlab Caching on their website.

The last step was to change my npm ci command to just npm i. The ci command deletes the exsiting node_modules folder, which defeats the point in caching it!

Conclusion

Removing the build from Docker, re-ordering my Dockerfile and leaning on caching are the main steps I took in dramatically dropping the build time of my Docker image using Gitlab CI.

At time of writing, the last deployment (including linting) took 3 minutes and 21 seconds - the build of that taking 2 minutes and 9 seconds.

Read time: 5 mins

Tags: Docker, Gitlab CI

Get your Eleventy Site into the real world (Netlify, Cloudflare pages or any host)

Mike Street — Mon, 19 Apr 2021 00:00:00 +0000

Eleventy (11ty) is a great entry point for getting a markdown or JSON content from files to a website. There are plenty of tutorials out in the world to get 11ty up and running in a development environment.

But once you are ready to get your website out into the public domain, what are the options and how do you go about it? There are a couple of free options (with paid extras), as well as self-hosting if that is your JAM (geddit? Because 11ty is JAMstack... No? I'll move on...)

There were 3 main contenders when I planned this post a few weeks ago, however a new arrival (Cloudflare pages) gives an extra bit of variance. For ease of access with low barriers of entry, I would recommend:

Self-hosted
Netlify
Github/Gitlab pages
Cloudflare Pages

Self-hosted

This is the most manual of the options, but gives you the most control over your website. It is also the only one that requires a domain name of sorts to access the final result (the other options give you a placeholder/branded domain name).

With 11ty, you can either generate the site in a one-off, or run a watcher, which recompiles on save.

When self-hosting, you should run the "compile" step of Eleventy and manually move the source files to your designated hosting.

npx @11ty/eleventy --serve

This will compile your site into your output dir as specified in your .eleventy.js. You can transfer this directory to your hosting environment, be it via FTP, SCP/RSYNC or you can zip it up to transfer it.

The other option is to git clone or similar your source files and run the compile step on the server you are hosting it on - some shared hosting environments, however, might not let you install npm/node.

Netlify

Netlify is a fantastic hosting service that provides a fairly straightforward deployment pipeline, as long as you are comfortable with git (although if you're not, they offer alternatives).

To get your 11ty site onto Netlify with git:

Push your git repository to Github, Gitlab or Bitbucket
Sign up for a Netlify account
Click "New site from Git"
Connect to your git repository host & select the repository
Choose the branch you want to use for deployment
Enter the build command and publish directory
- Build command is npx @11ty/eleventy
- Publish directory is your output directory specified in your .eleventy.js file
Click "deploy site"

Netlify will take a few minutes to deploy your website but, once complete you are provided with a random .netlify.app URL to view your website. You can rename this or you can link a custom domain (even using Cloudflare).

If you don't want to/can't connect your git provider (or your 11ty site isn't in a git repo), Netlify offer a drag and drop interface (at the bottom of the "Sites" page). Run the 11ty compilation steps locally and upload the output directory.

Side note: This site, Behind the Source and Hovélo are all hosted on Netlify.

Github/Gitlab pages

Gitlab and Github pages are another great way of hosting your website for free. It takes a bit more configuration than Netlify, but if your website data is hosted on one of these services, there might an argument to reduce the number of services required.

I'm not a Github user really, and haven't really investigated using Github pages so I won't attempt to muddle my way through a tutorial. Lea Tortay has written a comprehensive tutorial on this topic.

For Git lab however, I have just experimented and been able to spin my site up in the space of 15 minutes (pray the lords for 11ty!).

To get this working, the output directory had to be public.

Ensure your git repository is on gitlab.com (this can be done with self-hosted instances too, but the following steps follow the provided Gitlab instance)
Add a new file to the root directory called .gitlab-ci.yml
Add the following contents

image: node:latest# This folder is cached between builds# http://docs.gitlab.com/ee/ci/yaml/README.html#cachecache: paths: - node_modules/pages: script: - npm install - ./node_modules/.bin/eleventy artifacts: paths: - public only: - master

Enable CI/CD and Pages on your repository
Push your changes to Gitlab

CI/CD should fire and compile your pages, you website should then be available at <username>.gitlab.io/path/to/repo. You can add a custom domain in the Gitlab pages config if that is the route you choose to take.

Cloudflare Pages

Cloudflare pages are the new kids on the block and (at time of writing) in beta. Looking like a direct competitor to Netlify, it appeals to me as an avid Cloudflare user to keep everything in one place.

Currently they only support Github (but Gitlab and other things are planned), however the workflow should be very familiar.

Sign up to Cloudflare and ensure your git repository is on Github
Connect your Github account and select your repository
Give it a name and select the branch
Enter the build command (e.g. npx @11ty/eleventy) and your output directory
Click save and deploy

You'll get a pages.dev domain, but this can be replaced with a custom domain name if desired.

And there we have it. 4 methods for getting your Eleventy site out into the wild. You have no excuse now!

Read time: 4 mins

Tags: DevOps, 11ty, Netlify, Github, Gitlab, Cloudflare

Using Cloudflare workers to publish your scheduled 11ty posts

Mike Street — Mon, 05 Apr 2021 00:00:00 +0000

Intro

With my switch to 11ty away from a PHP based CMS I was struggling to see how I could schedule posts. When 11ty builds, it builds static HTML files - with no way of "show this post when this date has passed" functionality.

I like to write my posts and schedule them - I find it gives me more motivation to write, but I digress. I wanted a way to rebuild my site when a post was due to go live.

The straightforward approach is to trigger a re-build manually in Netlify on or after the day the post was scheduled. My "if" blocks would pick up the post in the "past" and populate the listing, RSS and sitemap pages with the new URL.

However, this required me to remember and be near a device that can do it (their interface is surprisingly mobile friendly).

The next approach was to use the Netlify build hook to trigger a rebuild every night. This would tick the automation box, but would quickly eat up my free Netlify minutes, along with doing several unnecessary builds (I publish every other week, generally).

I could schedule a build every fortnight, but for those times I have a Notes post scheduled or similar, it would potentially be 2 weeks before it appears (unless I publish there and then).

What I wanted is for a system to know when my next scheduled post is and regularly check for that date. It can then trigger a build if required.

Intro
Table of Contents
Objectives
TL:DR;
Steps
- Create an "upcoming" endpoint
- Get the build hook
- Make the Worker
- Using the UI
- Worker JavaScript
- Cloudflare Wrangler CLI
Conclusion

Objectives

With the preamble out the way we are going to

Build a page with 11ty which lists the next scheduled post
Check this date regularly, if it matches today (or is in the past) trigger a rebuild

TL:DR;

Create an upcoming.json or similar file to list your next post (Example on Gitlab)
Host your site with Netlify and get the build hook
Set up a scheduled Cloudflare Worker (Gitlab repo) to check the file regularly

Steps

Create an "upcoming" endpoint

The first step is to create an endpoint (page) somewhere with the date of your next post in it. For me, I found it the least path of resistance for this to be a json file.

Using Remy Sharp's Post I created a blog collection, which is posts that are not a draft neither in the future

const now = new Date(); const livePosts = p => p.date <= now && !p.data.draft;  config.addCollection('blog', collection => { return collection .getFilteredByGlob('./app/content/blog/*.md') .filter(livePosts) .reverse(); });

Using the same technique, I have also made a drafts collection in my .eleventy.js

config.addCollection('drafts', collection => { return collection .getFilteredByGlob('./app/content/{blog,drafts}/*.md') .filter(p => !livePosts(p)) .reverse(); });

The drafts collection contains both posts that are due to go live or are marked with draft: true in the front matter.

One method here would be to make a upcoming/scheduled/pending collection which doesn't include draft: true.

As I only needed this function in one place, I decided to use JavaScript front matter to create a "collection" on the fly.

Create a file called "upcoming.json.njk" with the following front matter

--------js{ permalink: "upcoming.json", post: function() { // Get the drafts collection let pending = this.ctx.collections['drafts'] .reverse() // Remove everything that is marked as draft .filter(p => !p.data.draft); // Return the title and date of the next post return { title: pending[0].data.title, date: pending[0].date }; }}---

The first line sets a nice permalink, while the following lines load the drafts collection, reverse and filter out the posts marked as a draft.

Lastly, we return just the first blog post - there is no need to include all of them as we only care about the next one.

As our function returns an object, we can use Nunjucks built in functions to output as valid JSON

{{ post() | dump | safe }}

This results in what you see on upcoming.json.

Get the build hook

Netlify have some fantastic documentation on how to find your build hook.

Make a new hook (I called mine "worker" but you could be more specific) and make a note of the URL it gives you.

We'll store that as an environment variable in our Cloudflare Worker settings.

Make the Worker

This uses Cloudflare Workers, but anywhere that can run JavaScript on a cron/scheduled task should work

The next step is to make a Cloudflare Worker to run as often as you wish to check the date of your "pending" post.

Note: For this step, I am assuming you have a Cloudflare account and a basic understanding of Cloudflare Workers.

You can either make a new worker through the user interface on the CLoudflare website or you can use the Cloudflare Wrangler CLI.

Using the UI

You can make a Cloudflare worker using their website with the "Quick Edit" button.

Go to the Workers page and click "Create a Worker"
Paste in the Javascript below (the same file is on Gitlab)
Navigate to the setting page and under the "Triggers" tab click "Add Cron Trigger" and input the pattern for how often you want it to trigger
Navigate to the "Settings" tab and create 2 environment variables

BUILD_HOOK - This is your netlify build hook URL
UPCOMING_JSON - This is the full path to your /upcoming.json made in step one (e.g. https://www.mikestreety.co.uk/upcoming.json)

Worker JavaScript

/*** Fire the check on both the schedule and when accessed via the URL */ addEventListener('scheduled', event => {  event.waitUntil(checkBuildRequirement())});addEventListener('fetch', async event => {   event.respondWith(checkBuildRequirement())});/*** Shall we build it?*/async function checkBuildRequirement() { // Get the pending posts and parse as JSON   let pending = await fetch(UPCOMING_JSON) .then(data => data.json()), // Should the build be triggered? rebuild = 'No build needed'; // If we have an item and the date is in the "past" if( pending && pending.date && (new Date(pending.date) < new Date())    ) { rebuild = `Building ${pending.title}`; await fetch(BUILD_HOOK, { method: 'POST', });    }   // Return some text return new Response(rebuild, { headers: { 'content-type': 'text/plain' }    });}

Cloudflare Wrangler CLI

If you chose to use the CLI, you can use my git repo as a base. Rename wrangler.toml.example to wrangler.toml and fill in the following:

name - lowercase, hyphenated
account_id - found on the workers overview page
zone_id - found on the right hand side of your main Cloudflare dashboard (as this worker is scheduled, it doesn't need to be the domain of your 11ty site - but it helps)
UPCOMING_JSON - This is the full path to your /upcoming.json made in step one (e.g. https://www.mikestreety.co.uk/upcoming.json)

Lastly, review and adjust, if required, the schedule for your worker. The notation is that of a standard crontab.

The last thing you need to do is set you build hook path. From your worker project directory, run the following:

wrangler secret put BUILD_HOOK

It will prompt for your URL and store it on Cloudflare.

The last thing is to run the following to get your worker live.

wrangler publish

Your output should look something like

✨ JavaScript project found. Skipping unnecessary build!
✨ Successfully published your script to
 https://your-worker-name.account.workers.dev
with this schedule
 0 9,15 * * *

Conclusion

You should now have your 11ty site rebuilding on Netlify only when you have a new post scheduled to go live.

Let me know if you have an questions or feedback.

Side Project Imposter Syndrome

Mike Street — Fri, 02 Apr 2021 00:00:00 +0000

Imposter syndrome is prevalent in every industry, not least the web world. Many people who I respect and admire greatly have blogged about their own feelings of not belonging or not deserving of their place.

I have been fortunate enough to not have experienced imposter syndrome that much in my professional career. I appreciate my privilege has no doubt played a huge role in that, but I've kept my head down and felt like I've worked hard to get where I am today.

More recently, however, I have experienced imposter syndrome more and more in a working day, especially since working my way up to director level. Having meetings with clients and trying to appear as this well-travelled, experienced and seasoned professional really kicks you in the feels when you know you're not.

I kind of understand feeling imposter syndrome at work as you are paid are being paid to have "knowledge" and if you feel like you don't have that knowledge then you feel like a fake.

What I don't get though, is why I experience impostor syndrome with my side & personal projects. It was a recent topic of conversation that came up on on a podcast I was recording and, since then, I've not been able to stop thinking about it.

I find it hard to self-promote and, even when I do, I don't have a lot of confidence in what I've written and it shows. So far, in 2021, I've managed to publish a blog post every other week. Despite this, I've only tweeted links to my personal site twice this year. This isn't me forgetting, I just don't trust the words i've written - especially when it comes to technical posts. I've been a "professional" developer (well, I've been paid to develop websites) for 12 years - but I still don't believe in my abilities. Even as I'm writing this, I'm questioning about whether I should even post it.

Throughout February and March, I recorded 6 episodes of a podcast series Make Life Work - On The Side Takeover. I didn't blog about it, I didn't tweet about it, I didn't even tell my family. Every week that podcast went out I felt like a fraud - who am I to have the authority to speak on a podcast?

When a project completes at work, you get the reassuring emails from clients, the congratulations from co-workers and the potential new biz won off the back of it. With personal projects, though, there is no validation that what you are doing is good or even correct. But then, every now and then, a little trickle of priase comes in. Be it a tweet or someone buying me a coffee. They are rare but they are reminders that people read (and learn from) things I write.

So please excuse me if my posts feel guarded, if they seem simplistic or they're not "pushing the boundaries". Despite blogging for 11 years I still have trust issues with my words.

Why should you consider Cloudflare for your website?

Mike Street — Mon, 20 Apr 2020 08:47:00 +0000

Cloudflare is a service which, among other things, provides a CDN (Content Delivery Network), firewall, and performance layer for your website. It has plenty of paid upgrades and features and is a developer’s dream, but what advantage does it have for you to put your website “behind” Cloudflare and how does it work?

Asset caching

Cloudflare has some mega servers which cache your static assets for you, once they have been requested once. This has a huge benefit of saving your server load and bandwidth. With a non-cloudflare’d server, if a user requests a web page which features 10 images, the browser has to make 10 additional requests to your server. If each image is 1mb, that would be an extra 10mb your server would have to find, process and serve. This, multiplied by each user on your website, is more load and takes up valuable resources which could impact performance for other users.

With Cloudflare, they do Asset Caching. Once each image has been served once, they store a copy on their servers. When the second user requests the same image, Cloudflare serves it up from their servers, meaning your server has more time to do the important things - like processing that transaction or sending that email. It also helps your website performance, as many hands make light work.

Below is a screenshot from one of our client’s Cloudflare accounts, showing statistics from the past week - that’s a 90% caching rate.

If you want to take it one step further, you can cache static pages as well. Pages without contact forms or dynamic content are ideal candidates for this.

Worldwide CDN

Along with the asset caching, Cloudflare has a worldwide Content Delivery Network. With this, your cached asset gets replicated across servers in over 200 cities worldwide. When your user requests an asset, rather than serve it from your server which is in one location, Cloudflare will serve up the asset from the nearest server, meaning your website loads fast all over the world, not just near the country your server is in.

When we put Liquid Light behind Cloudflare our worldwide average total time for loading the homepage went from 1.121 seconds to 0.357 seconds. This was a combination of the CDN and the caching.

Firewall & DDOS Protection

Along with the caching and CDN, Cloudflare helps protect your site against brute-force attacks and threats against your website. Cloudflare has the advantage of serving over 12 million websites and so can identify malicious bots and users more easily than any operating system firewall. If Cloudflare thinks you are suspicious, it can show a challenge which allows “normal” users to still access your site. These firewall and security measures are ever evolving and being developed, meaning every day your website is becoming more secure and protected.

Security - we don’t get logins to your domains or Cloudflare
On the topic of security, when using Cloudflare you can share your account with us without sharing your login details. More and more companies are heading in this direction recently and Cloudflare is no exception. You set up the account in your name with your email address and then you can allow us access to your account. This means you don’t have to share your password and can revoke access at any time.

Nice DNS management

This is a slightly nerdy one, but the DNS management within Cloudflare is excellent, and can be controlled via an API. DNS records are like driving directions for your browser. When you enter your domain name, the DNS records tell your browser where to go to get to their destination. This doesn’t normally affect day-to-day operation, but being able to update your records easily saves a lot of time should you need to change your server or add a verification record (usually Google Search Console requires this).

Advanced features

All of the features above come with the free (yes, free) Cloudflare account. If you decide to upgrade to Pro (or even Business), this unlocks a whole host of features including:

Image optimisation - serving images up in a next-gen image format if the user’s browser supports it (smaller, faster-loading images)
Next-gen delivery - You can enable HTTP/2, which optimises the speed and order the webpage loads (more about this on the Cloudflare blog)
More page rules - Page rules allow you to cache, redirect or do a multitude of other things to pages and urls before the user even reaches your server. With the free account you get 3, as you upgrade you unlock more of these
Better security - From the Pro package up, you get a more enhanced firewall

As a side-note, if you look at their pricing page, you will also notice that at least Pro is recommended for "mission-critical" projects.

Cloudflare workers

Cloudflare workers are not directly part of the standard Cloudflare service per se, they are a separate offering, but by using Cloudflare for your DNS allows you to utilise them to their full potential. They allow you to run your own apps on the Cloudflare servers or modify intercept requests for a page or image to add functionality.

For one of our clients, we have used Cloudflare workers to prevent an extra request to their server. At the time of writing we have saved their server from having to deal with over 3.5 million requests in the last week.

What are the different SSL modes on Cloudflare?

Mike Street — Sun, 19 Apr 2020 06:30:00 +0000

Cloudflare offers free SSL certificates as part of its service, even on the free tier. There are 4 options when in the SSL/TLS section of Cloudflare and, depending on the selected option, will determine which port your server will be required to serve traffic. By default on any web server (unless specifically changed otherwise):

Unencrypted traffic via http is served over port 80
Encrypted traffic via https is served over port 443

Cloudflare SSL options

Cloudflare provides an SSL to allow you to provide a https URL, without needing to provide an SSL certificate yourself or serve your site up over port 443.

Off (not secure)

This uses port 80 on your server to serve the files and uses the http protocol in your browser URL bar - something which Chrome (and other browsers) will flag as insecure

Flexible

From the outside, this uses a https URL, however the SSL is only between the user and Cloudflare. From Cloudflare, the traffic is unencrypted and served over port 80 (This one took me a while to realise - I've lost many hours wondering why, when serving over port 443, my content wasn't appearing). Some CMS's require the server to be running on port 443 if the URL starts with HTTPS - that's where the next one comes in.

Full

Full uses an SSL between both the user & Cloudflare and Cloudlfare & the server. Your server serves files over port 443 and must have an SSL certificate installed to operate. However, this SSL certificate can be self-signed or invalid. Cloudflare does not verify the certificate but trusts it is ok. Cloudflare themselves offer a self-signed "Origin" certificate for you to download and install on your server, which allows you to use Full without having to go through the hassle of generating a certificate yourself.

Full (strict)

Full (strict) mode in Cloudflare is the same as Full, but the certificate needs to be valid and signed by a proper authority, for example, Let's Encrypt. This is the most secure but requires a real SSL certificate to operate.

In most cases Full will suffice, however, I wanted to run through generating a wildcard SSL certificate with a DNS provider which provides an API.

DEV Community: Mike Street

Running Playwright in Gitlab CI

Why?

The solution

Other ways of solving it

Keep everything in sync

No custom framework

Dynamic image usage

Run unit tests in Playwright

The Code

The Test

Login with Puppeteer and re-use cookies for another window

What is Puppeteer?

Login to your site

Install the dependencies

Creating a login.js

Using the cookies

Adding Tina CMS to 11ty

Be Code Agnostic

How to delete a Git branch

Delete a local branch

Delete a remote branch

Updating your local branch data

How I improved the speed of Docker builds in Gitlab CI

Application Background

Requirements

Docker multi-stage build

Installing with Gitlab CI

Caching

Docker Caching

Gitlab CI Caching

Conclusion

Get your Eleventy Site into the real world (Netlify, Cloudflare pages or any host)

Self-hosted

Netlify

Github/Gitlab pages

Cloudflare Pages

Using Cloudflare workers to publish your scheduled 11ty posts

Intro

Table of Contents

Objectives

TL:DR;

Steps

Create an "upcoming" endpoint

Get the build hook

Make the Worker

Using the UI

Worker JavaScript

Cloudflare Wrangler CLI

Conclusion

Side Project Imposter Syndrome

Why should you consider Cloudflare for your website?

Asset caching

Worldwide CDN

Firewall & DDOS Protection

Nice DNS management

Advanced features

Cloudflare workers

What are the different SSL modes on Cloudflare?

Cloudflare SSL options

Creating a `login.js`