DEV Community: Matt Smith

Github Action for Lighthouse

Matt Smith — Wed, 05 Feb 2020 14:12:19 +0000

After using a Github action to assess web page performance with Google PageSpeed, I found out that PageSpeed leverages a tool called Lighthouse for most of what it now provides. When I was configuring a Github action to check for Javascript library security vulnerabilities, I remember it using a package called Lighthouse, too.

🤔

According to Google, the Lighthouse project provides . . .

"Automated auditing, performance metrics, and best practices for the web"

NOTE: This is the same suite of tests that execute when you launch Google Chrome and use the audits tab in Developer Tools, and the same suite of tests that now provide a large portion of what is reported back by Google PageSpeed.

Some quick scouting on Github revealed some existing efforts focused around making Lighthouse easier to plug in to CI/CD and packaging it into a Github Action.

For the purpose of asserting and maintaining a baseline performance level on this site, we're going to try leveraging the Lighthouse Github Action. Thanks to Aleksey and the contributors!

The Github Action

First step: add a workflow file .github/workflows/lighthouse.yml in the Github repository which should trigger Lighthouse:

name: Lighthouse
on: 
  push:
    branches:
    - master
jobs:
  lighthouse:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - name: Audit URLs using Lighthouse
        uses: treosh/lighthouse-ci-action@v2
        with:
          urls: 'https://mattorb.com/'
      - name: Save results
        uses: actions/upload-artifact@v1
        with:
          name: lighthouse-results
          path: '.lighthouseci'

After committing that workflow file, it executes, and we can go grab the artifact generated by lighthouse and see what our scores look like.

Find Github action artifacts here

That zip file contains both a json and html representation of the report.

Inspecting the summary portion of the html report gives us overall scores in various areas and provides much more detail below that fold:

Lighthouse scoring areas, summary. For breakdown and recommendations, see full report.

Based on those numbers, decide where to draw the line and set a threshold to not fall below in each of the categories. Leave some room for variability in server performance though. Nobody wants to be dealing with false alerts regularly.

Create a .github/lighthouse/lighthouserc.json file in the repository with Lighthouse assertions you want to enforce:

{
  "ci": {
    "assert": {
      "assertions": {
        "categories:performance": ["error", {"minScore": 0.90}],
        "categories:accessibility": ["error", {"minScore": 0.80}],
        "categories:best-practices": ["error", {"minScore": 0.92}],
        "categories:seo": ["error", {"minScore": 0.90}]
      }
    }
  }
}

Configure the stanza in the Github Action at .github/workflows/lighthouse.yml to enact those assertions by pointing it to that config file:

✂️ 

    steps:
      - uses: actions/checkout@v1
      - name: Audit URLs using Lighthouse
        uses: treosh/lighthouse-ci-action@v2
        with:
          urls: 'https://mattorb.com/'
          configPath: '.github/lighthouse/lighthouserc.json' # Assertions config file 

✂️

Next time the action executes, it will check that we don't drop below these thresholds.

If one of those assertions fails, it will fail the workflow with a message:

Failing to meet the performance criteria

You can fail the workflow based on overall scores or make even finer grained assertions using items called out in the detail of the report. Check out the Lighthouse CI documentation on assertions for examples.

Going Further

Run a Lighthouse CI server to provide tracking of these metrics over time in a purpose built solution for that.
Use multiple runs to average out server performance and draw more manageable thresholds.
Use the concept of a page performance budget to set guide-rails and limits around size and quantity of external (think .js and .css) resources.
Shift left and assess a site change while it is still in the PR stage
Design specific test suites and set unique performance/seo requirements based on what kind of page is being probed.
Leverage a Lighthouse plugin to assess pages that have opinionated and strict design requirements for their target platforms (i.e. - AMP).

Github Action for Javascript Vulnerability Scanning

Matt Smith — Wed, 29 Jan 2020 14:44:17 +0000

photo credit: gisela-bonanno

Part of what is served by this web site includes 3rd party javascript libraries. The libraries included in a page are a mash-up of libraries and dependencies from a few sources.

Those libraries occasionally have security vulnerabilities disclosed. In our last post, we put in automatic checks around performance of the site. Now, let's do something to detect Javascript library vulnerabilities.

The Github Action

I found this project and whipped up the changes necessary to turn it into a Github Action. Thanks Liran! Thanks too to Snyk, which provides the vulnerability list.

I adapted an existing Docker container, wrapping a Github action around it. One way to do this without overly leaking the Github Actions contract in to the container design is to map Github action parameters to environment variables and args that are agnostic and already expected by the container like so:

env:
     SCAN_URL: ${{ inputs.scan-url }}

... where inputs.scan-url comes from the Github Action contract (as a 'parameter') and 'SCAN_URL' is an environment variable that already works with the existing Docker container. This is in contrast to having the container need to understand how to look for 'INPUT_' prefixed vars that a Github Action provides by default (ref: github docs). If you don't want to, or can't modify the existing docker container, this is an option.

The container does still need to exit with an error to cause a Github Action to fail though. I had to modify the underlying Javascript code to accommodate that. There would not be much value in an automatic check that always passes.

Now that we have the action, we use it by creating a workflow file in .github/workflows/javascript_vulnerability_check.yml :

name: Test site for publicly known js vulnerabilities

on: 
  push:
    branches:
    - master # Check on every commit to master
  schedule:
    - cron: '0 13 * * 6' # Check once a week regardless of commits
  repository_dispatch:
jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      - name: Testing for public javascript library vulnerabilities 
        uses: mattorb/is-website-vulnerable@github-action_v1 # until PR to original repo is merged
        with:
          scan-url: "https://mattorb.com"

With that in place, we see the following check run after each commit, and once a week for good measure:

Shift security left!

So. Awesome.

Now we have at least have some awareness if any of the following happens:

A change we make introduces a library with a vulnerability
A change introduced by 3rd party dependency introduces a library with a vulnerability
No change is made at all, but a vulnerability is discovered and published for a JS library we were already using.

Automatic checks for the win!

Github Action for Google PageSpeed Insights

Matt Smith — Mon, 27 Jan 2020 14:38:20 +0000

photo credit: zbysiu-rodak

Lately, I have been making more broad sweeping changes to this site. I want to ensure that I don't accidentally make a change which slows down the site – especially the homepage. Keeping a web site performant has many benefits including better user experience and affecting search engine rankings.

Google provides a tool call PageSpeed Insights to analyze how quickly a page loads and renders on desktops and mobile devices. According to Google:

PageSpeed Insights analyzes the content of a web page, then generates suggestions to make that page faster.

It also calculates an overall score.

For this site, I want to start by setting a baseline and then regularly measure to make sure I don't do something to drop the score below that baseline performance level. For now, we're only measuring the performance of the homepage.

The Github Action

I found this project which provides a Github Action to run PageSpeed Insights. Thanks Jake!

I wired it up to my blog's Github repo by defining a workflow file .github/workflows/pagespeedinsights.yml:

name: Check Site Performance with PageSpeed Insights

on: 
  push:
    branches:
    - master
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Running Page Speed Insights
        uses: JakePartusch/psi-action@v1
        with:
          url: "https://mattorb.com"

It kicks off after each commit to the repo. Example results below:

Desktop PageSpeed Insights for mattorb.com

The overall 'performance' desktop score of 99 out of 100 is really good. Nice! I can't take credit for that. The hard work of the Ghost team and the contributors to the default 'Casper' theme put us there.

Now, let us iterate and see what we put together!

See the #comments in the snippets below, which mark what is changing.

Raise the threshold

name: Check Site Performance with PageSpeed Insights

on: 
  push:
    branches:
    - master
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Running Page Speed Insights
        uses: JakePartusch/psi-action@v1
        with:
          url: "https://mattorb.com"
          threshold: 96 # <--- default was 70

This fails the build if if the overall performance score drops below this number. I'm leaving a tiny bit a wiggle room here for volatility in server response times.

Measure after each blog post

...
on: 
  push:
    branches:
    - master
  repository_dispatch: # <-- External event hook
...

We have our Ghost based blog wired up to post a Github repository_dispatch event to trigger a Github action for new/updated posts. See the end of this post for more detail. This is relevant when you are not doing static site generation or have a content management system that manifests changes into a page. In those cases, you need something event based to retrigger an assessment after the site content changes.

Measure once a week

Our web site has some external dependencies and the rules we are measuring against that can evolve outside our immediate control. To catch any unexpected changes in those things, do this once a week, even if we change nothing on our site.

...
on: 
  push:
    branches:
    - master
  schedule:
    - cron: '0 13 * * 6' # <---- once a week
  repository_dispatch:
...

Hat tip to crontab.guru re: 0 13 * * 6

Measure both desktop and mobile page speed

...
    steps:
      - name: Running Page Speed Insights (Mobile) # <-- new step
        uses: JakePartusch/psi-action@v1
        id: psi_mobile                     
        with:
          url: "https://mattorb.com"
          threshold: 90 # <-- distinct threshold
          strategy: mobile # <-- different strategy
      - name: Running Page Speed Insights (Desktop)
        uses: JakePartusch/psi-action@v1
        id: psi_desktop                    
        with:
          url: "https://mattorb.com"
          threshold: 96
          strategy: desktop               
...

If you use an action with the same 'uses' clause, twice, you have to give each step a unique 'id'.

Wrapping up

Here's the workflow file for my Github Action, in full:

name: Check Site Performance with Page Speed Insights

on: 
  push:
    branches:
    - master
  schedule:
    - cron: '0 13 * * 6'
  repository_dispatch:
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Running Page Speed Insights (Mobile)
        uses: mattorb/psi-action@v1
        id: psi_mobile
        with:
          url: "https://mattorb.com"
          threshold: 90
          strategy: mobile
      - name: Running Page Speed Insights (Desktop)
        uses: mattorb/psi-action@v1
        id: psi_desktop
        with:
          url: "https://mattorb.com"
          threshold: 96
          strategy: desktop

For every commit, every blog post, every theme change, and once a week a mobile and desktop Page Speed Assessment measures how the mattorb.com home page performs and alerts us if something degrades significantly.

We also get history of the results of the performance measurements stored in the actions tab on Github, which can be super helpful in understanding which of the several things that is measured has degraded in performance.

When you browse to examine those results you can see this level of detail:

Mobile Page Speed Insights for mattorb.com

Desktop Page Speed Insights for mattorb.com

Fun stuff! Some potential improvements lie in the 'Opportunities' section of the report and making something run this test on several (or all) pages throughout the site.

Performance is one of those things that is a lot easier to keep going once you have a baseline established and some tools to help track and understand what causes changes in it.

CI your MacOS dotfiles with GitHub Actions!

Matt Smith — Mon, 02 Dec 2019 06:00:00 +0000

photo credit: goran-ivos

Dependencies that you can't or won't pin versions of in a reliable persistent cache, have the potential for drift. Such is the life of my dotfiles repo. I don't want to pin most things there. It is an outlier relative to what you might normally set up to produce the most consistent, reliable build process. For dotfiles, I want to lean way in towards making sure they stay current even if it breaks once in a while in exchange for that posture.

One of the challenges of making sure a dotfiles repo continues to function correctly on a clean machine is that running through them for the first time, in a clean environment, is a rare activity. It generally only happens when I am configuring a new laptop or others are installing it for the first time.

If I could do a clean install on a more regular basis, it would decrease the chance that little breakages are piling up over time. This is where a most basic form of continuous integration comes in.

I am starting with a sanity test . . .

Can the install script execute, start to finish, without error?

I have already seen this break when a dependency is renamed or when I introduce a subtle syntax error that someone finds in a fork when they try to run everything clean for the first time. I want to catch those issues as quickly as possible.

GitHub Actions for MacOS dotfiles CI

When perusing the GitHub Actions documentation, I noticed they included the ability to run an action on a MacOS VM!

wat.

For free.

wat.

For open source repositories.

oh, yah ok.

Still . . . In case you are not aware, getting a hosted MacOS VM, while possible, is generally not cheap relative to other server options. The fact that Github is offering [free] to run Mac workloads for opensource projects is a cause for celebration.

Ok. Let's get to the meat of it: GitHub Actions are behaviors you define in a yaml file checked in to the repository itself.

To get started I defined a 'Smoke Test CI' workflow in .github/workflows/smoke.yml in my dotfiles repo:

name: Smoke Test CI

on: [push]

jobs:
  build:

    runs-on: macOS-latest

    steps:
    - uses: actions/checkout@v1
    - name: Execute full install
      run: ./setup.sh

Every single push to the dotfiles repo provisions a temporary MacOS VM, checks out the code with git, and executes the full install of the dotfiles (via ./setup.sh).

The results for the builds that are triggered are in the Actions tab for that Repository on Github:

and you can drill in to see the results of each step and the logs:

That's it!

Getting this set up has already caught some bugs.**

_**One caveat: For a small handful of things, I had to detect that I was running inside a GitHub action and skip over them because they required privileges that were locked down or not available on that VM. Prudently, Github provides some default environment variables in the VM when running inside a Github Action. I used those as a clue to skip them. _

Goodbye Broken links: Ghost + Muffet + Github Actions

Matt Smith — Mon, 04 Nov 2019 14:36:06 +0000

photo credit: zbysiu-rodak

How big of let down is it when you are reading a web page, find something interesting enough to click on and are subsequently dropped down the 404 Not Found hole?

Often the maintainer of a site does not even know what is broken – especially for content oriented sites which have a healthy amount of outbound links.

Much like broken and flaky tests, compiler warnings, test coverage, code quality and consistent configuration+tooling across promotion environments, the sooner you establish [Picard voice] "the line must be drawn here", the better off you are. Once you have a set standard of what is acceptable and good visiblity to what is crossing that line, you have a fighting chance of understanding where to invest to maintain that standard or improve it.

For a blog like this one, hyperlinks can break a few different ways. Internal links can be broken from the start due to unforced errors (i.e. - typos) or software upgrades. Links going out to other sites can drift into a broken state due to an external site making changes or being taken offline.

Muffet

I wanted to see how this site was doing and had recently stumbled on Muffet, a supa-fast, open source, Go-based broken link checker.

The first time I ran Muffet, I discovered an embarrassingly long list of broken links here.

Here are some choice examples from a trimmed down version of that first run:

$ muffet https://mattorb.com

96
https://mattorb.com/swift-2-to-5/amp/
97
    404 https://mattorb.com/swift-2-to-5/swift%20half%20open%20range
103
https://mattorb.com/fuzzy-find-github-repository/
104
    404 https://mattorb.com/fuzzy-find-github-repository/GitHubAPIv3%7CGitHubDeveloperGuide
105
    404 https://mattorb.com/fuzzy-find-github-repository/github.com/shurcooL/githubv4

$

How to read this output : By default, Muffet only puts broken links in the output. Hierarchy is expressed through indention: so #97 above is a link that was walked while parsing the page at #96. The unindented number is a counter of links checked, and the indented numbers are HTTP return codes (404 = not found).

All of the link issues above were unforced errors as far as I can tell. Additionally, the stuff I trimmed out included errors for images that had gone missing and links to external sites that were no longer valid.

Awesome! We now have a way to assess the whole site for broken links. The problem is we just found a whole bunch of broken things all at once, which means a whole bunch of work to fix them.

Prefer small fixes right away

Next time a link breaks, I want to be fixing just that one thing and be done -- rather than looking at a large pile of issues that have accumulated over a longer period of time.

Ideally, I want broken links assessed:

Automatically, before publishing new content – to catch unforced errors before they go live
Automatically, on configuration changes and software upgrades – to catch unexpected interactions of new software and existing content
Automatically, on a schedule – to catch drift in the health of links to external party sites
On demand, to confirm I have fixed issues after making changes

Having a place to trigger the workload which executes a broken link checker manually or programmatically, record the results, and notify me when things break would hit all my needs.

After my other recent experiment with a Github Action, that seemed like a good candidate.

A Github Action for Muffet

Always Google first, to see if someone else has already done similar work.

I found an archived Github repo from peaceiris that had a Muffet Github action. I have no idea why he/she archived it, but it seems to work fine, so I forked it to keep a copy.

To use that action from our project [checked in to a Github repo], I added a workflow at .github/workflows/checklinks.yml :

name: checklinks

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Check links on site
      uses: mattorb/actions-muffet@v1.3.1
      with: 
        args: >
          --timeout 20 
          https://mattorb.com

This sets up triggering Muffet to check for broken links on every push to git master. It builds the needed action via a Docker build of the Github repo mattorb/actions-muffet, tag 1.3.1.

As noted earlier, sometimes external links go bad due to changes outside our awareness, so below we add a schedule stanza to trigger this check regularly as well:

name: checklinks

on: 
  push:
    branches:
    - master
  schedule:
    - cron: '0 13 * * 6'

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Check links on site
      uses: mattorb/actions-muffet@v1.3.1
      with: 
        args: >
          --timeout 20 
          https://mattorb.com

Now, in addition to executing on pushes to master, that cron schedule sets this check to happen automatically once a week. ('0 13 * * 6' == 1pm on Saturdays)

When it fails, Github sends you an e-mail. (default settings)

Also, this handy badge can be placed at the top of README.md in the git repo, or on the site itself:

That badge is live, so hopefully it reads 'passing' when you are reading this article! For how to make one, see the Github docs.

At this point, we have a Github Action in place that will be kicked off for a few scenarios:

Manually triggered via the Github web UI
Automatically triggered via the a push the git repo master
Automatically triggered once a week

For all the ways you can trigger Github Actions, see here.

Going Further

One of those GitHub Action triggers that I'm particularly interested in, since my blog workflow has not moved over to a static generation approach yet: repository_dispatch. It is still in developer preview but offers a way to trigger a Github Action workflow based on an external events.

Ghost has webhooks that can be triggered for various types of modifications:

Tying one or more of those to triggering the Github Action via a repository_dispatch event will require building something to either receive the Ghost JSON webhook Payload and post the Github expected JSON payload, or extending Ghost itself with a custom webhook integration for repository dispatch – a small future project.

Here is a quick stab at that in Go. I point Ghost webhooks at it for the 'New post published' and 'Published post updated' events to trigger broken link checking on those two events. It is a bit naive in that it scans the whole site every time a new post is published.

Site Reliability Engineering Book Trio

Matt Smith — Mon, 16 Sep 2019 13:37:40 +0000

photo credit: timothy-muza

What works for Google, what works for Facebook, and what works for Netflix may not be the right thing for the rest of us. Putting too much weight behind the opinions of a few large organizations can bite you. The same is true of charting a path forward based on the experience of a few individuals, without being aware of the broader landscape. This is why I'm a huge fan of studies that are more broadly based, like Accelerate and the accompanying yearly State of DevOps reports. Do what works in your context, but stay informed of what is working well for others, to make better and better choices as you go.

One particular area that has been getting more refined is Site Reliabilty Engineering. There are three great books I read over the last year that provide a peak into some experiences and experiments.

This trio of books is a treasure trove of ideas, techniques, practices, and organizational approaches for improving both the delivery of value to production, and how the teams around that are organized:

SRE is all about applying a software development mindset to infrastructure and operations.

I particularly enjoyed 'Seeking SRE' which is a series of essays. Each chapter stands on its own, and several are based on years of history and experience reports at well known companies.