Why visual regression testing
Your app code changes quite frequently - so does the UI. The problem with this flow is that sometimes unintentional UI changes result from incorrect code. We often forget to check the deep corners of the application that users might see for every potential regression; moreover, we don't have time for that, right 😅 Ultimately as with other testing types the goal is to ship with more confidence and o*wn the changes you make to the code*!
| Luckily, we can quickly establish automation that can help us with that
In this tutorial we will learn how to:
- setup a new next.js app
- use lost-pixel to enable CI visual regression testing
At this time, you might wonder what visual regression tests even are.
On top are two snapshots taken before/after a code change, below is how the engine compares those two to identify the visual regression.
Without further ado, let's jump into the code, and don't worry, as we will delve into some low-level dev-ops you don't need any specific knowledge, we will figure everything on the fly.
For the final code you could check out this repo
Next setup
This tutorial covers the most basic Next.js app, so you can follow official docs for the simple instructions for setting it up. First, let's add the *app page * and move the content from the index there so we have a clear path we will test!
Adding Lost Pixel
So far, our setup has the frontend application & the tooling to observe the frontend components/pages. However, to have complete control over the changes to the code and their repercussions on the UI, we use an open-source visual regression testing tool called Lost Pixel.
To set up the first tests, you need to add the lost-pixel.config.js
or lost-pixel.config.ts
at the root of your project by running one of the below commands:
npx lost-pixel init-js
or
npx lost-pixel init-ts
Don't forget to include
lostpixel.config.ts
file into yourtsconfig
when using typescript version of config so it's picked up correctly.npm install lost-pixel -D
is as well required for the types.
This will generate the config that powers Lost Pixel. As we are setting up the open-source version which persists the snapshots of your components in your repo, you need to add the generateOnly
key into your config - this will ensure that lost-pixel will not try to upload the images to the cloud, failOnDifference
flag will exit the GitHub action with the error. Here is how your config should look like:
import { CustomProjectConfig } from 'lost-pixel';
export const config: CustomProjectConfig = {
pageShots: {
pages: [
{ path: '/app', name: 'app', id: 'app' },
],
pageUrl: "http://172.17.0.1:3000",
},
generateOnly: true,
failOnDifference: true
};
Note the
172.17.0.1
IP address, which is specific to the GitHub actions run.
GitHub actions setup
With our current setup, it's pretty straightforward to enable the visual-regression testing on the CI. We will use GitHub actions that have native integration with lost-pixel for this example. Still, lost-pixel also being a docker container, it's possible to integrate it into the CI platform of your choice.
Let's get to business!
In .github/workflows/test.yml
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Setup Node
uses: actions/setup-node@v3
with:
node-version: 16.x
cache: 'npm'
- name: Build Next app
run: npm run build
- name: Run Next app
run: npm run start &
- name: Lost Pixel
uses: lost-pixel/lost-pixel@v2.15.0
Let's see what this workflow file does in a little more detail:
on: [push] // we want to execute this action on every push to our repository
jobs:
build: // we set up one build job
runs-on: ubuntu-latest // it runs on ubuntu
steps: // it will execute the steps outlined below
- name: Checkout
uses: actions/checkout@v2 // get the code of our repo
- name: Setup Node
uses: actions/setup-node@v2
with:
node-version: 16.x
cache: 'npm' // get node to work in our process
- name: Install dependencies
run: npm ci // install dependencies defined in package.json
- name: Build Next app
run: npm run build // build next.js application
- name: Run Next app
run: npm run start & // run next.js application in the background mode
- name: Lost Pixel
uses: lost-pixel/lost-pixel@v2.15.0 // run lost-pixel visual regression tests
As you commit this code, we will see that GitHub has already picked the job and is executing the flow!
We don't yet have anything in .lostpixel
baselines folder which means the action will fail and notify us that there are missing baselines. You can learn more about baselines and flow to update them here. So let's add another workflow to make sure we always have an easy way to update baselines!
In .github/workflows/test.yml
on: push
jobs:
lost-pixel-update:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Setup Node
uses: actions/setup-node@v3
with:
node-version: "16"
- name: Install dependencies
run: npm ci --legacy-peer-deps
- name: Build Next app
run: npm run build
- name: Run Next app
run: npm run start &
- name: Lost Pixel
uses: lost-pixel/lost-pixel@v2.15.0
env:
LOST_PIXEL_MODE: update
- name: Create Pull Request
uses: peter-evans/create-pull-request@v4
if: ${{ failure() && steps.lp.conclusion == 'failure' }}
with:
token: ${{ secrets.GH_TOKEN }}
commit-message: update lost-pixel baselines
delete-branch: true
branch: "lost-pixel-update/${{ github.ref_name }}"
title: "Lost Pixel update - ${{ github.ref_name }}"
body: Automated baseline update PR created by Lost Pixel
You must have already guessed that this workflow file is not much different from the previous one with one little catch - it will automatically create a new PR. Don't forget to add the update
mode env var so we have our baselines updated!
- name: Lost Pixel
uses: lost-pixel/lost-pixel@v2.11.0
env:
LOST_PIXEL_MODE: update // <--- this one here is really important for updater action to run
For this exercise we need a personal access token from github. To ensure that the action has access to the GitHub token which you, by the way, should never expose to the public - we will place it in the secrets of the repo.
Let's run the updater action:
If there are no baselines it will prompt a PR creating the Lost Pixel baselines, if there are existing baselines and there were some differences found it will as well prompt a new PR. In this case we see that there is one open PR with the new baseline. Let's verify it looks good and merge the pull request!
After PR merge we can see that Lost Pixel has ran the tests and all of them are passing 🍏🍏🍏
Amazing job, you now have visual regression testing running on your CI/CD pipeline 🚀
Closing thoughts
Whenever the Lost Pixel fails on CI, you need to decide if the change was intentional or not, which you could read up here. If it has been a deliberate change - you need to run update your baselines on CI. Another strategy could be generating the artefacts from the failed run and putting those manually into .lostpixel/baselines
folder!
The implementation is finished, you could try making some changes to your /app
page, like changing fonts, layout and color - as we have visual regression tests setup, we will be able to catch all the changes and ship with more confidence.
For the final code you could check out this repo as previously mentioned!
Top comments (0)