Automated visual regression testing with TypeScript, Playwright, Jest and Jest Image Snapshot

Introduction

As you know, to have automated visual regression tests (in addition to unit and integration tests) is important to ensure the users a good experience.

This assures that nothing is accidentally broken and confirms that user flows and the appearance of your application are functional when the existing code is modified or a dependency is updated.

Recently, I wrote an article on visual regression testing with TypeScript, Jest, Jest Image Snapshot and Puppeteer.

If you use Puppeteer, you know that you can test only under Chromium.
The implementation with Firefox is experimental, and Webkit is not supported at all.

Today, I will offer you an alternative to Puppeteer: Playwright
Playwright is pretty similar to Puppeteer. All the features present on Puppeteer are not yet present on Playwright. But the advantage with this tool is that you can run your tests on Chromium, Firefox, Webkit, Google Chrome & Microsoft Edge, and in addition with a single command.

In this article, I will explain how to set up and to run tests with Playwright, in a TypeScript project, using Jest - as the test runner - and Jest-Image-Snapshot (Jest matcher) - to take screenshots of current web pages and to compare generated screenshots with a screenshot baseline to find regressions in the user interface.

If you read my previous article, it will be pretty similar.

Project example

I will use the bpmn-visualization TypeScript library (version 0.13.0) as an example. (This example has been simplified so it shows more clearly the configuration and features explained in this article.) The goal of this project is to load BPMN content, and render it.
Automated visual tests will simplify our life with each refactoring, addition of a new component, update of the positioning algorithm of the different BPMN elements, or update of the MxGraph rendering library.
Also, I will configure the tests to run on Chromium, Firefox and Webkit.

Prerequisites

As first step, we need to install the required packages as
devDependencies:

• Jest + its type definition: A JavaScript Testing Framework

Jest is a fully featured testing framework, developed by Facebook. It needs very little configuration and works basically out of the box.

npm install -D jest @types/jest

• Playwright: A Node.js library to automate Chromium, Firefox,Webkit, Google Chrome & Microsoft Edge with a single API. It is possible to perform most of the actions that are done manually on a browser and take screenshots.

npm install -D playwright jest-playwright-preset

• Jest-Image-Snapshot + its type definition: A Jest matcher to perform image comparisons

npm i -D jest-image-snapshot @types/jest-image-snapshot

Configuration

Let’s configure the previous libraries.

Configure Jest

I won't go into detail here on all the different ways to configure Jest.
If you already use Jest for your unit/e2e tests, this is not new for you. If you would like more explanations about Jest, there are many great articles available.

In this example, we have 4 Jest configurations: unit tests, integration tests, e2e tests and performance tests. We added the visual tests in the e2e test suite. Here, I will just explain how we configure Jest for the e2e tests.

First, create the Jest configuration file (jest.config.js) at ./test/e2e directory:

	process.env.JEST_PLAYWRIGHT_CONFIG = './test/e2e/jest-playwright.config.js';
	module.exports = {
	rootDir: '../..',
	roots: ['./test/e2e', './src'],
	transform: { '^.+\\.ts?$': 'ts-jest' },
	testMatch: ['**/?(*.)+(spec|test).[t]s'],
	testPathIgnorePatterns: ['/node_modules/', 'dist', 'src'],
	testTimeout: 200000,
	}

view raw jest.config.js hosted with ❤ by GitHub

This configuration sets the root directory to the root project directory, runs .ts files with ts-jest module and looks for .spec.ts and test.ts files under any subdirectory of ./test/e2e directory.

Configure Playwright

• Specify the preset in the Jest configuration (./test/e2e/jest.config.js), as specified in the official documentation of Playwright:
  

  	// …
  	module.exports = {
  	// …
  	preset: 'jest-playwright-preset',
  	}

  view raw jest.config.js hosted with ❤ by GitHub

• Create a new file (./test/e2e/jest-playwright.config.js) for the Playwright configuration to run the server & launch the browser once for all tests:
  

  	module.exports = {
  	server: {
  	// How you build your bundle.
  	// If you use Rollup, add the plugin rollup-plugin-serve with the configuration serve({ contentBase: 'dist', port: 10002 })
  	command: `npm run start`,
  	port: 10002,
  	// If default or tcp, the test starts right await whereas the dev server is not available on http
  	protocol: 'http',
  	// In ms
  	launchTimeout: 30000,
  	debug: true,
  	usedPortAction: 'ignore', // your test are executed, we assume that the server is already started
  	},
  	launch: {
  	headless: process.env.HEADLESS !== 'false',
  	slowMo: process.env.SLOWMO ? process.env.SLOWMO : 0,
  	},
  	launchType: 'LAUNCH',
  	contextOptions: {
  	viewport: {
  	width: 800,
  	height: 600,
  	},
  	},
  	browsers: ['chromium', 'firefox', 'webkit'],
  	};

  view raw jest-playwright.config.js hosted with ❤ by GitHub

With this configuration, we start a server on the port 10002 with a timeout of 30s, and start 3 browsers.

Extend the Jest expect assertion mechanism to use Jest Image Snapshot

This is the part that might be new, but with a little configuration, we will be ready soon.

By default, Jest doesn’t know anything about Jest-Image-Snapshot and its assertion toMatchImageSnapshot. So we’ll need to extend Jest. For that, create a new file (./test/e2e/jest.image.ts), like the following:

	import { toMatchImageSnapshot } from 'jest-image-snapshot';
	expect.extend({ toMatchImageSnapshot });

view raw jest.image.ts hosted with ❤ by GitHub

To avoid extending Jest in each test file or import the previous file globally in all test files, we need to configure Jest to run it immediately after the test framework has been installed in the environment with setupFilesAfterEnv (Jest property) in ./test/e2e/jest.config.js.

	module.exports = {
	// ...
	setupFilesAfterEnv: ['./test/e2e/jest.image.ts'],
	}

view raw jest.config.js hosted with ❤ by GitHub

Add a new command

To simply the test execution, add the following script in the package.json file:

	{
	//...
	"scripts": {
	//...
	"test:e2e": "cross-env DEBUG=bv:test:* jest - runInBand - detectOpenHandles - config=./test/e2e/jest.config.js",
	}
	}

view raw package.json hosted with ❤ by GitHub

Now, you can run your e2e tests with the following command:
npm run test:e2e

Note: cross-env is useful if you run the tests on different OS.

Test

You can find the different properties to customize Jest-Image-Snapshot in its README on Github.

Create a basic test with Jest-Image-Snapshot

If everything is configured correctly, we are now ready to create our first visual regression test (./test/e2e/bpmn.rendering.test.ts) by combining Playwright and Jest and Jest-Image-Snapshot!

	// jest-image-snapshot custom configuration
	function getConfig () {
	return {
	diffDirection: 'vertical',
	// useful on CI (no need to retrieve the diff image, copy/paste image content from logs)
	dumpDiffToConsole: true,
	// use SSIM to limit false positive
	// https://github.com/americanexpress/jest-image-snapshot#recommendations-when-using-ssim-comparison
	comparisonMethod: 'ssim',
	};
	}
	it(`no BPMN Gateway visual regression`, async () => {
	// Redirect the current page in the browser to a new url with puppeteer
	const response = await page.goto(‘http://localhost:10002/non-regression.html?bpmn=./gateways.bpmn’);
	// Be sure the page is displayed correctly with puppeteer & Jest
	expect(response.status()).toBe(200);
	await expect(page.title()).resolves.toMatch('BPMN Visualization Non Regression' );
	await page.waitForSelector(‘#bpmn-container’, { timeout: 5_000 });
	// Take the screenshot of the page with puppeteer
	const image = await page.screenshot({ fullPage: true });
	// Compare the taken screenshot with the baseline screenshot (if exists), or create it (else)
	const config = getConfig();
	expect(image).toMatchImageSnapshot(config);
	});

view raw bpmn.rendering.test.ts hosted with ❤ by GitHub

After the test runs, a new directory will be created -
__image_snapshots__ - with an image for each toMatchImageSnapshot call. The name of the snapshot is computed by default with testPath, currentTestName, counter and defaultIdentifier.

Example of generated snapshot:

bpmn-rendering-test-ts-no-bpmn-gateway-visual-regression-1-snap.png

Note: Make sure that the snapshot files are committed in your source control so that they are shared with other developers and CI environments.

Test on different machines

One issue with one-to-one pixel matching is that there is a good chance that the test will be in error on a machine other than on which it was developed, because every environment has slightly different ways of rendering the same application.

For example, suppose that we want to run the tests on the CI environment every time we create a pull request to the master branch in GitHub.
Without any modifications to the code, the test is passed locally; but on the CI environment, it fails with a message like this:

Error: Expected image to match or be a close match to snapshot but was 0.0005804554357724534% different from snapshot (2.7861860917077763 differing pixels).

And a new image file for the diff is stored in the
__image_snapshots__/__diff_output__ directory with the name <snapshot_name>-diff.png.

If we use the previous test, we’ll have something like this:

bpmn-rendering-test-ts-no-bpmn-gateway-visual-regression-1-diff.png

You can modify the previous jest-image-snapshot configuration (./test/e2e/bpmn.rendering.test.ts), and update the value of failureThreshold (default value: 0) & failureThresholdType (default value: pixel). These properties are used to calculate the threshold of tolerated differences (before the test fails).

	// jest-image-snapshot custom configuration
	function getConfig () {
	return {
	// …,
	// anything less than 0.5 percent difference passes as the same
	failureThreshold: 0.005,
	failureThresholdType: 'percent',
	};
	}
	// ...

view raw bpmn.rendering.test.ts hosted with ❤ by GitHub

Warning: If you increase the failure threshold too much, when there is too much difference between local and CI environments, it may be impossible to detect visual regressions.

Order the snapshots

If you have 10 or more tests, it can become complicated to find which screenshot corresponds to which test / feature in the directory __image_snapshots__.

Modify the customSnapshotsDir property to have a different value according to the tests.

• ./test/e2e/helpers/image-snapshot-config.ts
  

  	// jest-image-snapshot custom configuration
  	export function getConfig (customSnapshotsDir: string) {
  	return {
  	// …,
  	// custom absolute path of a directory to keep this snapshot in
  	customSnapshotsDir,
  	};
  	}
  	export function async gotoPageWithBPMNContainer(): Promise<ElementHandle<Element>> {
  	// Redirect the current page in the browser to a new url with puppeteer
  	const response = await page.goto(‘http://localhost:10002/non-regression.html?bpmn=./gateways.bpmn’);
  	// Be sure the page is displayed correctly with puppeteer & Jest
  	expect(response.status()).toBe(200);
  	await expect(page.title()).resolves.toMatch('BPMN Visualization Non Regression' );
  	return await page.waitForSelector(‘#bpmn-container’, { timeout: 5_000 });
  	}

  view raw image-snapshot-config.ts hosted with ❤ by GitHub

• ./test/e2e/bpmn.rendering.test.ts
  

  	import { gotoPageWithBPMNContainer, getConfig } from './helpers/image-snapshot-config';
  	it(`no BPMN Gateway visual regression`, async () => {
  	await gotoPageWithBPMNContainer();
  	// Take the screenshot of the page with Playwright
  	const image = await page.screenshot({ fullPage: true });
  	// Compare the taken screenshot with the baseline screenshot (if exists), or create it (else)
  	const config = getConfig('__image_snapshots__/bpmn');
  	expect(image).toMatchImageSnapshot(config);
  	});

  view raw bpmn.rendering.test.ts hosted with ❤ by GitHub

• ./test/e2e/bpmn.navigation.test.ts
  

  	import { gotoPageWithBPMNContainer, getConfig } from './helpers/image-snapshot-config';
  	it(`no visual regression for drag & drop with the mouse of the BPMN content in the container`, async () => {
  	const bpmnContainerElement = await gotoPageWithBPMNContainer();
  	const bounding_box = await bpmnContainerElement.boundingBox();
  	// Move the mouse pointer to the center of the BPMN Container
  	const containerCenterX = bounding_box.x + bounding_box.width / 2;
  	const containerCenterY = bounding_box.y + bounding_box.height / 2;
  	await page.mouse.move(containerCenterX, containerCenterY);
  	// Drag & Drop the BPMN content in the BPMN Container
  	await page.mouse.down();
  	await page.mouse.move(containerCenterX + 150, containerCenterY + 40);
  	await page.mouse.up();
  	// Take the screenshot of the page with Playwright
  	const image = await page.screenshot({ fullPage: true });
  	// Compare the taken screenshot with the baseline screenshot (if exists), or create it (else)
  	const config = getConfig('__image_snapshots__/navigation');
  	expect(image).toMatchImageSnapshot(config);
  	});

  view raw bpmn.navigation.test.ts hosted with ❤ by GitHub

Reuse the snapshots

Sometimes the expected result/snapshot is the same even after different actions. To avoid having a lot of identical snapshots in the Github repository, it’s better to reuse a snapshot.

For that, it’s necessary to override the default customSnapshotIdentifier & customDiffDir properties.

• customSnapshotIdentifier: the custom name to give this snapshot. This prevents the name of the snapshots from being computed with testPath, currentTestName, counter and defaultIdentifier.

• customDiffDir: the custom absolute path of a directory to keep this diff in. As we use the same snapshot in different tests, to know which diff file corresponds to which test, we need to set a different value according to the tests.

Example

• ./test/e2e/helpers/image-snapshot-config.ts
  

  	// jest-image-snapshot custom configuration
  	export function getConfig (customSnapshotIdentifier: string, customDiffDir: string) {
  	return {
  	// …,
  	customSnapshotIdentifier,
  	customDiffDir
  	};
  	}
  	// …

  view raw image-snapshot-config.ts hosted with ❤ by GitHub

• ./test/e2e/diagram.rendering.test.ts
  

  	import { getConfig } from './helpers/image-snapshot-config';
  	function async gotoPageWithBPMNContainer(margin: number): Promise<ElementHandle<Element>> {
  	// Redirect the current page in the browser to a new url with Playwright
  	const response = await page.goto('http://localhost:10002/rendering-diagram.html?bpmn=./gateways.bpmn&fitMargin=${margin}');
  	// Be sure the page is displayed correctly with Playwright & Jest
  	expect(response.status()).toBe(200);
  	await expect(page.title()).resolves.toMatch('BPMN Visualization - Diagram Rendering' );
  	return await page.waitForSelector('#bpmn-container', { timeout: 5_000 });
  	}
  	it.each([-100, 0, null])('load with margin %s', async (margin: number) => {
  	const bpmnContainerElement = await gotoPageWithBPMNContainer(margin);
  	// Take the screenshot of the page with Playwright
  	const image = await page.screenshot({ fullPage: true });
  	// Compare the taken screenshot with the baseline screenshot (if exists), or create it (else)
  	const config = getConfig('load_with_no_margin',`__image_snapshots__/__diff_output__/${margin}`);
  	expect(image).toMatchImageSnapshot(config);
  	});

  view raw diagram.rendering.test.ts hosted with ❤ by GitHub

Conclusion

With so many operating systems, web browsers, and screen resolutions, Visual Testing can be a powerful tool to assure that an application works well in all possible environments. It is definitely worth trying it as a complement to other sets of tests.

Now, you have everything you need to start your first visual regression test in TypeScript with Jest & Playwright.

Thank you for reading and I hope I helped or inspired you :)

References

• Github repository of bpmn-visualization: https://github.com/process-analytics/bpmn-visualization-js

• Github repository of Jest-image-snapshot: https://github.com/americanexpress/jest-image-snapshot

• Playwright documentation: https://playwrightdev/

• Main image made with: https://www.canva.com