Accessibility audits with Playwright, Axe, and GitHub Actions

I recently given a talk at JS Poland Conference about one of my most favourite topics -> Improving Software Quality by utilising Continuous Integration. We usually use CI to lint the code, run unit tests, or check if our project was build correctly. While certainly important, these things are rather simple and we can use Continuous Integration for much more complex tests.

In this artice, I will focus about using Playwright and Axe builder, triggered automatically in GitHub Actions CI to continuously audit Accessibility of our application.

Enjoy!

🟢 How to use Playwright and Axe in CI?

Running Accessibility tests in Playwright does not differ that much from doing the normal E2E tests.

First, we need to instruct the Playwright to run the localhost application server before running the tests:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  webServer: {
    command: 'npm run start',
    url: 'http://127.0.0.1:3000',
    reuseExistingServer: !process.env.CI,
  },
});

Then, we can start writing our first A11Y test with Playwright:

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright'; // 1

test.describe('homepage', () => { // 2
  test('should not have any automatically detectable accessibility issues', async ({ page }) => {
    await page.goto('https://your-site.com/'); // 3

    const accessibilityScanResults = await new AxeBuilder({ page }).analyze(); // 4

    expect(accessibilityScanResults.violations).toEqual([]); // 5
  });
});

Let's stop for a minute and explain each of the steps marked with numbers individually:

1. Imports the @axe-core/playwright package
2. Uses normal Playwright Test syntax to define a test case
3. Uses normal Playwright syntax to navigate to the page under test
4. Awaits AxeBuilder.analyze() to run the accessibility scan against the page
5. Uses normal Playwright Test assertions to verify that there are no violations in the returned scan results

Apart from using the default preset of Axe, we can also customize it to check for certain WCAG violcations like A or AA like following:

test('should not have any automatically detectable WCAG A or AA violations', async ({ page }) => {
  await page.goto('https://your-site.com/');

  const accessibilityScanResults = await new AxeBuilder({ page })
      .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
      .analyze();

  expect(accessibilityScanResults.violations).toEqual([]);
});

If we run this test locally, we should see similar result:

If you would like to learn more about accessibility testing in playwright, check out here.

Now finally, the last step is to add the logic of running this test to CI (which we probably already have if we are using Playwright):

name: Playwright Tests
on:
  push:
    branches: main
jobs:
  test:
    //
    - name: Install dependencies
      run: npm ci
    - name: Install Playwright Browsers
      run: npx playwright install --with-deps
    - name: Run Playwright tests
      run: npx playwright test

And that's it! You have now implemented Accessibility audits in your Continuous Integration pipeline. Nicely done!

📖 Learn more

If you would like to learn more about Vue, Nuxt, JavaScript or other useful technologies, checkout VueSchool by clicking this link or by clicking the image below:

It covers most important concepts while building modern Vue or Nuxt applications that can help you in your daily work or side projects 😉

✅ Summary

Well done! You have just learned how to implement Accessibility audits in your CI Pipeline.

Take care and see you next time!

And happy coding as always 🖥️

Comments

[Bruno]

Pretty interesting article, Jakub! 👏🏻 I find testing accessibility within Storybook easier, but I believe this is a bit of how it works under the hood there, maybe? It does look quite useful when you don't want to add yet another tool, and configuration effort (I mean with setting up Storybook from scratch if it is not yet being used in the frontend).

[Jakub Andrzejewski]

Hey Buddy,

Thanks for the comment! I used Storybook as well but the problem I had with it that the a11y plugin was mainly reporting some simple issues while Axe reported much more complex ones from WCAG. Also, I think that Storybook would only test the components while Axe tests the whole functioning website.

My honest recommendation would be to use both if possible. There are never too many accessibility auditing tools :)