What we learned by running an accessibility audit of our app

We recently underwent an accessibility audit (thanks to Access42) for our app due to compliance regulations for large companies in France. Initially, we believed we were in good shape because we use modern tools, adhere to basic HTML structures, and avoid unusual ergonomics.

However, we decided to invest effort into this area as we needed to meet a minimum score, and the scoring system is quite stringent: any failure on a specific criterion on any audited page counts as a failure, which can quickly degrade the overall score.

That's when we realized just how poor our accessibility was, how complex the subject is, and how much work lay ahead of us...

Before diving in

As a typical developer, I initially believed that accessibility was merely a technical issue to be resolved. However, I soon realized I was mistaken, and I would like to share our experience before delving into more concrete details.

Get a proper training

It's easy to assume that accessibility is straightforward, but like everything in web development, there's a rich history, numerous quirks, and backward compatibility issues that complicate matters.

Fortunately, four developers and I had the opportunity to participate in a three-day training session, including hands-on testing of our own app. This was incredibly instructive. We were able to experiment with tools and ask questions on the spot, which was far more efficient than spending hours on Stack Overflow trying to figure out why VoiceOver wasn't behaving as expected...

Involve every stakeholder

A web app should be designed with accessibility in mind from the outset, much like performance and other cross-cutting concerns. While developers can implement fixes, these solutions are often less effective for users and require more effort from the team. Examples include manually adding aria-label attributes to icon-only buttons, creating textual alternatives for complex objects, and managing focus states manually when informations are spread on the interface.

Certain design-specific aspects, such as text size, color contrasts, and considerations for color blindness, cannot be addressed by developers alone. Currently, here, developers are primarily responsible for ensuring the accessibility of the product through testing. However, we hope that a culture of accessibility will gradually permeate all aspects of our work over time.

Understanding the accessibility tree

You might already be familiar with the DOM (Document Object Model) and CSSOM (Cascading Style Sheets Object Model), but we discovered that there is another, less visible underlying model: the accessibility tree.

It is primarily derived from HTML elements and results in four properties for each element:

• name: a kind of identifier
• description: a secondary string describing the element, similar to a tooltip
• role: one of the list of roles. By default, it is generic; a button will be button, an a will be link, and it can be overridden by the role="" attribute.
• state: indicates if the element is currently active, expanded, pressed, checked, etc.

Understanding these properties is crucial for correctly using attributes like aria-label, title, aria-describedby, and so on, as well as knowing their priority. For example, aria-labelledby takes precedence over aria-label. Modern browsers now offer a view of the accessibility tree in their developer tools, making it easier to debug. More on this later.

Subjectivity

One final note before diving in, regarding what may be the most challenging aspect of accessibility: subjectivity.

Here are some scenarios where it can be difficult because no tool can help you make a decision:

• Should this text be a heading or a paragraph?
• An image has an alt attribute, but is it truly meaningful?
• A "skip" button has been added, but is it genuinely useful??
• Should this notification be polite or assertive?

In short, even a 100% accessible website could be frustrating to use with a screen reader. The best way to assess this is to use it ourselves under "real" conditions. The only downside is that it's time-consuming, so I suspect few companies actually do it.

Using accessible librairies

Chances are that you rely on a UI library for common components. You first want to make sure it's already accessible enough, or else you'll lose a huge amount of time trying to fix errors that you don't actually control. In our case, we're migrating from ElementPlus to Radix Vue mainly for this reason.

It seems modern tools finally treat accessibility as a first class citizen, but it was not the case just a few years ago! Ark UI looks like a good alternative too. You can check their docs and even use the tool below to check its accessibility compliance.

We can also refer to Offical WCAG rules for simple and framework-agnostic examples.

Manual testing

Now that we have the basic theory, it's time to get our hands dirty and find issues and test fixes. Here are the main tools we used:

Browser accessibility devtools

Browsers have come a long way in terms of tooling, and it's really helpful to understand the state of a specific element and why it's considered that way. For instance, the aria-label attribute takes precedence over the title attribute, so Chrome indicates that it will be ignored:

It's not as clear in Firefox, but other information is provided:

Each browser offers different tools with their own advantages and disadvantages, so feel free to try them out to find your preferences.

Axe browser extension

Axe is probably the most well-known suite of tools for testing and improving accessibility. It is essentially included in every major testing solution, as we'll see next.

You can find the Axe extensions for browsers on their website (available for Chrome, Firefox, and Edge). It offers a lot of tools to identify accessibility issues:

While it might not be the most feature-complete, as indicated in this study by the British Government, it is probably better to have consistent test results across your tools (unit tests, manual tests, E2E tests, etc.) than ultra-exhaustive error reports.

Windows, Firefox & NVDA

This setup is the most widely used worldwide, and NVDA is the most valid and feature-complete free tool available, so it should be preferred when doing tests. If you're on Mac, you can use a virtual machine (we chose Parallels Desktop with Windows 11, which took about 30 minutes to set up and ~130€ one time payment).

MacOS, Safari & VoiceOver

VoiceOver is included in every Mac and can be toggled by pressing CMD+F5. It's pretty good by default, but it is known to have some quirks. For instance, it won't read aloud any alert that uses anything other than alert (log or status won't work). If you're on Windows, unfortunately, there's no way to emulate it locally.

Using BrowserStack

Online virtual machines can be useful too. We didn't try them, but BrowserStack currently supports screen readers, and there could be other tools offering similar functionality.

Automated Testing

Unit Tests

We believe unit tests are not particularly useful for accessibility, as checking only for HTML attributes is generally insufficient. Tools like Axe can handle this better than manual unit tests, so we didn't use them at all.

End-to-End Accessibility Tests with Cypress

End-to-end tests are better suited for accessibility checks because they closely mimic what users will see and interact with. However, there are some quirks with Cypress, such as handling tab keys to check for focus. You might want to look into cypress-real-events for a potential solution. (Edit: It seems like this issue is finally being considered in Cypress priorities).

I suppose Playwright might not suffer from those issues as it uses a different, more native way of controlling the browser, but we don't use this tool directly for now.

Axe Plugin

We used the cypress-axe plugin to easily integrate Axe into our tests. The upside is that it's straightforward to use; the downside is that reading errors can be challenging (clicking on lines provides better feedback, though):

Cypress Cloud accessibility add-on

Luckily for us, Cypress released a full featured accessibility add-on not so long ago.

It relies on Axe too, but it's much more integrated in the ecosystem. For instance, the online report is very nice (I can't expose the visual of our app unfortunately):

Also, the Slack integration is quite nice too:

The obvious downside is that it's a paid add-on, and it's quite expensive 🥵 so we probably won't be able to keep it in our suite.

Specific Tests

For more specific tests that don't rely on rules covered by Axe, you can add some manual checks. For example:

• The first tab should show and focus a "skip to content" link.
• After opening a dialog, its first input should be focused.
• A notification should be shown once an action is completed (also check its attributes to ensure it will be announced by screen readers).

You can easily find the focused element in Cypress by running cy.focused().

Component Accessibility Tests with Storybook

Storybook proved to be really useful for accessibility testing. It's easier to set up than end-to-end (E2E) tests but still provides visual feedback and debugging tools. It also allows for complex cases where components are used together or under special conditions. You can write specific play functions that use any code or components you need, not just the one being tested.

By the way, Storybook uses Testing Library, which you can think of as an element selector tool with accessibility in mind. For instance, it won't let you retrieve an element by its class name but forces you to check its role or name (remember the accessibility tree we talked about above?). This is because we should test what the user faces, not the underlying code. It might seem like a small detail, but it sets you in the right mindset when testing.

Lastly, Storybook uses Playwright under the hood, so it actually controls the browser instead of creating fake events in JavaScript like Cypress does. This makes it easier to work with native interactions like tabbing, which is huge for accessibility.

Axe Plugin

Once again, Axe is part of the solution with an official accessibility plugin. It provides a checkA11y() function that can be set up to run automatically at the end of every test or manually called in a specific test (using the play() function in Storybook with the addon interactions). Here's a truncated example of what can be done:

// In the config
const config = {
  async preVisit(page) {
    await injectAxe(page);
  },
  async postVisit(page, context) {
    await checkA11y(page, 'body', {
      detailedReport: true,
      detailedReportOptions: {
        html: true,
      },
    });
  },
};

// In the story
export const Default: Story = {
  play: async ({ canvasElement }) => {
    const canvas = within(canvasElement);

    canvas.getByRole('button', { name: 'Show a notification' }).click();

    const notification = canvas.getByRole('alert');
    expect(notification).toBeInTheDocument();
    expect(notification).toHaveTextContent('This is a notification message!');

    // Accessibility tests will be run automatically here
  },
};

Again, it's pretty straightforward to set up. Errors in the interface are clear and actionable as shown below, but it's hard to read in the CI (GitHub Actions in our case) and we didn't spend time to try to improve it for now:

Specific Tests

We especially appreciated Storybook for running our own specific tests. For instance, we had a hard time ensuring that tooltips were used correctly (tooltips are much harder to get right than we initially thought). So, we interacted with the trigger element by focusing it or not and checked that everything was working as expected:

play: async ({ canvasElement }) => {
  const canvas = within(canvasElement);

  const target = canvas.getByText('Hover me to show tooltip');
  // Check tabindex to be sure it's focusable
  await expect(target).toHaveAttribute('tabindex', '0');

  canvasElement.focus();
  await userEvent.tab();
  expect(document.activeElement).toEqual(target);

  const tooltip = canvas.getByRole('tooltip', { name: 'Hello world!' });

  await expect(tooltip).toBeVisible();

  // Check role button to be sure it's read by screen readers
  await expect(target).toHaveAttribute('role', 'button');
},

Another case was ensuring our focus trap always worked within an open dialog. We used the same mechanism to check that we looped through interactive elements without tabbing out.

Linting

ESLint can also help developers avoid simple mistakes early in the development process. We use the ESLint plugin VueJS accessibility, but there are alternatives for other major libraries. While it is the least capable of all the tools we listed, it offers the quickest feedback loop, making it ideal for simple cases.

We had to enforce and create custom rules for our specific use cases to ensure components are used correctly. For instance, our <Tooltip> component should have its interactive element as a direct child, like a <Btn>. We leveraged the vue/no-restricted-syntax rule as follows:

'vue/no-restricted-syntax': [
  {
    selector: 'VElement[rawName="Tooltip"] > VElement:not([rawName=/^(a|router-link|button|Btn)$/]) VElement[rawName=/^(a|router-link|button|Btn)$/]',
    message: 'Interactive elements should be the direct child of a tooltip, or its content will not be read aloud by screen readers',
  }
]

Conclusion

It's virtually impossible to achieve 100% accessibility on a complex web app. Companies that claim otherwise are likely mistaken, whether intentionally or not. Some conduct their own internal audits, which are clearly not as reliable as those performed by an external, independent organization.

We also discovered that there are some difficult and almost undocumented quirks. For instance, a tooltip must be set on an interactive element to be actually announced (e.g., with an implicit or explicit role of button or link; a span would be ignored) and I didn't see documented in common posts and examples. Therefore, the rule is to always test manually and not rely solely on green CI checks for the right attributes.

It must also be said that an audit is outdated as soon as it's done, as nothing prevents the code from changing from that date. In France, an accessibility declaration is only valid for a few years, and it must be rechecked when the website is "substantially" updated, which is quite subjective.

So, the "best effort" technique is the only way to go, and accessibility must be treated with sincerity to be achieved at its best. This involves truly treating it as an acceptance criterion, training developers and designers (and every stakeholder, by the way), having automatic tests, and using manual tools.

Finally, the Accessibility Object Model (AOM) is an experimental JavaScript API that allows interaction with the accessibility tree directly without using attributes, such as button.accessibleNode.expanded = true. This could greatly help developers, as it would eliminate the need to rely on easy-to-break IDs everywhere. However, as everything accessibility related, it might be years before it is widely supported...