DEV Community

Cover image for Automated Accessibility Part 3: Regression Tests
Mark Steadman
Mark Steadman

Posted on

Automated Accessibility Part 3: Regression Tests

Automated libraries such as axe-core and pA11y have been a very seamless way to bring accessibility testing into development teams UI testing. It can get development teams to begin to learn and grow accessibility in their teams. However, one big problem has appeared since the rise in popularity of these libraries.

Using these UI libraries has lead to teams thinking this is "all the automated testing we can do for accessibility" or that this is a "great start, but there isn't anymore". This thought process, has made development teams not go beyond just using those libraries to test.

Did you know that there is more automated testing you can do to help ensure the accessible functionality of the web content you create is accessible? It's true! Lets dive in!

Why A11y Regression Tests?

Automated testing libraries generically test your page or component for about 1/3 of accessibility issues. Yes, it is very simple to setup and make a test case like you can with Cypress-Axe:


  beforeEach(() => {
    cy.visit('https://www.spacejam.com/1996');
    cy.injectAxe();
  })

  it('is an accessible homepage', () => {
    cy.waitFor(500);
    cy.checkA11y();
  })

Enter fullscreen mode Exit fullscreen mode

We can go beyond just checking generically and write specific regression tests that ensure the accessible functionality of the component or page we created.

When we say accessible functionality, we are not talking replacing screen reader testing or a manual audit. It is quite the opposite actually. We are talking about using the results from a manual audit to help build automated tests that ensure that the HTML and JavaScript created are behaving as expected so when assistive technology is used they can be accessible.

Let's take at a couple of example scenarios where we could test the accessible functionality of the content we created!

Writing A11y Regression Tests

Let's say we are a development team who is using Cypress for our UI automated testing. We have two new components that we just wrote up, an expand collapse section and a more information modal.

When building the expand/collapse button, we include the state of the button using aria-expanded. We can test to ensure that there is expanded state before click(with keyboard included), after opening and then closing too!


  it('A11y - FAQ proper expanded state', () => {
      cy.get('#toggle').invoke('attr', 'aria-expanded')
        .should('eq', 'false');

      cy.get('#toggle').trigger('click');
      cy.wait(150);
      cy.get('#toggle').invoke('attr', 'aria-expanded')
        .should('eq', 'true');

      cy.get('#toggle').trigger('click');
      cy.wait(150);
      cy.get('#toggle').invoke('attr', 'aria-expanded')
        .should('eq', 'false');
  });

Enter fullscreen mode Exit fullscreen mode

As simple as that! Now we have tested that ARIA has been properly set for the component!

For our modal component, we need to ensure the focus order is proper. Focus order for modals is one of the most common accessible issues, and the great part is we can write regression test to ensure it properly works for all users!

The focus order on click of the button should go into the modal, in this case on the heading, and then also when it closes back to the button that triggered it.

  it('A11y - Focus order on modal', () => {
    cy.waitFor(1000); 
    cy.get('#modalButton').trigger('click');
    cy.focused().should('have.attr','id').and('eq', 'modalTitle');

    cy.waitFor(500);
    cy.get('#modalClose').trigger('click');
    cy.focused().should('have.attr','id').and('eq', 'modalButton');
  })  
Enter fullscreen mode Exit fullscreen mode

Just like that we have written two accessible automated tests that ensure both the HTML and JavaScript we have written are functioning accessibly!

These are only a couple of examples that are possible, there are many more including:

  • Focus indicator being set on actionable items
  • Custom buttons are keyboard acccessible
  • Disabled button state
  • Testing ARIA states
    • Hidden
    • Pressed
    • Selected
  • Link purpose (ensuring links have different text)
  • Focus Order
    • Skip to main links
    • Modals
    • Page change

The Benefits

One thing to make clear, writing these does NOT end manual testing with screen reader or keyboard. The idea is to be an enhancement for those that:

  • Makes debugging for accessibility issues easier
  • Ensures new code is accessible
  • Speeds up validation
  • Makes A11y apart of acceptance criteria.
  • Builds developers that think a11y first.

In Summary

It is time to take automated accessibility farther than where it's at in the field now. Automated libraries are a great starting point for generically catching issues, but we can build so much more!

Accessibility regression tests can help build a culture where tests MUST include accessibility as part of functional UI testing, and encourage developers to know and understand what it takes to make their components accessible.

The question is, why haven't you started yet?!?

For more information and examples on automated accessibility check out my new library on GitHub that will be updated weekly with new examples: https://github.com/Steady5063/Automated-Accessibility-Example-Lib

Top comments (6)

Collapse
 
priteshusadadiya profile image
Pritesh Usadadiya

[[..Pingback..]]
This article was curated as a part of #78th Issue of Software Testing Notes Newsletter.
Web: softwaretestingnotes.com

Collapse
 
gufu5 profile image
gufu5

In Agile, the focus is on delivering working software in short iterations, which means that regression testing inoxoft.com/blog/regression-testin... must be integrated into each iteration. Regression testing ensures that any new changes to the codebase do not adversely affect existing functionality and that the software continues to function as expected.

Collapse
 
somaalapati profile image
Soma S Alapati

Hi Mark, I know we need to perform functional regression testing as there is an integration between components. Change in one component may impact other integrated components, so regression testing is essential to make sure any new change doesn't break the existing functionality. On the other hand UI components and their attributes are mostly independent from each other. For example aria-expanded attribute scope is limited to expand/collapse button, so my question is what kind of benefit we can get by writing accessibility regression tests to this button or similar components. Can we really get ROI from these a11y regression tests?

Collapse
 
steady5063 profile image
Mark Steadman

Great Question!

The biggest benefit from ROI for development teams is time. The thing I see over and over again is development teams changing their components and behaviors because they do not understand ARIA or the accessible behavior of a component. Then we manual testing happens again from an accessibility team or QA it is broken again.

Adding regression tests ensures that that behavior stays put. It also helps cut the cost of debugging issues with screen reader bugs as well!

Collapse
 
segdeha profile image
Andrew Hedges

Thanks for this article, Mark, good stuff! I'm curious what you've seen in terms of frequency and impact of regressions. In the accessibility programs you've been part of, do regressions happen often? When they do, are they considered high priority to fix?

Collapse
 
steady5063 profile image
Mark Steadman

In my experience, it depends more on the team and less on the organization or program.

In the past I have set the standard we should be writing proper regression tests for accessibility. And truth be told, it takes time. Training, guidance and then the time for the teams to build them out.

However, the teams themselves that build it up, TRULY rock it. And the regressions save a whole lot of issues from going to production.