DEV Community: Craig Morten

Automating a11y testing: Part 2 - Beyond Axe

Craig Morten — Thu, 20 Jul 2023 09:51:40 +0000

Earlier this year I gave a talk on the topic of automating web accessibility (a11y) testing at the John Lewis Partnership Tech Profession Conference 2023 which I’m delighted to be sharing here in an article series format.

In Part 1 of this series I covered a number of tools from the Axe suite for static analysis of sites to find a11y violations, from framework specific packages to VSCode integrations, and sharing a few learnings with caveats and gotchas from using Axe tools.

However, this is just a small subset of available test automation tools. In this article we will start to explore some of the non-Axe tooling (though Axe will certainly pop up again!) in the wider a11y test automation space. Let’s get going!

Linting for a11y

Although some aspects of web accessibility require manual validation as they are subjective regarding the user experience, many requirements of the Web Content Accessibility Guidelines (WCAG) are for ensuring you use the correct semantic markup and associated attributes to correctly present your content to users. Such requirements are well suited to static analysis checks as they are deterministic and rule based — indeed there is such a set of official rules set out in the Accessibility Conformance Tests (ACT) specification which are used by static analysis tools such as Axe that we covered in Part 1.

"Throughout this series I will be mostly focusing on the automation side of a11y testing, but it is also key to remember the importance of manual validation and exploratory testing" — Key considerations from Part 1 still apply!

Although we can run static analysis checks in CI, as a form of push left and also early feedback, a really natural and effective place to run these checks is in code linting. What is linting if not static analysis of code / markup! We’ve already seen the VSCode Axe Linter in the previous article, let’s look at some others.

Eslint

If you are already plugged into the Eslint ecosystem, there are a number of plugins that you can reach for depending on your framework:

React and other JSX based frameworks — eslint-plugin-jsx-a11y
React Native — eslint-plugin-react-native-a11y
Vue — eslint-plugin-vue-a11y and / or eslint-plugin-vuejs-accessibility
Angular — @angular-eslint/eslint-plugin
Lit Web Components — eslint-plugin-lit-a11y

Mileage may vary with coverage from plugin to plugin, but each comes with the same great developer experience of real time feedback as you write your code as well as all the other awesome stuff you get with using Eslint such as IDE integrations with intellisense, auto-fix, etc.

A warning from eslint-plugin-jsx-a11y in VSCode

Same as your other linting rules, you can also lean into the power of running the linting both locally, whether manually or through git hooks, as well as in CI to ensure you catch issues before you even consider raising that pull request.

Svelte

Those reading closely may have noticed that the Eslint plugin list above (which was by no means exhaustive) was missing the popular framework Svelte.

"SvelteKit strives to provide an accessible platform for your app by default. Svelte’s compile-time accessibility checks will also apply to any SvelteKit application you build." — https://kit.svelte.dev/docs/accessibility

Svelte (and SvelteKit) have a number of accessibility warnings built into the compiler itself which cover a subset of a11y rules. It’s super cool to have these built into the framework itself, but it is worth flagging that these aren’t a catch-all. Even within the limited scope of static analysis checkers, as flagged by one of the Svelte maintainers Geoff Rich, there are a number of gotchas around these compiler checks.

For example, the Svelte compiler is only able to check hard-coded values in markup. If you provide a dynamic value via a variable the Svelte compiler isn’t able to determine the possible values for that attribute and thus doesn’t yeild the desired warning.

<script>
  let href = '#';
</script>

<a href={href}>More Information</a>

In this example the href="#" should normally triggers an a11y warning from the compiler, but using the variable results in the warning being surpressed.

Other

There are a limited number of other linting options out there for a11y, some of which are paid for options:

ember-template-lint — A linter for Ember projects that includes a11y rules.
AccessLint — A paid for (with free trial) GitHub App Integration: "AccessLint brings automated web accessibility testing into your development workflow."
Axe DevTools Linter — In addition to the VSCode plugin covered in Part 1, Axe also have paid for (with free trial) integrations with GitHub Actions, SonarQube, a REST API, and a CLI for their a11y linter.

Visual testing

Let’s pivot over to some ideas around a11y visual testing!

Emulated vision deficiencies

Back in Chrome 83 the Chromium team released a DevTools change that allowed you to emulate different vision deficiencies in the browser. This is accessible from the Chrome DevTools Command Palette (open DevTools and key CMD+SHIFT+P on Mac or CTRL+SHIFT+P for Windows) by searching for "Rendering", where you can then pick from six different emulated vision deficiencies.

Something that I’ve learned recently is that Chrome also boasts a Chrome DevTools Protocol (CDP) that allows you to set almost every feature you get in DevTools via an API. For example, for those using Playwright this example demonstrates how we can set up a CDP session and request to set an emulated blurred vision:

type EmulatedVisionDeficiency =
  | "none"
  | "blurredVision"
  | "reducedContrast"
  | "achromatopsia"
  | "deuteranopia"
  | "protanopia"
  | "tritanopia";

test("visually acceptable when have blurred vision", async ({ page }) => {
  const client = await page.context().newCDPSession(page);

  await client.send("Emulation.setEmulatedVisionDeficiency", {
    type: "blurredVision",
  });

  // ... visual test with Playwright / other third party lib
});

Provided your test framework supports a Chromium based browser then CDP should be available and you can make use of this capability in your based visual tests, e.g. using Playwright screenshots or Cypress image snapshots.

For example, here is emulated achromatopsia (where you can’t perceive colour) for a snapshot test on John Lewis Women’s Dresses product listing page (PLP):

Here’s the same PLP from the blurred vision test:

And finally one from the protanopia (where you can’t perceive Red light) test:

I wouldn’t recommend going overboard with running all of these variations for every test — visual tests are typically heavy and slow by nature, and fragile to browser upgrades, anti-aliasing, and other false positive issues (although this is somewhat mitigated through using third party vendors who typically have intelligent diffing algorithms).

Mileage may vary with third part integrations — if your provider performs the snapshot on the browser instance you’re running (locally or in CI) then this may well be an option, but if they do HTML + CSS snapshotting to recreate the page in different browsers on their servers then this is likely not a goer.

It could be worth considering adding to a couple to your suites for golden path smoke tests. Alternatively, it can be nice to have such tests running in production on a nightly, or weekly, where the snapshots form the basis of design review discussion opposed to a blocking gate in release.

"In the UK, more than 2 million people are living with sight loss. Of these, around 340,000 are registered as blind or partially sighted." — https://www.nhs.uk/conditions/vision-loss/

"Globally, at least 2.2 billion people have a near or distance vision impairment." — https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment

"Colour blindness (colour vision deficiency, or CVD) affects approximately 1 in 12 men (8%) and 1 in 200 women in the world. In Britain this means that there are approximately 3 million colour blind people (about 4.5% of the entire population), most of whom are male. Worldwide, there are approximately 300 million people." — https://www.colourblindawareness.org/

The key is to be aware and empathetic in your design and implementation towards folks who have visual impairments, and understanding of their experience of your site. A few extra, simple tests can help bring colour (pun very much intended) to otherwise dry requirements around font-size and colour contrast which are hard to appreciate in isolation — a quick change to a base font-size in your design system could be a game change for some folks experience on your site.

Automating zoom

Another important aspect of visual accessibility is supporting higher zoom levels, typically of up to 200%. This is laid out in several success criteria in WCAG for text resize and content reflow.

"This Success Criterion helps people with low vision by letting them increase text size in content so that they can read it." — https://www.w3.org/WAI/WCAG21/Understanding/resize-text.html

Unfortunately most major frameworks don’t support APIs to instrument browser zoom at the moment. Searching around you might see references to the --force-device-scale-factor=2.0 device scale factor flag for Chrome, but this relates to pixel density and is not for zoom (can be passed to launchOptions.args array within your framework’s browser setup options if you want to try and see).

This limitation has been flagged for some testing frameworks so we can hope it might be supported in future. For now you can upvote the likes of this Playwright issue for adding an option for browser zoom.

While we wait for frameworks to catch up, the best solution that I’ve found for the time being is to simulate zoom by adding a before hook in your tests which sets the zoom style of the body. For example, this is a snippet from a Playwright visual test in which the a 200% zoom is applied through CSS:

await page.evaluate('document.body.style.zoom=2.0');

In many cases this isn’t necessarily representative of using the native browser zoom feature, nor the OS level zoom features for enlarging text, which is what the WCAG guidance is really for. Nevertheless, it somewhat provides an idea as to how your page behaves when magnified, and can be a useful addition to your golden path test suite to understand if critical functionality still works.

If anything this goes to show how important it is to keep up manual and exploratory testing when it comes to a11y, there is no way to get the coverage otherwise! When it comes to vision deficiencies we are really still quite limited on how much we can shift to automation — some smoke tests on Chrome are great, but what about other browsers? What about tablet and mobile?!

If anyone has any ideas on automation in this space I would love to hear them!

Time to get funky

In this last section I wanted to share one last tool called Funkify.

"Funkify is an extension for Chrome that helps you experience the web and interfaces through the eyes of extreme users with different abilities and disabilities." — https://www.funkify.org/

It is questionably accessibility test automation, as it is a browser extension... but I’ve decided it qualifies as something that automates customer personas to supplement your exploratory testing. It’s also just very cool so worth the share on that premise alone.

Because it is rather good unfortunately (but quite understandably) the maintainers have introduced a premium level around a lot of the functionality. But even if you just have a dabble with the free features I think it’s well worth it for expanding horizons.

The extension works well on most sites, and as you can see, provides a series of simple personas or modes that you can adopt while browsing the site. Everything from vision defects to simulated dyslexia with scrambling letters to trembling hands.

For example, here is the Waitrose groceries landing page with the Dyslexia Dani persona example where we can get a flavour of what the user experience might be like (be wary this is an example simulation, it is not necessarily representative of the experience for all folks with Dyslexia):

One option that really struck home for me is the Trembling Trevor persona which simulates how it might be like to use the site if you suffer from motor degenerative disorders such as Parkinson’s. Unfortunately Medium doesn’t support embedding videos directly to show this off, so I really encourage giving it a go with the free trial at least once! Once you’ve had a play with the extension on desktop, have a think about touch devices — ask yourself how confident are you with your site’s buttons, links, and dropdowns being usable on a small touch screen?

Closing notes

In this article I’ve covered off a few additional tools and techniques for expanding a11y coverage, exploring extensions to the likes of visual testing and how this can supplement manual and exploratory testing.

This article hasn’t been extensive by far, but hopefully a flavour of what tooling is out there. If you’re keen to explore further, these are some awesome sites where you can learn more and find other tools in the a11y automation space:

https://www.a11yproject.com/resources/ — absolute wealth of informationa and resources, from blogs and books to tools and organisations.
https://a11y-automation.dev/automated-tools —a comprehensive list of automated tools for a11y testing.
https://www.w3.org/WAI/ER/tools/ — a list of evaluation tools for web accessibility as curated by W3.

Stay tuned for part 3 of this series where we will take a look at the popular "accessibility first" Testing Library framework and start to explore the emerging field of screen reader automation tooling. See you soon! 👋

Hi, my name is Craig Morten. I am a senior product engineer at the John Lewis Partnership. When I’m not hunched over my laptop I can be found drinking excessive amounts of tea or running around in circles at my local athletics track.

The John Lewis Partnership are hiring across a range of roles. If you love web development and are excited by accessibility, performance, SEO, and all the other awesome tech in web, we would love to hear from you! See our open positions here.

Eliminating CLS when using SSR for viewport specific responsive designs

Craig Morten — Tue, 04 Jul 2023 09:30:06 +0000

A common gotcha when building some responsive sites is that the desired designs for different viewports are sometimes quite different. In many cases, utilising CSS media queries is sufficient to be able to cater for these differences, but there are occasions where this isn’t necessarily sufficient on it’s own. It is also not always a pragmatic approach to refuse to implement such designs!

Photo by KOBU Agency on Unsplash.

CSS only gets you so far

For example, take the following markup for a ficticious article component where on desktop information about the author is contained within a slidedown from the top of the component, and on mobile it is inside a static section at the bottom:

<Article>
  <ArticleAuthorInformationDesktop />
  <ArticleImage />
  <ArticleHeading />
  <ArticlePreviewText />
  <ArticleAuthorInformationMobile />
</Article>

Now if the markup used for the “top” desktop and the “bottom” mobile author information components were exactly the same you could instead opt to make use of CCS grid or the CSS flexbox order to simply have the component once in markup, and use CSS media queries to set the appropriate CSS rule to position the component as desired for the different viewports.

However, this comes with caveats:

“Use of the order property has exactly the same implications for accessibility as changing the direction with flex-direction. Using order changes the order in which items are painted, and the order in which they appear visually. It does not change the sequential navigation order of the items. Therefore if a user is tabbing between the items, they could find themselves jumping around your layout in a very confusing way.” — https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_flexible_box_layout/Ordering_flex_items#the_order_property_and_accessibility

If we were to be clever here and try to achieve the desired differences in layout entirely with CSS, we would be introducing potential accessibility (a11y) issues as the experience for visual and non-visual users would differ.

In theory this can be worked around with either clever use of tabindex and / or aria-owns where in either case you can identify the order in which elements are expected to be traversed — however both cases are almost always strongly advised against. It is risky to try to hijack the default tab order of a page, and aria-owns is not supported by VoiceOver meaning it’s use would only resolve the issue for non fruit related devices. Both options are incredibly fragile to changes to associated components where there is a real maintenance overhead to ensuring this is never regressed.

This also all falls apart if the desktop and mobile versions of the component need to be completely different, e.g. a slidedown vs just a static content section — you can’t use CSS to change HTML markup! This means you will almost certainly be needing different components and then either not rendering or hiding the one that doesn’t fit the viewport.

Server-side rethinkings

This is fine right? Well, it is until you start server-side rendering (SSR) this article component. The server doesn’t have a concept of a viewport out of the box, so there is no way to know which of the two variations will need to be in the HTML response that you’re sending to your users’ browsers.

Photo by Jantine Doornbos on Unsplash.

In the absence of viewport details on the server when rendering, this means:

You can guess (ahem, I mean “adopt mobile-first”), but this will inevitably result in some sort of flash of unstyled content (FOUC) or cumulative layout shift (CLS) whenever you guess wrong, which is a bit naff!
You can choose to not render either server-side, but then you will almost definitely get a CLS when the component is hydrated client-side, or best case it will pop in late which is annoying.
You can choose to render both, but then there’s the same issues as above where one of the variations is there on page load and then get’s removed shortly after either looking visually poor and potentially causing a CLS.

This isn’t necessarily a new nor unsolved problem. Aside from the options above, one idea is to make use of the Client Hints to provide your server with context of where this component will be rendered so an appropriate choice can be made. Alternatively a similar flow can be achieved with “user-agent sniffing” through packages such as ua-parse-js where you detect the user-agent header and from that deduce a likely device type and size.

Be sure to take care that you include the source of the hint in any cache-keys you might have otherwise you might cache poison if you render differently for different viewports!

Lastly, perhaps my preferred solution is (1) to use one of the above hint techniques if possible (and reliable) and then (2) do the following:

If you have a trusted hint then render only what is needed in your HTML and stop here.
Otherwise server-side render both variations of the component in your HTML response.
In addition to this double render, add a small snippet of CSS and appropriate styles / classes to the components with CSS media queries ensuring that client-side they are only displayed if the viewport matches.
Upon hydration you can let JS take over, remove the component from the DOM which isn’t required for the viewport and strip the “SSR smoothing over CSS” attributes as they are no longer required.

If you want to see an example implementation of this idea check out the @artsy/fresnel package (and I’m sure there are others).

How can I do this myself?

It’s not always appropriate to start pulling in more packages to achieve a goal, certainly if you’re keeping performance in mind (with the likes of bundlephobia), and said package may be doing more than you need.

I was in this situation recently — the project was already using react-responsive, a wrapper around the matchMedia API with some server-side options for fallbacks, which is only useful if there is a sensible default (uncommon) or if you are using hints to be able to pass through to the package. This doesn’t do enough as I didn’t have hints to work with and pass to the package, so more work was needed to enable CLS-less SSR.

Here’s the first pass solution I settled on for my use-case (subject to change, and mileage may vary):

// index.tsx

import React, { useEffect, useState } from 'react';
import { useMediaQuery } from 'react-responsive';
import { BREAKPOINTS, BreakpointWidth } from './breakpoints';
import styles from './index.css';

interface SSRMediaQueryProps {
  maxWidth?: BreakpointWidth;
  minWidth?: BreakpointWidth;
}

interface DisplayValueProps extends SSRMediaQueryProps {
  breakpoint: BreakpointWidth;
}

const getDisplayValue = ({ breakpoint, maxWidth, minWidth }: DisplayValueProps) => {
  if (typeof maxWidth === 'undefined' && typeof minWidth === 'undefined') {
    return undefined;
  } else if (typeof minWidth === 'undefined') {
    return breakpoint >= maxWidth! ? 'none' : undefined;
  } else if (typeof maxWidth === 'undefined') {
    return breakpoint < minWidth ? 'none' : undefined;
  }

  return breakpoint < minWidth || breakpoint >= maxWidth ? 'none' : undefined;
};

/**
 * This component serves to allow viewport specific components be server-side
 * rendered without causing a CLS.
 */
export const SSRMediaQuery: React.FC<SSRMediaQueryProps> = ({ children, maxWidth, minWidth }) => {
  const [isClientSide, setIsClientSide] = useState(false);
  const isMatch = useMediaQuery({ maxWidth, minWidth });

  useEffect(() => {
    setIsClientSide(true);

    return () => {
      setIsClientSide(false);
    };
  }, []);

  if (!isClientSide) {
    const style = {
      '--xs': getDisplayValue({ breakpoint: BREAKPOINTS.xs, maxWidth, minWidth }),
      '--sm': getDisplayValue({ breakpoint: BREAKPOINTS.sm, maxWidth, minWidth }),
      '--md': getDisplayValue({ breakpoint: BREAKPOINTS.md, maxWidth, minWidth }),
      '--lg': getDisplayValue({ breakpoint: BREAKPOINTS.lg, maxWidth, minWidth }),
      '--xl': getDisplayValue({ breakpoint: BREAKPOINTS.xl, maxWidth, minWidth }),
    } as React.CSSProperties;

    return (
      <div className={styles.mediaQueryContainer} style={style}>
        {children}
      </div>
    );
  }

  if (isMatch) {
    return <>{children}</>;
  }

  return null;
};

Where the imported CSS file is roughly (I’m using SCSS mixins to avoid writing long-hand!):

/* index.css */

@media (max-width: 543px) {
  .mediaQueryContainer {
    display: var(--xs, unset);
  }
}
@media (min-width: 544px) and (max-width: 767px) {
  .mediaQueryContainer {
    display: var(--sm, unset);
  }
}
@media (min-width: 768px) and (max-width: 991px) {
  .mediaQueryContainer {
    display: var(--md, unset);
  }
}
@media (min-width: 992px) and (max-width: 1199px) {
  .mediaQueryContainer {
    display: var(--lg, unset);
  }
}
@media (min-width: 1200px) {
  .mediaQueryContainer {
    display: var(--xl, unset);
  }
}

And the usage is something like:

<Article>
  <SSRMediaQuery minWidth={BREAKPOINTS.sm}>
    <ArticleAuthorInformationDesktop />
  </SSRMediaQuery>
  <ArticleImage />
  <ArticleHeading />
  <ArticlePreviewText />
  <SSRMediaQuery maxWidth={BREAKPOINTS.sm}>
    <ArticleAuthorInformationMobile />
  </SSRMediaQuery>
</Article>

Walking through the implementation:

We first assume that we are always in a server-side or pre-hydration world until an effect (which only runs client-side) tells us otherwise.
In this server-side world we always render the provided children, and furthermore, do so with them wrapped in a <div> that has appropriate styles associated with it to ensure it is only displayed to users for on the desired viewports. Here I was feeling fancy so made use of CSS variables to determine how the display value is set for the wrapper <div>, defaulting to unset if no value is passed.
Once we hit the client we can be safe in the knowledge that the CSS media queries on the wrapper <div> will kick in meaning that although we’ve taken a slight performance hit shipping both variations of component, we don’t suffer from and visual quirks or CLS.
Post hydration we’ll still be in a state where both variations exist — indeed both variations will render on first pass so care is needed to make sure your components don’t have “onload / onmount” like side effects which could “double fire”. (For further reading on why we do this double pass see this GitHub issue. There is a better alternative which is more involved that I’m not covering here – this uses suspense boundaries and manual pruning of the DOM to remove the unwanted tree prior to hydration).
Following the first render the effect will fire, this will update the state to switch to the client-side mode. This triggers a re-render where we either continue to render the children as there is a viewport match, or nothing otherwise.

And just like that we’re free of CLS and can gracefully handle this kind of markup difference between different viewports. 🎉

Parting thoughts

The unfortunate state is that there is no perfect answer here, just trade-offs:

Get exactly what you want, but having to trust client hints which might not always be there... so maybe only sometimes getting what you want?
Have faster time to first byte (TTFB) and other similar page load metrics at the cost of having an annoying CLS vs almost certainly having CLS at the cost of page weight bloat slower that first impression.
Potential search engine optimisation (SEO) gotchas as a result of double rendering content (although it appears generally accepted that you shouldn’t get penalised for this kind of behaviour) coupled with the performance issues above for core web vitals having a direct impact on SEO.
Care still need for side-effects in SSR double rendered components (unless you put in more work).

Always evaluate what works for your use-case!

If you’re in an SPA world where you only client-render, this article was not for you!
If you are in a SSR world but the component that can be rendered differently is only rendered upon customer interaction or some other lazy or deferred trigger, then you don’t need to worry — just use client-side techniques.
If you’re in a SSR world, maybe consider some of the techniques discussed.

Hopefully this can save some research and thoughts for others in future. If you have any ideas, suggestions, or great ways (or packages) you use to solve this problem I’d love to hear it in the comments!

Like what I have to say? Consider a follow here or on Twitter.

Automating a11y testing: Part 1 — Axe

Craig Morten — Fri, 23 Jun 2023 13:38:27 +0000

Earlier this year I gave a talk at the John Lewis Partnership Tech Profession Conference 2023 on the topic of automating web accessibility (a11y) testing.

In this series I will cover the contents of this talk, kicking off with insights into the Axe suite of tools, followed by articles covering the wider a11y testing landscape: from visual regression testing for magnification and visual deficiencies, to emerging technology in the screen reader automation space.

I am not the user

Before going any futher I’m keen to express a disclaimer for this series: "I am not the user"!

I don’t have any training beyond "on the job" learnings, I am not a user of assistive technology in my day to day life outside of QA, and other than a mild astigmatism I consider myself able-bodied and neurotypical. Somewhat brings forth imposter syndrome when writing on the topic of a11y testing! Nevertheless, it’s something I’m passionate about.

Throughout this series I will be mostly focusing on the automation side of a11y testing, but it is also key to remember the importance of manual validation and exploratory testing. At the end of the day it is real people that are visiting and using your site, so it is incredibly important to ensure the quality of their experience, not just to tickbox that automation has passed. In fact, current tooling can only cover around 25% of WCAG requirements (granted an old article, but still holds true!) and although later in this series we will explore some emerging technology to start pushing that coverage higher, there are still numerous aspects of the customer experience that can only be tested by a real person.

With that all covered off, let’s have a wander through the world of automated web accessibility testing!

Current state of play

Axe

It wouldn’t be an article about web accessibility testing if the Axe suite by Deque wasn’t mentioned. Putting simply, this tooling has somewhat of a monopoly at the moment on accessibility automation, and for good reason.

"The axe family of tools help you check for digital accessibility" — https://www.deque.com/axe/

Outside of very specific, dedicated tools it has the best coverage out there, good APIs and interfaces, and although I think the technical documentation could be better (likely personal taste — it always ends up being there, just not structured how I expect), it is far from bad and is made up for by a wealth of community tutorials, articles, and repo examples.

Beyond their axe-core package Deque also maintain a number of dedicated packages for the majority of popular frameworks in the web test automation space, namely:

To give you a flavour, here is a snippet of how you might use the Playwright integration to perform static analysis on the live Waitrose Groceries site to access the accessibility of the primary navigation menu, lovingly referred to as the "mega menu":

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test('mega menu should not have a11y violations', async ({ page }) => {
  await page.goto('https://www.waitrose.com/ecom/shop/browse/groceries');

  await page.getByRole('button', { name: 'Groceries' }).click();

  await page.locator('#megamenu').waitFor();

  const accessibilityScanResults = await new AxeBuilder({ page }).include('#megamenu').analyze();

  expect(accessibilityScanResults.violations).toEqual([]);
});

Hopefully a very standard looking test — we navigate to the desired page, interact with the Groceries dropdown button, and wait for the mega menu to be visible. The Axe part is then remarkably straight-forward to get started with — we construct a new AxeBuilder, tell it to inspect the mega menu element tree, and run the analysis. Across the other framework packages the API usage is very similar.

cypress-axe

Now the eagle eyed of you will have noticed there was a popular framework missing from the list of Deque owned packages for Axe — Cypress!

Don’t worry, there is a really good community package to support Cypress as well in the form of the cypress-axe package.

The interface is a little different to how you use the other Axe packages, but personally I almost prefer the style which works quite naturally with the Cypress API style. Let’s see how we might test the accessibility of a John Lewis Search Product Listing Page (PLP):

describe('PLP Search A11y Violations', () => {
  beforeEach(() => {
    cy.visit('https://www.johnlewis.com/search?search-term=vans');

    cy.injectAxe();

    cy.configureAxe(AXE_CONFIG_OPTIONS);
  });

  it('should not have a11y violations on load', () => {
    cy.checkA11y(SELECTOR, AXE_RUN_OPTIONS);
  });
});

As is often the case with Cypress, the feedback loop from cypress-axe is also up there with clear logging explaining the issues when you are using the Cypress UI.

For example, in this failed test we can see the error description, some guidance, a URL for more information, and if we were to drill into the Nodes array it would give us pointers to the exact elements that are in violation of WCAG.

Pa11y

So what if you don’t currently use one of the aforementioned frameworks for your site?

In cases where you don’t have a setup ready to plug and play with one of the previous packages, and perhaps you’ve also not got the knowledge or resource to spend building up a suite using one of those frameworks (though these days the developer experience for setting up is pretty good!), then I would recommend taking a peek at Pa11y.

"Pa11y is your automated accessibility testing pal. It runs accessibility tests on your pages via the command line or Node.js, so you can automate your testing process." — https://pa11y.org/

The nice thing about Pa11y is you can run it straight from the command line making it natural to use for scripting, or for simple smoke checks in CI. It also ships a Node package so you can write a quick test in JavaScript or Typescript. Because it is a "wrapper" you can also benefit from it’s dual coverage utilising both Axe and HTML Code Sniffer (htmlcs), another static analysis tool. The documentation for Pa11y is also really good.

Let’s take a look at an example setup to test the page accessibility with the mega menu open:

import pa11y from 'pa11y';

const BASE_PA11Y_CONFIGURATION = {
  runners: ['axe', 'htmlcs'],
  standard: 'WCAG2AAA',
};

async function runPa11yTest() {
  try {
    const results = await pa11y('https://www.waitrose.com/ecom/shop/browse/groceries', {
      actions: ['click element #drop-down-nav', 'wait for element #megamenu to be visible'],
      ...BASE_PA11Y_CONFIGURATION,
    });

    console.log(results);
  } catch (error) {
    console.error(error);
  }
}

runPa11yTest();

Pa11y’s actions API for perfoming customer interactions is a little restricted, but it does has you covered for basic behaviours like clicking, typing, and waiting for something to be visible. Because it uses English based instructions, it has a similar feel to using BDD testing libraries (e.g. think Gherkin syntax for Cucumber) which lowers the barrier to entry for developers or even non-developers to easily tweak or extend the scope of tests — caveated of course if English is not your first language!

Unfortunately the style of this actions API does mean the tool doesn’t sit well within other frameworks — if you’re trying to use Playwright or Cypress to first act on the page for setup it will just get ignored. What Pa11y does under the hood is use Puppeteer to spin up it’s own Chrome tab to instrument and act out the instructions and a11y testing, so anything set up outside of Pa11y’s actions array will effectively be ignored.

But if you’re performing integration testing through the likes of Jest, Vitest, Mocha, etc., i.e. a testing framework that doesn’t support it’s own browser instrumentation, this can be a nice fit to extend a suite of tests to include a11y coverage.

In summary:

Conveniently wraps both Axe and htmlcs
Really good documentation
Simple to use action, though somewhat restricted
Doesn’t play so well with others

Axe browser extension

Moving out of what some folks might strictly class as test automation, it’s worth discussing some of the other tooling in the Axe family.

To complement the Axe packages, Deque also have a really nice Axe browser extension. Some features are premium, but the "scan all" feature meets most, if not all, needs for supplementing exploratory testing with a degree of automation, taking the heavy lifting away from manually inspecting HTML, colours, etc. to try and work out if components are compliant.

Axe Linter

There is also a VSCode plugin for an Axe Linter which can be a useful tool in proactively writing accessible code write at the early development stage — eliminating the slower feedback loop of only finding violations at your integration test stage in CI say.

Lighthouse

Lastly something that hopefully might be familiar to most readers!

If you’ve ever used the Accessibility reporting feature of Google Lighthouse, it uses Axe under the hood to drive the tests and provide the nicely displayed violation information.

This is not just the case for the Chrome DevTools instance of Lighthouse — the browser extension, CLI, and Node package all make use of Axe for the accessibility score and reporting features.

There are also some nice packages that wrap Lighthouse for easy use in test automation — for example, the cypress-audit package for Cypress worthy of a mention for it’s easy to use cy.lighthouse() command which can also be used for performance budget tests (it also has a cy.pa11y() command so you can kick off Pa11y tests if you prefer that interface instead).

Axe lessons

Having realed off and recommended a number of Axe tools, it’s worth pointing out some of the gotcha’s that I’ve experienced with them over the years.

React Axe issues

A big lesson for me has been around @axe-core/react. Initially I was very much in favour in the idea of having a tool that can instrument React and report issues to the console while running apps locally in development.

However... for any application of any size this package can massive tank performance which can make local development painful and slow — especially if you are running a hot module reloading (HMR) setup where every small tweak triggers a hot reload and a potential new Axe scan of the page. If you are using it currently and ever wonder why animations are super laggy and page updates look jarring, then React Axe is a real possibility as to why. Obviously this is my experience working on large applications, mileage may vary!

My personal recommendation is to scrap React Axe and instead reach for the Axe browser extension instead — you don’t get "live" reports to the console, but the flow is fairly natural and is a definite improvement for developer experience. The UI and level of detail you get on the extension is also far superior to the red text you get in the console scattered amongst everything else that is getting logged.

Infallible?

It’s also worth calling out that Axe isn’t perfect — it can occasionally have quite annoying false positives.

The most common case I’ve found is in it’s reporting of colour contrast, specifically when doing anything complex with either nested elements or using pseudo-elements.

For example, imagine you have a setup (admittedly contrived, but funnily enough I have experienced this in a side menu implementation!) where there is an element with a green background and then a number of nested transparent that cover the parent element. Above these is then say an absolutely positioned element, or CSS "Z translated" element, or perhaps an element from an entirely different part of the DOM positioned also above the green element and it’s covering children. Say this too has some nested children — the top layer of which has some white text.

The white text is above the green background, so this is WCAG AA compliant but the element containing the text is sufficiently removed from the background element that Axe fails to recognise which elements (and colours) are involved in the contrast comparison and fails with a violation.

The trick here is to keep it simple — if you need to ensure contrast just apply the appropriate background (or equivalent) to the element with the text. Yes, this is somewhat fixing a non-issue due to tooling limitations, but it is slightly less fragile to changes in your page markup implementation — in future someone might refactor to reduce the complexity of this component, and having the element satisfy the necessarily colour contrast requirements itself rather than relying on some other element will save future rework getting things compliant again in such a refactor. As the principle says, "things that move together should sit together".

There is also another lesson here — if you’re seeing false positives reported from Axe you’re probably doing something off in your markup. There are few scenarios that warrant such a complex element layout so Axe failing is probably a good hint that you’re in need of simplifying your HTML. Even if you are able to successfully satisfy the Axe requirement, it is very likely with such a layout that you will have potential issues with other accessibility requirements —what about users who make use of 2x zoom, or assistive technologies like magnifiers and screen readers? There is a real risk with something like above behaving poorly for some of these tools, e.g. screen readers are known for not handling non-semantic deeply nested markup very well.

Friction in WIP projects

Unless you’re in the rare position of starting from scratch with a greenfield project, the chances are you may well be pulling something like Axe into an existing project for a site that isn’t all that accessible at the moment, as a means to start improving the situation.

The thing to be aware of here is Axe’s limitations on ignoring violations, a capability that is often needed as a practical measure when you are looking to progressively improve a site’s accessibility which has a number of existing violations.

Of course ideally we don’t want to be ignoring anything, but a nice flow when working on such a codebase is to always create a ticket, and then add ignore rules named after or commented with the ticket reference and start ticking these off.

The frustrating thing with Axe ignores is that you can’t ignore a specific rule for a specific element. Unfortunately you can either ignore all rules for an element, or you can ignore all cases of a rule. The consequence is that often you will be over-ignoring elements or rules until you have cleared the decks of most or all violations, meaning until then you’re not really getting the protection from regression that you might hope for — folks adding new code may well introduce further violations of a similar type and Axe starts ignoring these as well!

What I recommend here is set up your tests from the start with sensible per test configuration and avoid having a single global configuration (though for DRY, it might be some common config is centralised). This way you can narrow the exclusions just for specific tests, e.g. say the contrast is only inaccessible when you toggle a button, you don’t want to ignore that button or the contrast rule for all other tests and scenarios in your suite!

Another technique that I’ve seen employed to reasonable effect is to introduce attributes to elements that you want to ignore, for example data-axe-ignore="true", and use this as a way to target elements that you need to temporarily ignore (through a selector such as [data-axe-ignore="true"]) in order to get CI up and running with Axe tests. Once you fix the violation, just remember to remove the data attribute! This does have the downside of polluting production markup unless you have steps to purge the attributes, so there are trade-offs here.

Closing notes

In this article I’ve covered off the majority of the Axe suite, if you want to find out more or explore some of Deque’s other offerings check out their site at https://www.deque.com/axe/.

In part 2 of this series I will cover a number of non-Axe tools and techniques for a11y test automation. See you there! 👋

"Reducing" a cryptic memory leak in Production

Craig Morten — Sun, 19 Mar 2023 21:58:47 +0000

I recently came across an interesting memory leak scenario which caught me off-guard, let’s see if you can spot the issue before the reveal at the end?

Below is some simplified replica code:

const express = require("express");
const { getSponsored, getRecommendations, getProducts } = require("./fetchers");

const app = express();

const baseResponse = {
  filters: [],
  products: [],
  subCategories: [],
};

function mergeResponses(responses) {
  return responses.reduce((finalResponse, response) => {
    finalResponse.filters.push(...response.filters);
    finalResponse.products.push(...response.products);
    finalResponse.subCategories.push(...response.subCategories);

    return finalResponse;
  }, baseResponse);
}

app.use("/products", async (req, res) => {
  const responses = await Promise.all([
    getSponsored(),
    getRecommendations(req),
    getProducts(),
  ]);

  res.send(mergeResponses(responses));
});

app.listen(3000);

The functionality is relatively straight-forward — we have a /products endpoint when is getting some sponsored, recommended, and normal products, merging parts of the results, and returning this aggregation as the response.

Now the use of Array.prototype.reduce() for the response merge is curious — we’re concatenating arrays so a simple for loop or .forEach() feels more appropriate, but hey for small responses at reasonably low RPS this is micro-optimisation, nothing to get too hung up on!

Photo by Clément Hélardot on Unsplash

But yet running something similar in a production like environment it was apparent that something was off... monitoring showed that old heap size was growing linearly with the number of requests — we had a memory leak!

The memory heap is divided into two major spaces: the New space where new objects are allocated, and the Old space where older objects are moved to after surviving for some time having not been garbage collected.

Debugging a memory leak

A straight-forward way to start an investigation into a Node memory leak when you can run the project locally (debugging in production is a different matter!) is to use the Node inspector, this can be invoked easily through a flag when starting your Node application:

$ node --inspect index.js

Debugger listening on ws://127.0.0.1:9229/2a7f05cd-96c8-4f6d-8c5f-adea987ab63b
For help, see: https://nodejs.org/en/docs/inspector

This will prompt the Node process to listen for a debugging client which, once connected, allows you to instrument your application with breakpoints as well as resource metrics and analysis for CPU and memory.

There a few options for debuggers — major IDEs like Visual Studio, Visual Studio Code, and JetBrains Webstorm all support the protocol making for easy debugging from within your editor. Here I’ll demonstrate an alternative, using Chrome DevTools.

Once your Node process is running with the inspector, open up Chrome and navigate to chrome://inspect in the URL address bar.

Here under the “Remote Target” section we can click on the “inspect” link to starting interrogating our index.js Node process.

Clicking this link pops open a new Chrome DevTools tab which is very similar to what you would get while debugging a page in the browser.

On the Memory tab we can click the “Take snapshot” button while the “Heap snapshot” option is selected to record a snapshot in time of the memory heap. This will pop up on the left sidebar and generally will automatically open up.

From this snapshot we can see all of the “constructors” (think primitives or objects) with columns for the distance from the window object, the size of each object in bytes, and the retained size of the object in bytes (the size of the object and the graph it retains).

This doesn’t yet give us enough information to easily assess where a memory leak might be, but can be interesting to peruse to understand the shape of your application’s memory.

Now in our case the memory leak appeared to be correlated to the amount of traffic the server was receiving, so to simulate that we can use tools like autocannon to very easily generate some base level load on the server.

Here was run one of three:

$ npx autocannon -c 100 http://localhost:3000/products 

Running 10s test @ http://localhost:3000/products
100 connections

    ┌─────────┬────────┬─────────┬─────────┬─────────┬───────────┬───────────┬─────────┐
    │ Stat    │ 2.5%   │ 50%     │ 97.5%   │ 99%     │ Avg       │ Stdev     │ Max     │
    ├─────────┼────────┼─────────┼─────────┼─────────┼───────────┼───────────┼─────────┤
    │ Latency │ 158 ms │ 1232 ms │ 2860 ms │ 3440 ms │ 1296.6 ms │ 786.21 ms │ 5350 ms │
    └─────────┴────────┴─────────┴─────────┴─────────┴───────────┴───────────┴─────────┘
    ┌───────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
    │ Stat      │ 1%      │ 2.5%    │ 50%     │ 97.5%   │ Avg     │ Stdev   │ Min     │
    ├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
    │ Req/Sec   │ 27      │ 27      │ 46      │ 147     │ 61.8    │ 35.14   │ 27      │
    ├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
    │ Bytes/Sec │ 34.4 MB │ 34.4 MB │ 60.6 MB │ 80.7 MB │ 60.7 MB │ 13.3 MB │ 34.4 MB │
    └───────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘

Req/Bytes counts sampled once per second.
# of samples: 10

718 requests in 10.09s, 607 MB read

... and run three of three:

$ npx autocannon -c 100 http://localhost:3000/products

Running 10s test @ http://localhost:3000/products
100 connections

    ┌─────────┬─────────┬─────────┬─────────┬─────────┬────────────┬────────────┬─────────┐
    │ Stat    │ 2.5%    │ 50%     │ 97.5%   │ 99%     │ Avg        │ Stdev      │ Max     │
    ├─────────┼─────────┼─────────┼─────────┼─────────┼────────────┼────────────┼─────────┤
    │ Latency │ 1554 ms │ 5400 ms │ 9049 ms │ 9189 ms │ 5389.29 ms │ 2438.84 ms │ 9314 ms │
    └─────────┴─────────┴─────────┴─────────┴─────────┴────────────┴────────────┴─────────┘
    ┌───────────┬─────┬──────┬─────────┬────────┬─────────┬─────────┬─────────┐
    │ Stat      │ 1%  │ 2.5% │ 50%     │ 97.5%  │ Avg     │ Stdev   │ Min     │
    ├───────────┼─────┼──────┼─────────┼────────┼─────────┼─────────┼─────────┤
    │ Req/Sec   │ 0   │ 0    │ 6       │ 36     │ 10.6    │ 11.62   │ 6       │
    ├───────────┼─────┼──────┼─────────┼────────┼─────────┼─────────┼─────────┤
    │ Bytes/Sec │ 0 B │ 0 B  │ 22.4 MB │ 128 MB │ 38.1 MB │ 41.5 MB │ 22.4 MB │
    └───────────┴─────┴──────┴─────────┴────────┴─────────┴─────────┴─────────┘

Req/Bytes counts sampled once per second.
# of samples: 10

232 requests in 10.09s, 381 MB read
26 errors (26 timeouts)

We can see here that something is very definitely wrong — after only 3 batches of 10 second load periods serving 100 connections we’ve gone from an average latency of 1296.6 ms to 5389.29 ms, a degradation of over 4 seconds!

Let’s take another heap snapshot using the “record” button at the top left of the Chrome DevTools Memory tab.

We can see that there certainly appears to be a lot more memory in play — the total snapshot size (listed under the snapshot name on the left sidebar) has increased from 5.6 MB to 24.7 MB and there are some large numbers listed for shallow and retained size.

However hunting blind in any one snapshot doesn’t get you very far typically, where the DevTools comes into it’s own is with the comparison options.

The last option in the Memory Tab’s control panel is a dropdown which will state “All objects” by default. Clicking on this dropdown gives you a number of options to compare between snapshots.

Here we select the “Objects allocated between Snapshot 1 and Snapshot 2” option to show us the delta between the two snapshots.

This immediately reduces the number of constructors in play when compared with the long lists for the individual snapshots — anything that hasn’t changed between the snapshots hasn’t been listed, removing a lot of the noise from the view.

Now comes the interesting part — finding the source of the leak! We start by sorting the list by retained size (typically the default) and navigating down the list of constructors one by one. Here we expand the (string) options first and take a look:

There’s nothing of interest here — we can tell from the small shallow and retained sizes of each of the strings listed that the issue doesn’t lie here. Although the combined size of each string does make for the largest sized collection, the size associated with each string is very small.

Intuitively we’re also not expecting a string to be the primary cause of our problem — nowhere in our logic are we growing a string (perhaps if we were doing some sort of string concatination, but we’re not). It’s apparent that a lot of these strings are likely long-lived constants, we likely took our first snapshot too early — if we were to take a third snapshot they would like be removed from the comparison view now the server has had some requests.

Moving onto the second constructor we can see something a little more interesting:

Here the shallow and retained sizes for the top three children are particularly large considering this is a delta!

Clicking on the first (object elements)[] row we can populate the “Retainers” bottom panel with detailed information on the object in question. This immediately points to a suspect:

In the bottom panel we can see that the massive retained size has been associated with the elements in a products array that exists on a baseResponse object used by the Express middleware stack! If we click on the second and third child we can see that these are for subCategories and filters respectively.

Somehow these arrays on our baseResponse object are growing in size after requests. This must mean that they are being mutated with more and more values! In fact if we expand the (object elements)[] children we can see the exact values that are populating our arrays.

So our leak is something causing the baseResponse arrays to grow on a per request basis... let’s go have another look.

Want to learn more about debugging memory with Node? Checkout the Node Memory Diagnostics tutorial page.

When does a reduce grow?

Let’s take another look at our mergeResponses() method:

const baseResponse = {
  filters: [],
  products: [],
  subCategories: [],
};

function mergeResponses(responses) {
  return responses.reduce((finalResponse, response) => {
    finalResponse.filters.push(...response.filters);
    finalResponse.products.push(...response.products);
    finalResponse.subCategories.push(...response.subCategories);

    return finalResponse;
  }, baseResponse);
}

We’re reducing over the array of responses, accumulating the associated filters, products, and subCategories and return the result. And in doing so somehow the baseResponse is being mutated?

Well exactly that — TIL!

Turns out that although Array.prototype.reduce() does not mutate the underlying array it does mutate the initialValue if it is provided. In many cases where you might provide a value directly like a number or an empty object this is fine, but if you create a bootstrap object like we have here then it will get mutated by each call, and re-used every time!

This isn’t mentioned anywhere on MDN or similar from what I can find, so is a bit of a gotcha — though I suspect the number of times this pattern is used is likely (hopefully) small.

So how do we fix this issue?

Well one option is to move the baseResponse into the mergeResponses() method so that it is no longer in top level scope and can be garbage collected once the request has been responded to. I.e.

function mergeResponses(responses) {
  const baseResponse = {
    filters: [],
    products: [],
    subCategories: [],
  };

  return responses.reduce((finalResponse, response) => {
    finalResponse.filters.push(...response.filters);
    finalResponse.products.push(...response.products);
    finalResponse.subCategories.push(...response.subCategories);

    return finalResponse;
  }, baseResponse);
}

Or as we opted for, move away from a reduce when we’re fundamentally not changing object shape:

function mergeResponses(responses) {
  const baseResponse = {
    filters: [],
    products: [],
    subCategories: [],
  };

  responses.forEach((response) => {
    baseResponse.filters.push(...response.filters);
    baseResponse.products.push(...response.products);
    baseResponse.subCategories.push(...response.subCategories);
  });

  return baseResponse;
}

So did you spot the memory leak before the big reveal?

Have you got the same code in your apps? 😱

Have you faced a memory leak recently?

Let me know in the comments along with your thoughts, comments, and queries! Till next time folks 🚀

A11y Unlocked: Screen Reader Automation Tests

Craig Morten — Sun, 16 Oct 2022 17:31:19 +0000

In the past year since my last post A11y Testing: Automating ScreenReaders on the topic of automating screen readers, we're now in a better place with progress made in standards and tooling.

It is exciting to see the W3C ARIA-AT Community Group formed with a mission to improve screen reader interoperability by building out a suite of specifications, test standards, and test automation to mature the screen reader ecosystem towards first-class tooling.

"Enabling AT interoperability is a large, ongoing endeavor that requires industry-wide collaboration and support. The W3C ARIA-AT Community Group is focusing on a stable and mature test suite for five screen readers by the end of 2023. In the future, we plan to test additional screen readers and other kinds of assistive technologies with a broader set of web design patterns and test material."

Source: https://aria-at.w3.org/

While the standards community slowly work towards their long term goal, I am pleased to share some screen reader automation tooling which you can use today! 🚀

Guidepup for playwright

In this tutorial we're going to make use of the package @guidepup/playwright to write e2e tests for testing the accessibility of pages for people who use the VoiceOver screen reader for MacOS.

guidepup / guidepup-playwright

Screen reader driver for Playwright tests.

Guidepup Playwright

Documentation | API Reference

This package provides Guidepup integration with Playwright for writing screen reader tests that automate VoiceOver on MacOS and NVDA on Windows.

Capabilities

Full Control - If a screen reader has a keyboard command, then Guidepup supports it.
Mirrors Real User Experience - Assert on what users really do and hear when using screen readers.

Getting Started

Set up your environment for screen reader automation with @guidepup/setup:

npx @guidepup/setup

If you are using GitHub Actions, check out the dedicated guidepup/setup-action:

- name: Setup Environment
  uses: guidepup/setup-action

Install @guidepup/playwright to your project:

npm install --save-dev @guidepup/playwright @guidepup/guidepup @playwright/test

Note: you require @guidepup/guidepup and @playwright/test as they are peer dependencies to this project.

And get cracking with your first screen reader tests in Playwright!

Examples

Head over to the Guidepup Website for guides, real world examples, environment setup, and complete API documentation…

View on GitHub

The package provides a reliable set of APIs to automate your screen reader a11y workflows when using Playwright. It does this by providing both Playwright configuration as well as a custom Playwright test instance that exposes a voiceOver object for controlling VoiceOver with.

Getting setup

Let's create a new project:

npm init

Install the @guidepup/playwright package as a dependency:

npm install @guidepup/playwright

We'll also need Playwright as the test runner for our browser:

npm install @playwright/test

Last but not least we'll want to test against Safari using Playwright, so let's use their tool get grab the browser:

npx playwright install webkit

We now have all the dependencies we need! 🚀

A touch of config

Create a new playwright.config.js file:

import { voConfig } from "@guidepup/playwright";

const config: PlaywrightTestConfig = {
  ...voConfig,

  // Your custom config ...
};

export default config;

Here we're pulling in the minimum recommended Playwright config for using Guidepup - you can check it out in the source code. This sets the number of workers to 1 and sets Playwright to use "headed" mode because when we're using VoiceOver we can only control one browser at a time (and ideally it's visible!).

We should now be completely set to get going with our first test! 🎉

Writing your first screen-reader test

Create a new file voiceover.spec.js and copy in the following contents:

// We use a special test instance from the Guidepup package
// that gives us access to VoiceOver!
import { voTest as test } from "@guidepup/playwright";
import { expect } from "@playwright/test";

// The test setup is exactly the same as normal for
// Playwright, expect we now get a `voiceOver` object as well
// as the `page` object!
test.describe("Playwright VoiceOver", () => {
  test("I can navigate the Guidepup Github page", async ({
    page,
    voiceOver,
  }) => {
    // Let's navigate to Guidepup GitHub page and wait for
    // page to be ready, nothing special here!
    await page.goto("https://github.com/guidepup/guidepup", {
      waitUntil: "domcontentloaded",
    });
    await expect(
      page.locator('header[role="banner"]')
    ).toBeVisible();

    // This is where things start to get awesome! Let's tell
    // VoiceOver to interact with the main page content, just 
    // the same as you would when use VoiceOver normally.
    await voiceOver.interact();

    // Let's do something a lil more exciting - move across
    // the page's headings until we reach the main Guidepup
    // repo title in the README using VoiceOver!
    while ((await voiceOver.itemText()) !== "Guidepup heading level 1") {
      await voiceOver.perform(
        voiceOver.keyboard.commands.findNextHeading
      );
    }
});

Check out the comments in the code for what will happen in each part - pretty awesome huh? 🙌

Last checks before we launch

In order to automate screen-readers a few OS level settings need to be sorted first!

You can head over to the Guidepup environment setup docs for more details, but I recommend trying out the @guidepup/setup package which is designed to get you setup no faff!

npx @guidepup/setup

You're now ready to rock! 🤘

Running your first screen-reader test

No special commands needed here, just the same as with any other Playwright test! Fire off the command then hold tight - we don't want to interact with the machine while the test is running as it might move VoiceOver's focus!

npx playwright test

What's next?

To find out more there's a whole load of reading, modules, and docs - check some of these out:

Time to try out some e2e screen-reader tests?

Let me know your thoughts, questions, and opinions in the comments below!

Till next time folks 👋

Setting up Stable Diffusion for MacOS

Craig Morten — Fri, 26 Aug 2022 13:48:00 +0000

With the landscape quickly changing, this article is fast becoming outdated!

If you face issues with the tutorial below I recommend you checkout the latest advice here.

Stable Diffusion is a latent text-to-image diffusion model that was recently made open source.

For Linux users with dedicated NVDIA GPUs the instructions for setup and usage are relatively straight forward. However for MacOS users you can't use the project "out of the box". Not to worry! There are some steps to getting it working nevertheless!

Environment Setup

To begin you need Python, Conda, and a few other libraries set up:

# Install Python, Cmake, Git, and Protobuf
brew install python \
  cmake \
  git \
  protobuf

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Install Conda:
## Either use this for older "pre-M1" Macs:
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
bash Miniconda3-latest-MacOSX-x86_64.sh

## Or use this for older M1 Macs:
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh
bash Miniconda3-latest-MacOSX-arm64.sh

You may need to restart your terminal at this point for the changes for the new libraries to be picked up.

Clone this fork of the project and checkout the apple patch branch:

git clone https://github.com/magnusviri/stable-diffusion
cd stable-diffusion
git checkout apple-silicon-mps-support

At this point you will need to make sure you're using Python 3, check out this article for different ways to make Python 3 the default version on your Mac.

Set up the Conda environment:

conda env create -f environment-mac.yaml
conda activate ldm

And finally set the following environment variable:

export PYTORCH_ENABLE_MPS_FALLBACK=1

Code Changes

Our environment is now set up, but we have a few tweaks that we need to allow the code to gracefully fallback to using the CPU (if required!).

Append .contiguous() at ldm/models/diffusion/plms.py#L27 resulting in:

-        attr = attr.to(torch.float32).to(torch.device(self.device_available))
+        attr = attr.to(torch.float32).to(torch.device(self.device_available)).contiguous()

Similarly append a new line x = x.contiguous() after ldm/modules/attention.py#L211 so it looks something like:

def _forward(self, x, context=None):
+       x = x.contiguous()
        x = self.attn1(self.norm1(x)) + x

Download Stable Diffusion Weights

Let's install our diffusion weights

curl "https://www.googleapis.com/storage/v1/b/aai-blog-files/o/sd-v1-4.ckpt?alt=media" > sd-v1-4.ckpt

Create Images 🚀

You should now be ready to generate images on your MacOS device using Stable Diffusion! 🎉 🎉

python scripts/txt2img.py --prompt "a drawing of web developers" --plms --ckpt sd-v1-4.ckpt --skip_grid --n_samples 1

Tricks and hacks gleamed from https://github.com/CompVis/stable-diffusion/issues/25 - credit to all the folks in that thread for figuring out how to get things working!

Testing Web Vitals With Cypress

Craig Morten — Sun, 03 Apr 2022 10:48:34 +0000

It is well understood that performance is a critical consideration for any website which can have far reaching impacts on everything from customer satisfaction, SEO, and ultimately your bottom line. You cannot determine the success of performance work without the ability measure the results and compare to performance budgets - this calls for testing infrastructure to make sure you have the necessary visibility on metrics... introducing cypress-web-vitals.

cypress-web-vitals allows you to test against the Google Web Vital signals within your Cypress workflows through a new cy.vitals() custom command.

Web Vitals is an initiative by Google to provide unified guidance for quality signals that are essential to delivering a great user experience on the web.

Reference: https://web.dev/vitals/

Getting started

In your project, install the dependencies:

npm install --save-dev cypress-web-vitals cypress-real-events

Note: cypress-web-vitals currently makes use of cypress-real-events to click the page to calculate first input delay. Hence it is needed as a peer-dependency.

Within you support commands file (normally cypress/support/commands.js), add this one liner to get setup:

import "cypress-web-vitals";

And now you're set to get going with Web Vital performance budget tests in your Cypress workflow! 🎉

Add you first test like so:

describe("Web Vitals", () => {
  it("should pass the meet Google's 'Good' thresholds", () => {
    cy.vitals({ url: "https://www.google.com/" });
  });
});

You are now set up to test against all of the Web Vitals using Google's "Good" threshold values:

Largest contentful paint (LCP) - 2500.
First input delay (FID) - 100.
Cumulative layout shift (CLS) - 0.1.
First contentful paint (FCP) - 1800.
Time to first byte (TTFB) - 600.

Customise your tests

You can further customise your tests through additional optional configuration which is passed to the cy.vitals(webVitalsConfig) call:

Optional url: string - The url to audit. If not provided you will need to have called cy.visit(url) prior to the command.
Optional firstInputSelector: string - The element to click for capturing FID. The first matching element is used. Default: "body".
Optional thresholds: object - The thresholds to audit the Web Vitals against. If not provided Google's "Good" thresholds will be used. If provided, any missing Web Vitals signals will not be audited.
Optional vitalsReportedTimeout: number - Time in ms to wait for Web Vitals to be reported before failing. Default: 10000.

For example:

// Use the `main` element for clicking to capture the FID.
cy.vitals({ firstInputSelector: "main" });

// Test the page against against a CLS threshold of 0.2.
cy.vitals({ thresholds: { cls: 0.2 } });

For more details on usage refer to the API docs.

How does it work?

The url is visited with the HTML response intercepted and modified by Cypress to include the web-vitals script and some code to record the Web Vitals values.
Several elements (if exist) starting with the provided element (based on firstInputSelector) are then clicked in quick succession to simulate a user clicking on the page. Note: if choosing a custom element, don't pick something that will navigate away from the page otherwise the plugin will fail to capture the Web Vitals metrics.
The audit then waits for the page load event to allow for the values of LCP and CLS to settle; which are subject to change as different parts of the page load into view.
Next the audit simulates a page visibility state change which is required for the CLS Web Vital to be reported.
The audit then attempts to wait for any outstanding Web Vitals to be reported for which thresholds have been provided.
Finally the Web Vitals values are compared against the thresholds, logging successful results and throwing an error for any unsuccessful signals. Note: if the audit was unable to record a Web Vital then it is logged, but the test will not fail.

Testing sites in the wild

Here are some example test runs against FAANG homepages to see cypress-web-vitals in action:

Facebook

cy.vitals({ url: "https://www.facebook.com/" });

Amazon

cy.vitals({ url: "https://www.amazon.com/" });

Netflix

cy.vitals({
  url: "https://www.netflix.com/gb/",
  firstInputSelector: ".our-story-card-title",
});

For Netflix we have had to introduce a custom element choice for the simulated "first click". Even though first input delay can be measured when in cases where there is no event listener registered to the element, there are scenarios where clicking the body doesn't cut it. Some good examples of elements that will reliably trigger an FID metric are:

Text fields, checkboxes, and radio buttons (<input>, <textarea>)

Select dropdowns (<select>)

Links (<a>)

In order to ensure the Web Vitals can be tested against, it is best to try find an element that reliably triggers an FID, but won't navigate away from the page (perhaps avoid <a>!).

Google

cy.vitals({ url: "https://www.google.com/" });

Wrap-up

Using any awesome performance testing tooling lately? Tried out cypress-web-vitals on your site and have results to share? Got any comments, queries, or questions? Leave a comment below!

That's all folks! 🚀

Create Your Own Live Browser Refresh In Deno

Craig Morten — Sat, 15 Jan 2022 17:39:07 +0000

In modern web development we've grown accustomed to rich developer experience features such as hot module replacement (HMR) from the likes of Webpack HMR and React Fast Refresh which allow us to iterate on our apps fast, without the pain of slow server restarts.

Ever wondered how this tooling might work? In this tutorial we will build a simple live browser refresh in Deno that demonstrates so of the basics!

Getting started 🤔

To begin you will need to install Deno and create a new directory to work in, e.g. ./refresh/.

For this tutorial I am using Deno v1.17.3, but the code should work with future versions of Deno as it doesn't require any external dependencies, provided there are no breaking changes to the Deno APIs (e.g. for v2).

In your new directory create a mod.ts file. This will act as the entrypoint for our Deno module, and contain all of our server-side code.

Watching files 👀

The first part of our live browser refresh module is a function to watch for file changes - this will later allow us to tell the browser to refresh when we make and save changes to our application.

/**
 * Watch files from current directory
 * and trigger a refresh on change.
 */
async function watch() {
  // Create our file watcher.
  const watcher = Deno.watchFs("./");

  // Wait for, and loop over file events.
  for await (const event of watcher) {
    // TODO: trigger a browser refresh.
  }
}

Here we define our initial watch() function which makes use of the built-in Deno.watchFs(...) method to watch for file system events anywhere within our current directory. We then loop over the events picked up by the watcher, where we will add code to trigger a browser refresh.

Before we move onto code for triggering browser refresh, it's worth taking a look at different file system events that can fire. Namely:

interface FsEvent {
  flag?: FsEventFlag;
  kind:
    | "any"
    | "access"
    | "create"
    | "modify"
    | "remove"
    | "other";
  paths: string[];
}

It would be a bit annoying if our application reloaded every time a file was accessed but not necessarily changed. Let's update our loop to filter out some of these events.

  for await (const event of watcher) {
    // Skip the "any" and "access" events to reduce
    // unnecessary refreshes.
    if (["any", "access"].includes(event.kind)) {
      continue;
    }

    // TODO: trigger a browser refresh.
  }

WebSocket middleware 🏎

In this tutorial we will be using WebSockets to communicate the need to trigger a browser refresh. It is worth noting that you could also use Server Sent Events to achieve a similar result. If you try it out, do share in the comments below!

We will start with setting up the server-side WebSocket behaviour. For this we will create a small middleware function that will accept specific requests to the server, and upgrade the connection to a WebSocket.

/**
 * Upgrade a request connection to a WebSocket if
 * the url ends with "/refresh"
 */
function refreshMiddleware(req: Request): Response | null {
  // Only upgrade requests ending with "/refresh".
  if (req.url.endsWith("/refresh")) {
    // Upgrade the request to a WebSocket.
    const { response, socket } = Deno.upgradeWebSocket(req);

    // TODO: handle the newly created socket.

    return response;
  }

  // Leave all other requests alone.
  return null;
};

Here our function first checks the request URL to see if it ends with "/refresh". If not we leave the request alone.

When we get a match we use the built-in Deno.upgradeWebSocket(...) method to upgrade our connection to a WebSocket. This method returns an object including the response that must be returned to the client for the upgrade to be successful, and a socket instance.

Given we will be using the socket instance as our means of instructing the client to refresh the browser, let's add some code to store the WebSocket as well as handle when it closes.

/**
 * In-memory store of open WebSockets for
 * triggering browser refresh.
 */
const sockets: Set<WebSocket> = new Set();

/**
 * Upgrade a request connection to a WebSocket if
 * the url ends with "/refresh"
 */
function refreshMiddleware(req: Request): Response | null {
  if (req.url.endsWith("/refresh")) {
    const { response, socket } = Deno.upgradeWebSocket(req);

    // Add the new socket to our in-memory store
    // of WebSockets.
    sockets.add(socket);

    // Remove the socket from our in-memory store
    // when the socket closes.
    socket.onclose = () => {
      sockets.delete(socket);
    };

    return response;
  }

  return null;
};

We've now added an in-memory store for created WebSockets. When we upgrade the connection we add the new socket to our store as well as a handler for removing the socket from the store when it closes.

Triggering browser refresh 🙌

We're now ready to update our file watching code to trigger a browser refresh. We will do this by using the WebSockets created in our middleware to send a refresh event to the client.

/**
 * Watch files from current directory
 * and trigger a refresh on change.
 */
async function watch() {
  const watcher = Deno.watchFs("./");

  for await (const event of watcher) {
    if (["any", "access"].includes(event.kind)) {
      continue;
    }

    sockets.forEach((socket) => {
      socket.send("refresh");
    });
  }
}

Here we loop over the sockets in-memory store, and for each WebSocket we send our custom refresh event.

Finishing our server module 🧑‍💻

To finish off our server module we just need to tie our file watching and server middleware together. For this we create our refresh() function module export which users can consume to their servers.

/**
 * Constructs a refresh middleware for reloading
 * the browser on file changes.
 */
export function refresh(): (req: Request) => Response | null {
  watch();

  return refreshMiddleware;
}

This final exported function ties our work together. First it starts the file watcher, and then returns the middleware that can be used to handle the refresh communications between the server and the browser.

Handling refresh events client-side 💥

Now we're all sorted on the server, let's hop over the some coding for the client. First we need to create a client.js file to host our code.

Let's just dive in with the full code:

(() => {
  let socket, reconnectionTimerId;

  // Construct the WebSocket url from the current
  // page origin.
  const requestUrl = `${window.location.origin.replace("http", "ws")}/refresh`

  // Kick off the connection code on load.
  connect();

  /**
   * Info message logger.
   */
  function log(message) {
    console.info("[refresh] ", message);
  }

  /**
   * Refresh the browser.
   */
  function refresh() {
    window.location.reload();
  }

  /**
   * Create WebSocket, connect to the server and
   * listen for refresh events.
   */
  function connect(callback) {
    // Close any existing sockets.
    if (socket) {
      socket.close();
    }

    // Create a new WebSocket pointing to the server.
    socket = new WebSocket(requestUrl);

    // When the connection opens, execute the callback.
    socket.addEventListener("open", callback);

    // Add a listener for messages from the server.
    socket.addEventListener("message", (event) => {
      // Check whether we should refresh the browser.
      if (event.data === "refresh") {
        log("refreshing...");
        refresh();
      }
    });

    // Handle when the WebSocket closes. We log
    // the loss of connection and set a timer to
    // start the connection again after a second.
    socket.addEventListener("close", () => {
      log("connection lost - reconnecting...");

      clearTimeout(reconnectionTimerId);

      reconnectionTimerId = setTimeout(() => {
        // Try to connect again, and if successful
        // trigger a browser refresh.
        connect(refresh);
      }, 1000);
    });
  }
})();

A lot going on here!

First we create some variables for storing the current WebSocket and a reconnection timer id. We then construct the url that will be used by the WebSocket for requests. Notice how it ends in /refresh, just as we coded our server middleware function to detect and handle. Then we kick off the connection with a call to the connect(...) method.

The connect(...) function is where the majority of the work takes place. We ensure that any pre-existing sockets are closed - we want to avoid situations where there are more than one connection resulting in "double" refreshes! The WebSocket is then constructed using the request url, and a series of event listeners are setup to handle different WebSocket events:

The main event listener is for "message" events. This receives messages from the server, and if it receives our custom refresh event, it fires a call to the refresh() function which refreshes the browser.
The "close" event listener handles when we lose the connection from the server. This can happen easily with network blips (e.g. when you pass through a tunnel and lose signal!) so always good to handle. Here we setup a timeout to try and restart the connection again by calling connect(...) after a second delay. This time we pass the refresh function as a callback to trigger a refresh once our connection is back.
Finally, the "open" event listener fires when the connection opens, and here we just execute the provided callback. This is used in the aforementioned reconnection logic to trigger the browser refresh when we get our connection back.

Congratulations!! 🥳 🎉

And we're done! Between the server mod.ts and the browser client.js we've now got all we need to successfully implement a live browser refresh on code change.

Don't believe me? Let's try it out!

First we will need to write a simple server to consume our new refresh module. Let's create a server.ts:

import { serve } from "https://deno.land/std/http/server.ts";
import {
  dirname,
  fromFileUrl,
  join,
} from "https://deno.land/std/path/mod.ts";
import { refresh } from "./mod.ts";

// Create useful file path variable for our code.
const __dirname = fromFileUrl(dirname(import.meta.url));
const clientFilePath = join(__dirname, "./client.js");
const indexFilePath = join(__dirname, "./index.html");

// Construct the refresh middleware.
const refreshMiddleware = refresh();

// Start a server on port `8000`.
serve((req: Request) => {
  // Handle custom refresh middleware requests.
  const res = refreshMiddleware(req);

  if (res) {
    return res;
  }

  // Handle requests for the client-side refresh code.
  if (req.url.endsWith("client.js")) {
    const client = Deno.readTextFileSync(clientFilePath);

    return new Response(client, {
      headers: {
        "Content-Type": "application/javascript"
      }
    });
  }

  // Handle requests for the page's HTML.
  const index = Deno.readTextFileSync(indexFilePath);

  return new Response(index, {
    headers: { "Content-Type": "text/html" }
  });
});

console.log("Listening on http://localhost:8000");

This server code uses the Deno Standard Library for some server and path utilities. It constructs some variables storing the path to files the server needs to return, constructs the refresh middleware using the module that we've created in this tutorial, and then uses the standard library serve(...) method to start a server on port 8000.

We first call our refresh middleware with the request, and if we get a non-null response we return it - this means the request was for a WebSocket connection! Otherwise we handle requests for our client.js code, and otherwise fallback to returning an index.html. Let's create this index.html file now:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Example Refresh App</title>
    <style>
      body {
        background: #2c3e50;
        font-family: Verdana, Geneva, Tahoma, sans-serif;
        color: #ddd;
        text-align: center;
        font-size: 18px;
      }
    </style>
  </head>
  <body>
    <script src="/client.js"></script>
    <h1>Hello Deno!</h1>
  </body>
</html>

And there we have it! Let's run our new server:

deno run --allow-read --allow-net ./server.ts

If we open a browser on http://localhost:8000 we should see our simple "Hello Deno!" webpage.

Now for the exciting part - let's see if the live browser refresh works! Head to your index.html and try changing the text or some of the CSS. Notice anything different about the page in the browser? 💥

For an example of all this code working (and more!), check out the finished version at https://deno.land/x/refresh. 🦕

Written any cool Deno code lately? Perhaps you've built your own live browser refresh, or even HMR module that is worth a share?

Reach out on my twitter @CraigMorten, or leave a comment below! It would be great to hear from you! 🚀🚀

A11y Testing: Automating Screen Readers

Craig Morten — Thu, 30 Dec 2021 17:37:43 +0000

Since this article there have been some developments, check out A11y Unlocked: Screen Reader Automation Tests for the latest!

There are lots of toolkits out there for ensuring the inclusivity and accessibility of our sites: with the likes axe-core and htmlcs for static analysis, and pa11y, koa11y, and cypress-audit (there are more) for helping us to explore or automate static analysis checks into our CI pipelines. 🚀🚀

"Chosen by Microsoft, Google, development and testing teams everywhere, axe is the World’s leading digital accessibility toolkit."

Alongside our a11y tooling we've seen amazing growth in the landscape of browser testing automation tooling, with the rise of Cypress, Playwright (a new personal fav 🥰), and several other players. These have evolved to provide a rich collection of reliable APIs and tooling for end-to-end or integration testing sites across a solid matrix of browsers - complemented by looking to the likes of SauceLabs, BrowserStack, and Applitools who can provide an extensive range of browsers and devices so you don't need to worry about purchasing an expensive device cupboard! 💰💰💰

The ScreenReader Test Tooling Gap

But... these browser testing tools tend to lack any visibility or emphasis on a11y. Certainly through the likes of the testing library suite and it's guiding principles you can write tests in such a way that it mirrors the customer's usage, but only provides so much confidence, and it very much orientated to mouse, keyboard, and touch users. 🧐

And with regards to the a11y tools that are out there... we don't typically rely on static analysis for test coverage for other functional aspects of sites - indeed we use our favourite aforementioned cross-browser testing framework! It feels a shame that the "World’s leading digital accessibility toolkit" should be lacking in providing the ability to integration test with the actual assistive technologies that folks are using to navigate the web.

As a finger in the air of the limitations of current tooling, this article by Karl Groves from 2012 explains how 40-50% of the WCAG is not feasible to be tested by existing (and I'll admit in some cases also future) tooling.

"On the other hand, there are things which can be tested for but the results would need to be verified by a skilled human reviewer. For example, while machine testing can verify that the alt attribute exists for an image, it cannot tell you whether the value supplied for the alt attribute is a suitable alternative for the image. That requires a human."

Instead teams set on treating the accessibility of their sites as first class, rather than an afterthought, are forced to provide this testing coverage manually. This isn't necessarily a bad thing - manual exploration by your QAs adopting a ScreenReader using personna is the closest you can likely get to gaining confidence your site will work for folks out in the wild. 👩‍💻

But can we truly afford to have to manually QA every change to our sites to ensure they continue to work functionally for assistive technology? It is already resource-draining enough that teams typically automate golden path flows with browser testing tools to reduce the overhead of manual testing - there are simply too many browsers which demands a degree of trade-off. To name a core few:

Safari on MacOS, iOS
Chrome on MacOS, Windows, Android
Firefox on MacOS, Windows

We're already at 7 different browser vs device combinations, and this isn't accounting for different viewports, locales, current and previous versions, and many other variations.

If we now throw VoiceOver, JAWS, Narrator, NVDA, Talkback, and friends into the mix we can easily triple the workload on our QAs. 😰

What is out there currently?

This gap in automated tooling for mirroring existing functional tests for mouse, keyboard, and touch, but for ScreenReaders has been bugging me for a couple of years. A few weekends back I decided to finally explore to see if I simply hadn't looked hard enough.

Initially it didn't look hopeful...

// Detect dark theme var iframe = document.getElementById('tweet-852866736848293888-677'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=852866736848293888&theme=dark" }

Apr 12 '17

No. At least not one that is any good nor represents how a screen reader actually reads a page or responds to ARIA.

The best answer is to test in real screen readers, ideally by getting real users as they know how to use these tools. Consider contacting your local…

Open Full Answer

There doesn't appear to be any (large) players out there providing automated solutions for ScreenReader testing.

There are at least promising signs that this could be on the horizon, with companies starting to offer live testing solutions around assistive technologies such as Assistiv Labs and BrowserStack Live offering the ability to use ScreenReaders on remote VMs so you can manually test setups for devices or tooling you don't own or have licenses for. This makes manual a11y testing more accessibile (🥁), but doesn't solve our manual overhead problem.

Digging further started to yield a little more promise with the discovery of auto-vo, screen-reader-reader, and web-test-runner-voiceover (article - just found this while writing this post!) following the finding of a tweet from Smashing Magazine and then rabbit-holing on twitter... 🕳🐇

// Detect dark theme var iframe = document.getElementById('tweet-1407690678591700992-480'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1407690678591700992&theme=dark" }

You can check out the Smashing Magazine article here.

These libraries offer a promising start to the idea of automating ScreenReader testing. Focus across the 3 appears to be primarily on VoiceOver for MacOS (perhaps, like me, because thats the development machine they use!), but screen-reader-reader starts to pave the way for automating NVDA on windows machines. 💥

I haven't managed to have much success with auto-vo but, you can see it in action with explanations through @ckundo's twitter thread:

// Detect dark theme var iframe = document.getElementById('tweet-1402711360216223747-253'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1402711360216223747&theme=dark" }

And with it's precedent, you can imagine taking the core of it and making a far more generic and powerful API for control over VoiceOver, and potentially any ScreenReader... which is exactly what screen-reader-reader appears to have aimed to do 2 years ago! Sadly neither of these packages appear to have maintained velocity to mature into the testing ecosystem... my suspicions being 2-fold:

They aren't gracefully plug-n-play with the popular browser testing frameworks of today
They don't work in (any?) CI environments

The prior point is where I see more promise with web-test-runner-voiceover, the ability to easily integrate ScreenReader test flows into your existing test suite is an excellent sell - though I suspect success would need to come in the form of a "core" ScreenReader driver module with additional integration modules so you get support for the Cypress, Playwright, WDIO, Nightwatch, Protractor, etc. of the world.

The latter point, I'll come to... 😅

Adding to the soup

Fueled by the promise that it could actually be feasible to automate testing with ScreenReaders I have added my own flavour into the mixing pot with:

guidepup / guidepup

Screen reader driver for test automation.

Guidepup

Documentation | API Reference

Guidepup is a screen reader driver for test automation.

It enables testing for VoiceOver on MacOS and NVDA on Windows with a single API.

Capabilities

Full Control - If a screen reader has a keyboard command, then Guidepup supports it.
Mirrors Real User Experience - Assert on what users really do and hear when using screen readers.
Framework Agnostic - Run with Jest, with Playwright, as an independent script, no vendor lock-in.

Getting Started

Set up your environment for screen reader automation with @guidepup/setup:

npx @guidepup/setup

Install Guidepup to your project:

npm install @guidepup/guidepup

And get cracking with your first screen reader automation code!

Examples

Head over to the Guidepup Website for guides, real world examples, environment setup, and complete API documentation with examples.

You can also check out these awesome examples to learn how you could use Guidepup in your projects.

Alternatively check…

View on GitHub

I fear with 3 attempts already out there I am somewhat living one of my favourite xkcd comics... but worst case hopefully the guidepup code will serve as a platform to help bolster whatever tool emerges on top! 🙃

Beyond some of the previous attempts I have codified 226 of the keyboard gestures, 129 keyboard commander commands, as well as exposed most of the built-in AppleScript APIs for VoiceOver. So far I've had good success integrating with Playwright, and suspect it would be similar for other browser testing tools.

The real gotcha which I sense others have faced is trying to get any such solution into a CI environment - if a tool can't run reliably on a remote VM then it simply isn't scalable for teams to use for automated workflows. Specifically, trying to get VoiceOver to be controllable in CI.

Getting over the VoiceOver issue

The goto for scripting and automating applications on MacOS is through AppleScript. To ensure users have a degree of protection from malicious scripts, applications that want to run AppleScript to control other applications have to first be allowlisted in the "Accessibility" section of the "System Preferences" "Security & Privacy" pane.

This leads to the first hurdle, if we want to automate VoiceOver from the terminal instance in CI we can't remote in and click links and enter passwords every run! Luckily, this is a well understood problem space, and many CI providers are already setup with the required changes to the TCC.db (transparency, consent, control!) on MacOS. Check out this article for details on TCC. Curious what the code for this configuration looks like? Check out the GitHub actions virtual-environment repo for a good example. 🤓

There is a second hurdle though. In order to control VoiceOver through AppleScript you also need this checkbox ticked:

Locally you can automate the checking of the tickbox with an AppleScript (feels like privilege escalation to use the very script to unlock more control!), e.g. the following works on Monterey:



-- Startup delay to reduce chance of "Application isn't running (-600)" errors
delay 1

set timeoutSeconds to 5.000000

-- Open VoiceOver Utility
tell application "VoiceOver Utility" to activate

tell application "System Events"
    set endDate to (current date) + timeoutSeconds
    repeat until (exists window 1 of application process "VoiceOver Utility")
        if ((current date) > endDate) then
            log "Can not find VoiceOver Utility window"
            exit repeat
        end if

        delay 0.2
    end repeat

    tell application process "VoiceOver Utility"
        -- Prevent VoiceOver welcome message on startup
        set endDate to (current date) + timeoutSeconds
        repeat until (exists checkbox 1 of splitter group 1 of window 1)
            if ((current date) > endDate) then
                log "Can not find VoiceOver welcome message checkbox"
                exit repeat
            end if

            delay 0.2
        end repeat

        click checkbox 1 of splitter group 1 of window 1

        -- Enable AppleScript control
        set endDate to (current date) + timeoutSeconds
        repeat until (exists checkbox 2 of splitter group 1 of window 1)
            if ((current date) > endDate) then
                log "Can not find AppleScript control checkbox"
                exit repeat
            end if

            delay 0.2
        end repeat

        click checkbox 2 of splitter group 1 of window 1
    end tell

    -- Wait for SecurityAgent dialog to open
    set endDate to (current date) + timeoutSeconds
    repeat until (exists window 1 of application process "SecurityAgent")
        if ((current date) > endDate) then
            log "Can not find SecurityAgent window"
            exit repeat
        end if

        delay 0.2
    end repeat

    -- Enter credentials
    key code 48 using shift down
    delay 0.2
    keystroke "<USERNAME>" -- update accordingly, though beware of credentials in plaintext!
    delay 0.2
    key code 48
    delay 0.2
    keystroke "<PASSWORD>" & return -- update accordingly, though beware of credentials in plaintext!

    -- Wait for SecurityAgent dialog to close
    set endDate to (current date) + timeoutSeconds
    repeat while (exists window 1 of application process "SecurityAgent")
        if ((current date) > endDate) then
            log "SecurityAgent window won't close"
            exit repeat
        end if

        delay 0.2
    end repeat
end tell

tell application "VoiceOver Utility" to quit

But when you try such a script in the likes of GitHub actions the "SecurityAgent" dialog never appears meaning we can't authenticate this configuration.

Diving into the depths of VoiceOver Utility configuration, it appears that the checkbox drives two pieces of configuration:

The creation of a /private/var/db/Accessibility/.VoiceOverAppleScriptEnabled file (containing the single character a)
The setting of VoiceOver system preference defaults

The latter can be easily resolved through running:



defaults write com.apple.VoiceOver4/default SCREnableAppleScript 1

But the prior is a bit more of an issue... the /private/var/db/ directory and files therein are protected by Apple's SIP (Security Integrity Protection - read more here!). This can be disabled in recovery mode (be careful, it’s there for a reason) but rebooting a VM isn’t an option in CI typically. 😞

This lead me to wonder how do "native" Apple apps manage to write to SIP files if even sudo is a no-go?! Transpires they use the Security Foundation Framework to obtain authorisation with a special "system.preferences" right using the obtainWithRight API.

So after a several hour download of Xcode I dived into writing my first piece of swift... somewhat wary that the only prior art for this was for preferred networks, this unanswered thread and some defcon Apple malware talks...

This yielded something that appears to ask for permissions... and then doesn't work 😅



#!/usr/bin/env swift

import SecurityFoundation

let filePath = "/var/db/Accessibility/.VoiceOverAppleScriptEnabled"

do {
    guard
        let authref = SFAuthorization.authorization() as? SFAuthorization
    else {
        throw CocoaError(.fileWriteNoPermission)
    }

    let authent = Authenticator.sharedAuthenticator()

    try authref.obtain(withRight: "system.preferences", flags: [.extendRights, .interactionAllowed, .preAuthorize])
    defer { authref.invalidateCredentials() }

    do {
        let fileManager = FileManager.default;

        // Check if file exists
        if fileManager.fileExists(atPath: filePath) {
            // Delete file
            try fileManager.removeItem(atPath: filePath)
        } else {
            print("File does not exist")
        }
    } catch let error as NSError {
        print("An error took place: \(error)")
    }

} catch {
    print(error)
}

...



Error Code=513 "“.VoiceOverAppleScriptEnabled” couldn’t be removed because you don’t have permission to access it."

It transpires even with authorization, SIP is a no-go without Apple signing keys for special entitlements.

So where do we go from here?

Despite some of the complications mentioned above, it's not all doom and gloom! The folks at Microsoft who maintain the GitHub Actions virtual machines have been very receptive to getting this setting enabled, and there's no indication (yet) that we should face the same issues for NVDA. ☺️

Seen any cool ScreenReader automation tools out there that are worth a share?

Got any ideas to solve some of the challenges mentioned?

Want to help out building out some cool tooling for a11y automated testing?

Reach out on my twitter @CraigMorten, or leave a comment below! Will be great to hear from you! 🚀🚀

Kickstart Your Deno Web Project With The Opine CLI

Craig Morten — Sat, 09 Jan 2021 12:35:16 +0000

In this third Opine article we will be looking at how you blast through your website project setup by using the Opine CLI.

Prerequisites: Opine Tutorial Part 2: Creating A Website In Deno

Objective: Start your own new website projects using the Opine CLI.

Overview

This article will cover how you can create a basic template for a website using the Opine CLI for Deno.

In fact, the template created will be remarkably similar to what we covered in the previous article in this series! We recommend checking that out if you want experience building a project from scratch.

Because we have already covered project structure, and explained concepts such as views, we will focus primarily here on just how to bootstrap your project and the different options available.

Using the Opine CLI

You can install the Opine CLI using Deno's script installer commandlet:

deno install -f -q --allow-read --allow-write --allow-net --unstable https://deno.land/x/opinecli@1.0.2/opine-cli.ts

This install command creates a small executable shell script wrapper which runs Deno using the specified CLI flags and main module. The generated script is then placed into the installation root's bin directory. The exact root directory used is determined by:

The --root flag, if provided;
The DENO_INSTALL_ROOT environment variable, if set;
$HOME/.deno.

Note you will need to add the installation directory to your path in order to invoke the CLI with providing a full directory path.

For example, add export PATH="$HOME/.deno/bin:$PATH" to your $HOME/.bashrc. Remember you will need to source this file or restart your shell / terminal for it to take effect!

You should now be able to execute the Opine CLI from any shell! 🎉 🎉

The Opine CLI has several options, the easiest way to view them all is using the --help or -h flag:

> opine-cli --help

  Usage:   opine-cli [option] [dir]
  Version: v1.0.2                  

  Description:

    Opine's application generator.

  Options:

    -h, --help                      - Show this help.                            
    -V, --version                   - Show the version number for this program.  
    -v, --view     <engine:string>  - add view <engine> support (ejs|eta)        
    -g, --git                       - add .gitignore                             
    -f, --force                     - force on non-empty directory

You can set up an Opine project in the current directory you are in without any view engine, just plain old CSS, HTML and JS, by simply running the CLI without any options or flags:

> opine-cli

   create : public/
   create : public/js/
   create : public/images/
   create : public/css/
   create : routes/
   create : public/index.html
   create : public/css/style.css
   create : routes/index.ts
   create : mod.ts
   create : routes/users.ts
   create : app.ts
   create : deps.ts

   run the app:
     $ deno run --allow-net --allow-read --allow-env mod.ts

The CLI will list all of the files and directories that it creates, and at the end of the output explains how you can run the application.

You can also specify a directory to use by providing it to the CLI as an argument (it will be created if it does not exist):

> opine-cli hello-opine

   create : hello-opine/
   create : hello-opine/public/
   create : hello-opine/public/js/
   create : hello-opine/public/images/
   create : hello-opine/public/css/
   create : hello-opine/routes/
   create : hello-opine/public/index.html
   create : hello-opine/public/css/style.css
   create : hello-opine/routes/index.ts
   create : hello-opine/mod.ts
   create : hello-opine/routes/users.ts
   create : hello-opine/app.ts
   create : hello-opine/deps.ts

   change directory:
     $ cd hello-opine

   run the app:
     $ deno run --allow-net --allow-read --allow-env mod.ts

The Opine CLI also offers the ability to set up a view / template engine within your project. To add view logic, simply provide a --view or -v flag followed by a supported view engine.

For example, we can opt to use the eta view engine in our Opine web project by running:

> opine-cli opine-eta-example --view eta

   create : opine-eta-example/
   create : opine-eta-example/public/
   create : opine-eta-example/public/js/
   create : opine-eta-example/public/images/
   create : opine-eta-example/public/css/
   create : opine-eta-example/routes/
   create : opine-eta-example/views/
   create : opine-eta-example/routes/users.ts
   create : opine-eta-example/public/css/style.css
   create : opine-eta-example/mod.ts
   create : opine-eta-example/routes/index.ts
   create : opine-eta-example/views/error.eta
   create : opine-eta-example/app.ts
   create : opine-eta-example/views/index.eta
   create : opine-eta-example/deps.ts

   change directory:
     $ cd opine-eta-example

   run the app:
     $ deno run --allow-net --allow-read --allow-env --unstable mod.ts

Notice how by adding the --view flag the CLI has now introduced a views directory with our templates, and the app.ts is automatically set up to use the provided engine.

Running the application

Let's use the CLI to build an Opine web application that uses ejs view templates via the dejs Deno module, and put in a new ./opine-ejs-example directory:

> opine-cli opine-ejs-example --view ejs

   create : opine-ejs-example/
   create : opine-ejs-example/public/
   create : opine-ejs-example/public/js/
   create : opine-ejs-example/public/images/
   create : opine-ejs-example/public/css/
   create : opine-ejs-example/public/css/style.css
   create : opine-ejs-example/routes/
   create : opine-ejs-example/routes/index.ts
   create : opine-ejs-example/routes/users.ts
   create : opine-ejs-example/views/
   create : opine-ejs-example/views/error.ejs
   create : opine-ejs-example/views/index.ejs
   create : opine-ejs-example/mod.ts
   create : opine-ejs-example/app.ts
   create : opine-ejs-example/deps.ts

   change directory:
     $ cd opine-ejs-example

   run the app:
     $ deno run --allow-net --allow-read --allow-env mod.ts

Then we navigate to the new project directory:

> cd opine-ejs-example

And now we can run our Opine web application using the provided command:

> deno run --allow-net --allow-read --allow-env mod.ts

Let's load http://localhost:3000/ in your browser to check out the app. You should see something like:

Amazing! 🎉 🎉 You now have a working and running Opine application. Great Job!

Inspiration: This article draws some pointers from the Express web framework series.

Top 6 Web Security Take-Aways From Google CTF 2020

Craig Morten — Thu, 03 Sep 2020 17:47:52 +0000

A few weekends ago Google hosted it's annual Capture The Flag (CTF) competition: a set of computer security challenges involving reverse-engineering, cryptography, web technologies, and much more.

The aim of a CTF is to solve challenges by exploiting vulnerabilities in the provided application, server etc. in order to find a "flag", usually an unguessable string, which can be traded for points. Teams try to find flags and receive points during the limited competition time window, so they rise up the leaderboard. Generally top teams can receive prizes, or are invited to a finals round (which happens in the Google CTF).

In this post I will cover 5 of my top web security take-aways from the Google CTF web challenges. I won't go into full detail for every challenge, but instead focus on the vulnerabilities themselves and what you can do to prevent similar security holes in your applications.

If you are interested in full challenge write-ups I recommend you check out the CTFTime Writeups. Google also post past challenges and solutions on the Google CTF GitHub.

So let's check out some vulnerabilities!

1. Avoid Writing Custom Sanitizer Code

Google's beginner challenge for the CTF involved creating "pastes" which could then be shared with another user.

Most challenges involving user inputted content which is then reflected back to the user, and potentially other users, is almost certainly a cross-site scripting [OWASP 7 - XSS] challenge. Indeed, being a beginner challenge Google gave a fairly big clue in the page source with a comment including a backlog ticket number for fixing an XSS bug:

In this instance, the pasted content gets passed through the DOMPurify library's sanitize() method, which in this case does not have a known vulnerability. The reference to /source combined with our pasted contents being added to the note variable hints at attacking the server code, which for this challenge was provided.

It is in the server source code that we find the Googlers have created their own custom sanitizer method:

/* Who wants a slice? */
const escape_string = unsafe => JSON.stringify(unsafe).slice(1, -1)
  .replace(/</g, '\\x3C').replace(/>/g, '\\x3E');

The intent is clear: our note is to be written into a double quoted string using ejs templates, so first off a quick way to ensure strings are escaped (and therefore can't close off a set of quotes and perform XSS) is to use JSON.stringify() which will add backslashes to quotes (i.e. \") in any passed string. Indeed if we copy this function into a JS REPL (e.g. Node prompt or developer tools console) we can see a payload of - " - becomes - \" -.

The .slice(1, -1) operation then removes the first and last character from the output of JSON.stringify(), which for a string are double quotes. The last two replaces then escape all triangle brackets characters so to prevent you closing / adding script tags.

At first this might seem like a neat trick for escaping inputs - it certainly seems to work for any payload you can paste into the challenge's website, and it's neat and short. Unfortunately it's made a fundamental flaw in a key assumption about the user's input: that it will always be a string.

Passing an array (e.g. ['- " -']) to the above method you'll instantly notice a difference. Instead of the first and last characters being a double quote, they are now square brackets which leaves a pair of unescaped double quotes as the first and last characters of the remaining string.

This means passing a payload of ["; alert('xss'); //"] would allow us to bypass this custom sanitizer and execute an XSS attack. Passing an array is possible because the Express server has the extended bodyParser.urlencoded() middleware enabled, allowing us to pass malicious payload in a POST body by using the extended syntax content[]=; alert('xss'); //.

Ultimately this is a manifestation of OWASP 8 - Insecure Deserialization. An insecure parser of a payload allowing attackers to perform a secondary XSS attack. 💥

Suggestions

Where possible, always use well tested third party sanitizer libraries which cover all possible inputs. Avoid custom sanitizers as it is very easy miss something.
Reduce the allowed Accept types to a know allowlist for API endpoints to reduce the scope of user payloads. For example, don't use unnecessary or overscoped body parsing middleware.
Validate user payloads for type and contents and consider returning with 400 Bad Request like responses for invalid payloads. Using libraries such as express-validator can help make this simple.

Funnily enough, we find that the log-me-in web challenge suffers from very similar issues to the beginner challenge. Extended body parser syntax allows the attacker to bypass sanitization and leads to an SQL Injection [OWASP 1]. I add this note to encourage you to still be cautious with well maintained third party libraries, as they may still have gotchas despite seeming to perform sanitization. Specific payload validation will always harden your server's endpoints. 🔥

2. Beware Of document.referrer

A gotcha that caught even the Google CTF creators out is the existence of the document.referrer property.

This property is set to either:

An empty string in the case of direct navigation;
The URL of the page where you navigated from, similar to the Referer header.
The same value as the href of the parent window's document.location when inside an iframe.

In the case of the Tech Support challenge, the last property setting meant that an iframe vulnerable to XSS leaked the credentials of the admin user, as the frame inherited the parent window's href in this referrer property [OWASP 3 - Sensitive Data Exposure]. 😢

// Detect dark theme var iframe = document.getElementById('tweet-1297882502661640193-685'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1297882502661640193&theme=dark" }

Suggestions

Avoid plaintext (or otherwise) credentials in any part of the URL, ideally for all pages, but especially for any public facing pages or pages containing iframes with a public interface.
Educate your users about credential security and management best practices.

3. Avoid User Inputted HTML If Can!

The least solved web challenge with only 10 completions was the Safe HTML Paste challenge. This challenge was remarkably similar to the beginner Pasteurized challenge mentioned above, and allowed you to paste arbitrary content, view it and share it with an admin user.

Unlike the beginner challenge the server code was off-limits and appeared to be rock-solid. What this CTF demonstrated was how difficult it is to correctly sanitize arbitrary HTML, and how even a popular and well maintained library such as the Google Closure Library can have weaknesses. Further still, it demonstrates how easy it is to use a library that has a well documented vulnerability and patched version and yet be using an outdated and vulnerable version! [OWASP 9 - Using Components with Known Vulnerabilities]

The attack is well documented in this write-up and this research if you are interested to go through the DOM mutation details. 😄

Suggestions

Avoid user inputted HTML content when possible.
Always use the latest versions and patches of third party libraries.
Regularly audit your libraries and their dependencies using tools like retire.js or snyk.

4. Self-XSS Should Not Be Ignored

Coming back to the Tech Support challenge, the intended vulnerability path had a very interesting message - self-XSS when paired with cross-site request forgery (CSRF) can lead to dangerous session-hijacking.

In the challenge we find that the lack of CSRF controls on the login allows us to force the victim to join our session in a frame which subsequently runs a self-XSS.

Given the logged in frame is running in the victim's context, the self-XSS is granted privileged access to sibling frames allowing the attacker to manipulate, or in this case scrape, pages already open with the victim's previous session.

// Detect dark theme var iframe = document.getElementById('tweet-1297885195702501376-959'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1297885195702501376&theme=dark" }

This kind of vulnerability is sufficiently open that you don't even require a third-party domain to send leaked data to! See the video below of an attack on the challenge that uses the self-XSS to store the flag in the attacker's address field.

// Detect dark theme var iframe = document.getElementById('tweet-1299007923885146113-710'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1299007923885146113&theme=dark" }

Suggestions

Enable CSRF protection on all forms, or ideally at least on any authentication / login flows.
Close out any self-XSS vulnerabilities to prevent paired / secondary attacks.
Enable strict content security policies (CSP) to prevent inline script execution without CSRF protection (e.g. nonce tokens).

5. Prototype Pollution Is A Real Issue

Similar to the Tech Support, the All The Little Things challenge also had an unintended solution.

// Detect dark theme var iframe = document.getElementById('tweet-1297796633552400384-706'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1297796633552400384&theme=dark" }

One of the issues of this challenge was that user inputted content (via the window.name property) was able to pollute the prototype of a heavily relied upon object by using the __proto__ property.

Prototype pollution can be a serious problem, especially in server-side authentication flows where attackers can attack to mutate the prototype to escalate their privileges. Several well known libraries such as Lodash have also been caught out as recently as this year making this a very current and real issue.

In the case of this challenge, it was interesting to see the vulnerability exposed client-side and is yet another clear warning for website maintainers to always sanitize and validate any user input, no matter how inconspicuous!

Suggestions

Getting similar to previous ones now!

Where possible, always use well tested third party sanitizer libraries which cover all possible inputs. Avoid custom sanitizers as it is very easy miss something.
Always use the latest versions and patches of third party libraries. Be sure to regularly audit your libraries and their dependencies using tools like retire.js or snyk.
Where performing custom object assignment, merging or otherwise, ensure that you are deny-listing malicious keys such as __proto__, constructor, and any variations thereof which might allow an attacker to change the intended property values of an object.

6. Parantheses-less XSS Attacks In Strict CSP

The final learning point from the Google CTF was the discovery of parentheses-less XSS attacks. I recommend you check out the below Medium article by the challenge creator @terjanq.

Arbitrary Parentheses-less XSS. against strict CSP policies | by terjanq | Medium

terjanq ・ Aug 17, 2023 ・
Medium

Ultimately what we learn is that even in a setting as restricted as a JSONP callback, where almost all characters have been restricted, it is still possible to execute arbitrary XSS. In fact, there are several different attack payloads depending on the situation that can be used - check out this GitHub repo of example payloads.

What this shows is that even in restricted content security policy (CSP) situations, even the smallest XSS vulnerability can be exploited and escalated to an arbitrary attack.

Suggestions

Ensure no XSS exploits possible on your site (to the best of your ability!). Check out tools like OWASP ZAP to help with discovering issues - be sure to always have the permission of the target site first before running any penetration tools!
Perform strict validation on potential user input. In this challenge, restricting the allowed JSONP callback values to a defined enum of strings would have prevented the exploit.

That's all for now folks! Hope it made for interesting reading. 😄

Did you take part in the Google CTF this year? If so, what was your favourite challenge? What security points did you learn? I would love to hear your comments, ideas and suggestions - drop a note in the section below.

Till next time, stay secure! 🎤

K8s Chaos Dive: Chaos-Mesh Part 2

Craig Morten — Tue, 25 Aug 2020 14:45:11 +0000

In this series I walk through several different open source offerings for performing chaos testing / engineering within your Kubernetes clusters.

In K8s Chaos Dive: Chaos-Mesh Part 1 I covered an introduction to Chaos-Mesh in which we installed the toolkit into our cluster and used a PodChaos experiment to kill off a random percentage of Nginx pods on a schedule.

K8s Chaos Dive: Kube-Monkey

Craig Morten ・ Aug 19 '20 ・ 10 min read

#kubernetes #devops #testing #tutorial

Killing pods can be a great exercise for validating resiliency to pod death, something that can happen for a list of reasons in Kubernetes. Solutions tend to revolve around horizontal scaling (dependent on your target SLA):

Ensure you have sufficient replicas for your application to handle < 100% failure - e.g. Node restarts, Node network disruption causes health-checks to fail. For those wanting to achieve a ~99.9% uptime SLA (depending on cloud provider).
Ensure you have a multi-cluster setup across availability zones to handle 100% failure in a single data-centre - e.g. data-centre power-outage. For those wanting to achieve a ~99.99% uptime SLA (depending on cloud provider).
Ensure you have a multi-region setup to handle 100% failure in a single region. For those wanting to achieve a ~99.999% uptime SLA (depending on cloud provider).

However, short of planned restarts by yourself or your cloud provider, generally it is good to try and avoid your pods dying everywhere! Pod death is generally a secondary effect of some other root cause in the cluster, and it may be that finding and fixing a root issue may save you from having to pay for excessive redundancy in compute.

In this post we're going to take a look at StressChaos experiments in the Chaos-Mesh toolkit to see how we can validate our CPU and memory resourcing.

StressChaos

Introduction

Chaos-Mesh offers two main supported forms of stress chaos:

cpu-burn - Simulate pod CPU stress.
memory-burn - Simulate pod memory stress.

For example, to generate a StressChaos which will burn 100% of 1 CPU for 30 seconds, every 5 minutes, for one of your pods in the my-app namespace, you could write:

apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: burn-cpu
  namespace: chaos-mesh
spec:
  mode: one
  selector:
    namespaces:
      - my-app
  stressors:
    cpu:
      workers: 1
      load: 100
  duration: "30s"
  scheduler:
    cron: "@every 5m"

Equally, to generate a StressChaos which will continually grow memory usage for 30s, every 5 minutes, for one of your pods in the my-app namespace, you could write:

apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: burn-memory
  namespace: chaos-mesh
spec:
  mode: one
  selector:
    namespaces:
      - my-app
  stressors:
    memory:
      workers: 1
  duration: "30s"
  scheduler:
    cron: "@every 5m"

Chaos-Mesh injects these desired stressors directly into the target pods via the chaos-daemon, which uses some nsenter magic to pass appropriate stress-ng commands.

You can further customize your stressors as follows:

You can provide both a cpu and memory and stressor in your config.
Both the cpu and memory stressors support an additional options field where you can pass additional flags / arguments to the stress-ng command.
Alternatively, you can define fully custom stressors by providing a stressngStressors string instead of the stressors object. This expects a string of stress-ng flags / arguments.

Note that providing custom options (last two points) are not fully tested so should be seen as experimental!

So without further ado, let's try out some stress chaos!

Walk-through

Further details on Chaos-Mesh can be found on it's GitHub repository and in the documentation.

Here we'll walk through setting up and executing the following two tests:

A CPU stress test using Kubernetes manifest files.
A Memory stress test using Kubernetes manifest files.

Setting Up The Cluster

Please refer to previous tutorials for detailed instructions! Ultimately we run the following two commands:

minikube start --driver=virtualbox
minikube addons enable metrics-server

Deploying A Target Application And Chaos-Mesh

If you are following on from part 1 of the Chaos-Mesh series then you may not have to do anything! We will re-use our Nginx and Chaos-Mesh deployments from before.

If you've skipped over the previous tutorial then I recommend following it's target application setup and deploying chaos-mesh instructions. If you followed the clean-up in the previous tutorial, then you can reinstate the setup with the following commands:

# Create Nginx Helm charts
helm create nginx

# Deploy Nginx to the cluster
kubectl create ns nginx
helm upgrade --install nginx ./nginx \
  -n nginx \
  --set replicaCount=10

# Check the Nginx deployment is successful
helm ls -n nginx
kubectl get pod -n nginx

# Pull down the Chaos-Mesh repo
git clone https://github.com/chaos-mesh/chaos-mesh

# Apply Chaos-Mesh Custom Resource Definitions
kubectl apply -f ./chaos-mesh/manifests/crd.yaml

# Deploy Chaos-Mesh to the cluster
kubectl create ns chaos-mesh
helm upgrade --install chaos-mesh ./chaos-mesh/helm/chaos-mesh \
  -n chaos-mesh \
  --set dashboard.create=true

# Check the Chaos-Mesh deployment is successful
helm ls -n chaos-mesh
kubectl get pods -n chaos-mesh -l app.kubernetes.io/instance=chaos-mesh

# Load the Chaos-Mesh dashboard
minikube service chaos-dashboard -n chaos-mesh

Experiment 2: CPU Chaos

Let's start by creating a new stress-cpu.yaml file with the following contents:

apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: stress-cpu
  namespace: chaos-mesh
spec:
  mode: all
  selector:
    labelSelectors:
      "app.kubernetes.io/name": nginx
    namespaces:
    - nginx
  stressors:
    cpu: 
      workers: 1
      load: 100
  duration: "30s"
  scheduler:
    cron: "@every 5m"

This will add a single CPU stress worker to each of our nginx pods that will load the pods 100% of the scheduled time. Check out the stress-ng manpage for the --cpu and --cpu-load flags for further details.

These workers will be scheduled every 5 minutes and run for 30s in the pod before being stopped, meaning we should expect to see our Nginx pods' CPU spike for 30s and then drop back to near 0 for the remaining 4.5 minutes.

Let's install our experiment and see what happens!

$ kubectl apply -f stress-cpu.yaml

stresschaos.chaos-mesh.org/stress-cpu created

If we check the Chaos-Mesh dashboard we can see that the experiment has been deployed and is running:

minikube service chaos-dashboard -n chaos-mesh

The dashboard shows us that the test is successfully running for 30 seconds every 5 minutes 🎉.

Let's say we don't trust the dashboard? Let's check out what the CPU levels are actually like in the cluster!

Firstly in a terminal window install the watch command if you don't already have it and run:

watch kubectl top pods -n nginx

This should output something like:

Every 2.0s: kubectl top pods -n nginx              c-machine: Tue Aug 25 14:16:40 2020

NAME                     CPU(cores)   MEMORY(bytes)
nginx-5c96c8f58b-2l85d   0m           25Mi
nginx-5c96c8f58b-btws8   0m           2Mi
nginx-5c96c8f58b-c28ws   0m           2Mi
nginx-5c96c8f58b-gmnkc   0m           2Mi
nginx-5c96c8f58b-hzgx5   0m           2Mi
nginx-5c96c8f58b-jzk2p   0m           2Mi
nginx-5c96c8f58b-vhmqm   0m           2Mi
nginx-5c96c8f58b-vs2cr   0m           21Mi
nginx-5c96c8f58b-zc9qf   0m           2Mi
nginx-5c96c8f58b-zjfc4   0m           2Mi

Which will update every 2 seconds. There can be a bit of a lag getting metrics back from the cluster's metrics-server, but we should see that every 5 minutes these values jump up for ~30 seconds and then return back to 0. Waiting for the next scheduled stress test we can see this happening:

Every 2.0s: kubectl top pods -n nginx       c-machine: Tue Aug 25 14:19:11 2020

NAME                     CPU(cores)   MEMORY(bytes)
nginx-5c96c8f58b-2l85d   110m         25Mi
nginx-5c96c8f58b-btws8   80m          2Mi
nginx-5c96c8f58b-c28ws   94m          2Mi
nginx-5c96c8f58b-gmnkc   89m          2Mi
nginx-5c96c8f58b-hzgx5   70m          2Mi
nginx-5c96c8f58b-jzk2p   132m         2Mi
nginx-5c96c8f58b-vhmqm   63m          2Mi
nginx-5c96c8f58b-vs2cr   115m         21Mi
nginx-5c96c8f58b-zc9qf   80m          2Mi
nginx-5c96c8f58b-zjfc4   94m          2Mi

Awesome! The single CPU worker in each pod seems to use up around 100m (100 millicores or 0.1 CPU cores) 💥.

If you wanted to apply more load to your pods, you could just up the number of workers in the StressChaos yaml accordingly 🙃.

If you want to go deeper, we can even check out what is happening in the pod itself! 😲 First we find a target Nginx pod:

$ kubectl get pods -n nginx

NAME                     READY   STATUS    RESTARTS   AGE
nginx-5c96c8f58b-2l85d   1/1     Running   0          23m
nginx-5c96c8f58b-btws8   1/1     Running   0          23m
nginx-5c96c8f58b-c28ws   1/1     Running   0          23m
nginx-5c96c8f58b-gmnkc   1/1     Running   0          23m
nginx-5c96c8f58b-hzgx5   1/1     Running   0          23m
nginx-5c96c8f58b-jzk2p   1/1     Running   0          23m
nginx-5c96c8f58b-vhmqm   1/1     Running   0          23m
nginx-5c96c8f58b-vs2cr   1/1     Running   0          23m
nginx-5c96c8f58b-zc9qf   1/1     Running   0          23m
nginx-5c96c8f58b-zjfc4   1/1     Running   0          23m

We'll use the first in the list nginx-5c96c8f58b-2l85d. Next we can actually open a shell inside the pod as follows:

$ kubectl exec -it nginx-5c96c8f58b-2l85d -n nginx -- bash

root@nginx-5c96c8f58b-2l85d:/#

From here we can install some utility packages:

root@nginx-5c96c8f58b-2l85d:/# apt-get update && apt-get install -y procps

Which will allow us to use the ps and top commands to interrogate the processes running inside the pod. If we wait until load is happening, we can see a stress-ng processes start running inside the container with:

root@nginx-5c96c8f58b-2l85d:/# watch ps aux

Every 2.0s: ps aux                                                                                 nginx-5c96c8f58b-9npqq: Tue Aug 25 13:48:56 2020

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.2  32656  5204 ?        Ss   13:36   0:00 nginx: master process nginx -g daemon off;
nginx        6  0.0  0.1  33112  3072 ?        S    13:36   0:00 nginx: worker process
root        27  0.0  0.1  18136  3284 pts/0    Ss   13:46   0:00 bash
root       355  0.1  0.1  11104  2520 pts/0    S+   13:48   0:00 watch ps aux
root       383  0.0  0.1  18700  3696 ?        SL   13:48   0:00 stress-ng --cpu 1 --cpu-load 100
root       384 21.0  0.2  19348  4668 ?        R    13:48   0:01 stress-ng --cpu 1 --cpu-load 100
root       397  0.0  0.0  11104   652 pts/0    S+   13:48   0:00 watch ps aux
root       398  0.0  0.0   4280   772 pts/0    S+   13:48   0:00 sh -c ps aux
root       399  0.0  0.1  36636  2804 pts/0    R+   13:48   0:00 ps aux

Or equally we can use top to see our processes and their resource usage with a slightly nicer UI:

root@nginx-5c96c8f58b-2l85d:/# top

From this command we can see a process list which shows a stress-ng parent process and a stress-ng-cpu worker process which consumes around 18% CPU. After 30s these processes disappear and CPU returns to near 0%.

Let's stop our experiment by removing the StressChaos Kubernetes object we added earlier:

kubectl delete -f stress-cpu.yaml

So there we have it, we can successfully stress our pods with increased CPU levels! 🍾

Experiment 3: Memory Chaos

Now let's perform a similar experiment, but this time we will load the memory of a single pod.

Create a new StressChaos manifest called stress-memory.yaml:

apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: stress-memory
  namespace: chaos-mesh
spec:
  mode: one
  selector:
    labelSelectors:
      "app.kubernetes.io/name": nginx
    namespaces:
    - nginx
  stressors:
    memory: 
      workers: 1
  duration: "10s"
  scheduler:
    cron: "@every 2m"

This will add a single memory stress worker to one of our nginx pods that will run for 10s every 2 minutes. The worker will grow it's heap by reallocating memory at a rate of 64K per iteration. Check out the stress-ng manpage for the --bigheap flag for further details.

Let's install our experiment and see what happens!

$ kubectl apply -f stress-memory.yaml

stresschaos.chaos-mesh.org/stress-memory created

Watching our Nginx pods, we can see that every 5 minutes one of the pods has an outburst in memory usage:

$ watch kubectl top pods -n nginx
Every 2.0s...  c-machine: Tue Aug 25 15:12:52 2020

NAME                     CPU(cores)   MEMORY(bytes)
nginx-5c96c8f58b-5tvz5   0m           3Mi
nginx-5c96c8f58b-6fb9h   0m           2Mi
nginx-5c96c8f58b-bjzd2   0m           2Mi
nginx-5c96c8f58b-gfqww   0m           3Mi
nginx-5c96c8f58b-jtn5f   0m           2Mi
nginx-5c96c8f58b-rdlgk   145m         3Mi
nginx-5c96c8f58b-sgx4s   0m           2Mi
nginx-5c96c8f58b-szffx   0m           3Mi
nginx-5c96c8f58b-ttlhr   65m          621Mi
nginx-5c96c8f58b-xgh7j   0m           2Mi

Similarly, if we're quick and exec on a targetted pod and install top we can see that the memory worker very quickly starts to consume a lot of memory!

The image above shows our targeted Nginx pod hitting 30.4% of the available memory. Given that we haven't set any resource limits on our Nginx deployment, that is actually 30.4% of the cluster's memory allocated to Minikube!!

This is why in this experiment I've set the mode to one and not all. Doing so will likely consume all of the memory resources in your cluster and very potentially break it - I tried this initially and had to completely delete the Minikube cluster and start again as it became 100% non-responsive! 😱

Let's remove our experiment:

kubectl delete -f stress-memory.yaml

Clean-up

Let's clean-up and remove everything we've created today:

helm delete chaos-mesh -n chaos-mesh
kubectl delete ns chaos-mesh
kubectl delete crd iochaos.chaos-mesh.org
kubectl delete crd kernelchaos.chaos-mesh.org 
kubectl delete crd networkchaos.chaos-mesh.org
kubectl delete crd podchaos.chaos-mesh.org 
kubectl delete crd podnetworkchaos.chaos-mesh.org
kubectl delete crd stresschaos.chaos-mesh.org
kubectl delete crd timechaos.chaos-mesh.org
helm delete nginx -n nginx
kubectl delete ns nginx
minikube stop
minikube delete

Learnings

So what have these two experiments taught us today?

Our first experiment showed that it is very simple to apply additional CPU strain to our pods, but ultimately 1 worker didn't apply too much pressure. Given we weren't actually trying to use our Nginx pods in any way the experiment is a little moot and there is likely little to be gained.

However considering CPU is very important in a cluster, and for many applications the CPU profile can directly impact key performance indicators such as latency. By having the tooling to generate artificial strain on your pods (and cluster) you can measure how your applications work when under the kind of pressure you see during a peak load period.

Given the impact on latency, CPU stress can also be a good way to validate your deployment's healthchecks, where the aim should be to ensure that your application is highly available and that traffic is routed to happily working pods opposed to ones having a CPU meltdown! This could be tested using one of the percentage based modes available.

Monitoring your application under pod CPU stress can also be a useful way to validate your pod resource and limit values. Applying too small a resource value for CPU will likely result in CPU throttling which can have devastating impacts on applications, particularly multi-threaded applications.

Our second experiment showed that memory stressor is certainly quite aggressive! In fact, leaving it running too long on too many pods could result in quite the flurry of out of memory killer jobs, if not entire cluster resource exhaustion and failure!

Fine-tuning the memory stressor using the options value is probably advised, but a key take away is that no one pod (or collection of pods) should be allowed to consume an entire cluster's memory allocation! Strict memory resource request and limits should be set on deployments to ensure that they are killed before they have the chance to disrupt other pods or fundamental cluster processes 🔥.

Thanks for reading folks!

Enjoy the tutorial? Have questions or comments? Or do you have an awesome way to run chaos experiments in your Kubernetes clusters? Drop me a message in the section below or tweet me @CraigMorten!

Till next time 💥