How to test dark mode accessibility in CI with Playwright and axe-core

If you run axe in CI, it is almost certainly testing one state: light colour scheme, no motion preference, forced colors off, desktop viewport. That is what a headless browser boots into, and axe can only evaluate what was actually rendered.

Here is how to test the others with Playwright, and the four traps that make the naive version report nothing useful.

The one-line part

Playwright sets user preferences at the browser-context level:

const context = await browser.newContext({
  colorScheme: 'dark',              // prefers-color-scheme: dark
  reducedMotion: 'reduce',          // prefers-reduced-motion: reduce
  forcedColors: 'active',           // forced-colors: active (Windows High Contrast)
  viewport: { width: 320, height: 1024 }   // the reflow width WCAG 1.4.10 requires
})

You can also flip it on an existing page with page.emulateMedia({ colorScheme: 'dark' }).

Then inject axe and run it:

const fs = require('fs')
const AXE = fs.readFileSync(require.resolve('axe-core/axe.min.js'), 'utf8')

const page = await context.newPage()
await page.goto(url, { waitUntil: 'load' })
await page.waitForTimeout(400)      // let webfonts swap; they move contrast values
await page.addScriptTag({ content: AXE })

const results = await page.evaluate(async () =>
  await axe.run(document, { runOnly: { type: 'tag', values: ['wcag2a','wcag2aa','wcag21a','wcag21aa'] } }))

That is the whole mechanism. Now the parts that bite.

Trap 1: the worst contrast defects are not in violations

I shipped a button whose label was the same colour as its own background. 1:1 contrast — literally unreadable. In dark mode only. CI stayed green even after I started rendering dark mode.

Here is the probe output, same page, two schemes:

=== scheme: light ===
color: 'rgb(255, 255, 255)'   background: 'rgb(28, 93, 63)'
violations: []      incomplete: []

=== scheme: dark ===
color: 'rgb(108, 196, 154)'   background: 'rgb(108, 196, 154)'
violations: []      incomplete: [ 'color-contrast x1' ]

Text identical to its background lands in incomplete, not violations.

axe is right to do this. Matching foreground to background is a legitimate way to hide something — a visually-hidden label, a fade-in that has not started, print-only text. axe reports what it can prove and hands the rest to a human.

It is the wrong default for a pipeline, because pipelines assert on violations and drop incomplete. The failure mode is asymmetric in the worst direction: a 4.3:1 ratio is a confident violation, a 1:1 ratio is a question.

So collect both, and keep them clearly separate:

const findings = []
for (const [kind, list] of [['violation', results.violations], ['incomplete', results.incomplete]])
  for (const rule of list || [])
    for (const node of rule.nodes)
      findings.push({ kind, rule: rule.id, target: node.target.join(' ') })

Do not set resultTypes: ['violations']. It looks like a harmless optimisation. It makes axe return a single representative node for every other bucket, so your incomplete results get silently truncated before you ever see them.

Trap 2: a strict CSP blocks the scan entirely

page.addScriptTag({ content }) injects an inline script. Any site with a strict script-src refuses it, and the run fails at the first page.

The sites that ship a strict CSP are disproportionately the ones with a security review and a design system — which is to say, the ones worth scanning. Fix:

const context = await browser.newContext({ bypassCSP: true, /* ... */ })

Trap 3: raw counts are useless, deltas are not

Running seven states and printing 40 findings gives someone a chore. Running seven states and printing these 3 exist in dark mode and not in your baseline gives them a bug report with a cause attached.

Compare each state against the baseline and report only the difference:

const key = f => f.kind + '|' + f.rule + '|' + f.target
const baselineKeys = new Set(baselineFindings.map(key))
const newInThisState = stateFindings.filter(f => !baselineKeys.has(key(f)))

Two things to get right here, and I got both wrong first:

Change one variable at a time. A full cross-product of six preferences is 64 runs and tells you the page is broken without telling you which preference broke it. Six single-factor runs name the cause.

Deduplicate across states before counting. A defect caused by a narrow layout appears in your mobile run and your 320px run and any dark+mobile combination. Summing per-state counts reports one element three times. The tell in my own data was a page showing exactly 30 incomplete results in each of three states — not similar numbers, identical ones. My first write-up would have claimed 210 findings on a site that had 72.

Trap 4: real pages are not deterministic

Carousels, rotating promos, A/B tests and lazy-loaded media mean two identical loads do not always produce identical output. Before believing any delta, measure your noise floor: run the same state twice, change nothing, and see what differs.

Across 68 public sites I found 62 were byte-identical between two identical runs. The other six were not — one shopping site produced 715 apparent findings and a churn of +122/−125 between two identical loads. That is a product carousel, not a defect. Without that control I would have published the 715.

What this actually finds

I ran this across 70 public homepages — standards bodies, framework docs, design systems, plus EU government portals, banking, transport and e-commerce.

42 of 62 stable sites (68%) had at least one finding a baseline-only run never surfaced. 26 of 62 (42%) had a violation, not merely something needing review. Median one per site.

Eleven were clean in all seven states, including W3C, MDN, GOV.UK and three EU government portals.

Worth stating plainly: axe detects a minority of WCAG failures — 57% by issue volume in Deque's own study, roughly a third by success criteria. None of this replaces manual testing, and a finding is not a failure.

If you would rather not wire it up

All of the above is packaged in a11y-matrix — MIT, no account, no telemetry:

npx github:henriqueyuri00/a11y-matrix https://your-site.example
npx github:henriqueyuri00/a11y-matrix --sitemap https://your-site.example/sitemap.xml

Across multiple pages it deduplicates by element, so a header defect on 18 pages is reported once as every page rather than eighteen times. The study directory has the method, the control run and the raw per-site data.

---

Your pipeline being green is worth exactly as much as the states you rendered and the buckets you read.

Comments

[Alex Shev]

Dark mode accessibility is easy to miss because the light theme often gets all the review attention. Putting it in CI is a good move, especially if screenshots and contrast checks cover real component states rather than only the happy-path page.