DEV Community: Henrique Yuri

How to test dark mode accessibility in CI with Playwright and axe-core

Henrique Yuri — Tue, 11 Aug 2026 03:28:53 +0000

If you run axe in CI, it is almost certainly testing one state: light colour scheme, no motion preference, forced colors off, desktop viewport. That is what a headless browser boots into, and axe can only evaluate what was actually rendered.

Here is how to test the others with Playwright, and the four traps that make the naive version report nothing useful.

The one-line part

Playwright sets user preferences at the browser-context level:

const context = await browser.newContext({
  colorScheme: 'dark',              // prefers-color-scheme: dark
  reducedMotion: 'reduce',          // prefers-reduced-motion: reduce
  forcedColors: 'active',           // forced-colors: active (Windows High Contrast)
  viewport: { width: 320, height: 1024 }   // the reflow width WCAG 1.4.10 requires
})

You can also flip it on an existing page with page.emulateMedia({ colorScheme: 'dark' }).

Then inject axe and run it:

const fs = require('fs')
const AXE = fs.readFileSync(require.resolve('axe-core/axe.min.js'), 'utf8')

const page = await context.newPage()
await page.goto(url, { waitUntil: 'load' })
await page.waitForTimeout(400)      // let webfonts swap; they move contrast values
await page.addScriptTag({ content: AXE })

const results = await page.evaluate(async () =>
  await axe.run(document, { runOnly: { type: 'tag', values: ['wcag2a','wcag2aa','wcag21a','wcag21aa'] } }))

That is the whole mechanism. Now the parts that bite.

Trap 1: the worst contrast defects are not in `violations`

I shipped a button whose label was the same colour as its own background. 1:1 contrast — literally unreadable. In dark mode only. CI stayed green even after I started rendering dark mode.

Here is the probe output, same page, two schemes:

=== scheme: light ===
color: 'rgb(255, 255, 255)'   background: 'rgb(28, 93, 63)'
violations: []      incomplete: []

=== scheme: dark ===
color: 'rgb(108, 196, 154)'   background: 'rgb(108, 196, 154)'
violations: []      incomplete: [ 'color-contrast x1' ]

Text identical to its background lands in incomplete, not violations.

axe is right to do this. Matching foreground to background is a legitimate way to hide something — a visually-hidden label, a fade-in that has not started, print-only text. axe reports what it can prove and hands the rest to a human.

It is the wrong default for a pipeline, because pipelines assert on violations and drop incomplete. The failure mode is asymmetric in the worst direction: a 4.3:1 ratio is a confident violation, a 1:1 ratio is a question.

So collect both, and keep them clearly separate:

const findings = []
for (const [kind, list] of [['violation', results.violations], ['incomplete', results.incomplete]])
  for (const rule of list || [])
    for (const node of rule.nodes)
      findings.push({ kind, rule: rule.id, target: node.target.join(' ') })

Do not set resultTypes: ['violations']. It looks like a harmless optimisation. It makes axe return a single representative node for every other bucket, so your incomplete results get silently truncated before you ever see them.

Trap 2: a strict CSP blocks the scan entirely

page.addScriptTag({ content }) injects an inline script. Any site with a strict script-src refuses it, and the run fails at the first page.

The sites that ship a strict CSP are disproportionately the ones with a security review and a design system — which is to say, the ones worth scanning. Fix:

const context = await browser.newContext({ bypassCSP: true, /* ... */ })

Trap 3: raw counts are useless, deltas are not

Running seven states and printing 40 findings gives someone a chore. Running seven states and printing these 3 exist in dark mode and not in your baseline gives them a bug report with a cause attached.

Compare each state against the baseline and report only the difference:

const key = f => f.kind + '|' + f.rule + '|' + f.target
const baselineKeys = new Set(baselineFindings.map(key))
const newInThisState = stateFindings.filter(f => !baselineKeys.has(key(f)))

Two things to get right here, and I got both wrong first:

Change one variable at a time. A full cross-product of six preferences is 64 runs and tells you the page is broken without telling you which preference broke it. Six single-factor runs name the cause.

Deduplicate across states before counting. A defect caused by a narrow layout appears in your mobile run and your 320px run and any dark+mobile combination. Summing per-state counts reports one element three times. The tell in my own data was a page showing exactly 30 incomplete results in each of three states — not similar numbers, identical ones. My first write-up would have claimed 210 findings on a site that had 72.

Trap 4: real pages are not deterministic

Carousels, rotating promos, A/B tests and lazy-loaded media mean two identical loads do not always produce identical output. Before believing any delta, measure your noise floor: run the same state twice, change nothing, and see what differs.

Across 68 public sites I found 62 were byte-identical between two identical runs. The other six were not — one shopping site produced 715 apparent findings and a churn of +122/−125 between two identical loads. That is a product carousel, not a defect. Without that control I would have published the 715.

What this actually finds

I ran this across 70 public homepages — standards bodies, framework docs, design systems, plus EU government portals, banking, transport and e-commerce.

42 of 62 stable sites (68%) had at least one finding a baseline-only run never surfaced. 26 of 62 (42%) had a violation, not merely something needing review. Median one per site.

Eleven were clean in all seven states, including W3C, MDN, GOV.UK and three EU government portals.

Worth stating plainly: axe detects a minority of WCAG failures — 57% by issue volume in Deque's own study, roughly a third by success criteria. None of this replaces manual testing, and a finding is not a failure.

If you would rather not wire it up

All of the above is packaged in a11y-matrix — MIT, no account, no telemetry:

npx github:henriqueyuri00/a11y-matrix https://your-site.example
npx github:henriqueyuri00/a11y-matrix --sitemap https://your-site.example/sitemap.xml

Across multiple pages it deduplicates by element, so a header defect on 18 pages is reported once as every page rather than eighteen times. The study directory has the method, the control run and the raw per-site data.

Your pipeline being green is worth exactly as much as the states you rendered and the buckets you read.

I scanned 70 well-known sites in seven browser states. The default one missed something on 68% of them.

Henrique Yuri — Tue, 11 Aug 2026 00:03:07 +0000

Every automated accessibility check I have ever seen loads the page once, in whatever state the headless browser boots in: light colour scheme, no motion preference, forced colors off, desktop viewport.

That is one user. I wanted to know what it costs, so I measured it across 70 public homepages — W3C, MDN, React, Vue, Kubernetes, GitHub, Wikipedia, the BBC, a handful of design systems, and four sectors carrying an explicit EU legal accessibility duty: government portals, banking, passenger transport and e-commerce.

(This started as 36 mostly-technical sites. I extended it afterwards, and the numbers below are the larger run — the update is described at the end.)

Each site was loaded seven times. One run is the baseline: light, desktop, no preferences — what an ordinary pipeline tests. The other six each change exactly one variable away from it. A finding counts only if it exists in a non-baseline state and not in the baseline.

The result

70 attempted. 68 loaded — Allegro and Air France did not, and are excluded rather than counted as clean. Of those 68, 62 produced byte-identical output across two identical runs, and only those 62 are counted.


Had a finding the baseline never surfaced	42 / 62 — 68%
Had one axe classes as a violation, not "needs review"	26 / 62 — 42%
Distinct findings missed by the baseline	348
Median per site	1

The first pass, on 36 mostly-technical sites, returned 71% and 47%. Nearly doubling the sample and adding regulated sectors moved it to 68% and 42%. The finding replicated, which is worth more than either number by itself.

The median is the honest headline, not the 348. The distribution is badly skewed: half the sites have one or none, and a handful carry most of the volume — Vercel (72), Tailwind CSS (52), Radix UI (40), Wikipedia (31). Quoting the total alone would imply everyone is sitting on dozens. They are not.

Eleven sites were clean in all seven states: W3C, WebAIM, MDN, Playwright, Primer, GOV.UK, Service-Public (FR), Rijksoverheid (NL), Suomi.fi (FI), Fnac, SNCF Connect. Several either write the standard, teach it, or are bound by a public-sector duty and appear to be meeting it.

The control is the part that makes this a study

Real sites carry carousels, rotating promos, A/B tests and lazy-loaded media. Two identical loads do not necessarily produce identical output. Without measuring that, "unique to dark mode" might just mean a different hero image rendered.

So I ran the same state twice, changing nothing, on all 36 sites:

146 findings appeared and 156 disappeared — concentrated on six sites. The other 62 were byte-identical.

Those six are excluded from every figure above rather than quietly left in the denominator, and this is the part that earned the control its place. One shopping site produced 715 apparent findings and a churn of +122/−125 between two identical loads. That is a rotating product carousel, not a defect.

Excluding it moved its whole sector's total from 729 findings to 14. Had I skipped the control, that one number would have been the loudest thing in the study, and it would have been wrong.

What exposed the findings

Variable	Sites	Distinct findings
Narrow viewport (mobile / 320px reflow)	42	340
Dark colour scheme	38	303
Dark scheme alone — no narrow state found it	9	28
Reduced motion	3	8
Forced colors	0	0

The first two overlap heavily — a lot of sites break in dark mode and at 320px — which is why "dark scheme alone" is broken out. On those 9 sites the colour scheme is the only variable that can explain the finding.

By rule: color-contrast on 32 sites, then 9 sites whose document is genuinely wider than a 320px viewport — the WCAG 1.4.10 condition measured directly, since axe has no rule for it — then scrollable-region-focusable, link-in-text-block and button-name.

Forced colors found nothing at all on any stable site. Reporting that is the point: a negative result is a result.

I got the counting wrong first, and the error was a factor of three

Worth writing down, because the wrong way is the obvious way.

My first pass summed the per-state counts. But a defect caused by a narrow layout appears in mobile, reflow-320 and dark-mobile — three states, one element. I was counting it three times.

The tell was in the raw output: Wikipedia showed exactly 30 incomplete results in each of those three states. Not similar numbers. Identical ones.

Had I published, I would have claimed "210 findings on Vercel". The real figure is 72. Findings are now keyed by (kind, rule, element) and deduplicated across states.

The same mistake had a second half: attributing a finding to a single state when three overlapping states could equally claim it. Attribution is now by variable group.

What this is not

Not an audit. axe detects a minority of WCAG failures — 57% by issue volume in Deque's own study, roughly a third by success criteria. Every number here is a lower bound on a subset.
Not a claim that any site is inaccessible. A finding is not a failure, and an incomplete result is explicitly a request for human review.
Not a ranking. The gap is a property of how pipelines are configured, not of how much any team cares. That is exactly why the sites that write the accessibility standards came out clean: they are testing the states.

Reproduce it

The tool is a11y-matrix — MIT, no account, no telemetry:

npx github:henriqueyuri00/a11y-matrix https://your-site.example

The study directory has the site list, both scripts, the raw per-site JSON and the control run. Every figure above comes from report.js and nowhere else, so you can disagree with the interpretation without re-running anything.

Does the legal duty change anything?

Only four sectors in the sample carry an explicit EU accessibility obligation — public sector under the Web Accessibility Directive, and banking, passenger transport and e-commerce under the European Accessibility Act, in force since 28 June 2025.

Sector	Affected
Framework and tool docs	13 / 16 (81%)
Design systems	3 / 4 (75%)
EU passenger transport	3 / 4 (75%)
EU banking	3 / 4 (75%)
EU public sector	8 / 12 (67%)
Developer platforms	4 / 7 (57%)
EU e-commerce	4 / 8 (50%)
Standards bodies	2 / 4 (50%)

The regulated sectors are not visibly better than the rest. They are also not visibly worse, and I would rather say that than reach for the more quotable claim.

To be explicit, because this is the kind of number people misquote: none of this is evidence of non-compliance by anyone. Conformance is assessed across a whole service against fifty-odd success criteria, most of which no scanner can evaluate. This looks at one page with one engine and reports differences between rendering states. What it shows is narrower: the states where these defects hide are the states an automated pipeline does not render — including at organisations that have a legal duty and watch their checks pass green.

If your accessibility pipeline is green, that is worth exactly as much as the states you rendered.

Text at 1:1 contrast is not an axe violation. It is incomplete.

Henrique Yuri — Mon, 10 Aug 2026 22:07:30 +0000

I shipped a button whose label was invisible. Same colour as its own background — 1:1 contrast, no readable text at all. It was only broken in dark mode, and my CI was green the entire time.

It was green for two reasons. I expected the first one. The second one is why I am writing this.

Reason one: the headless browser boots in light mode

axe evaluates the colour scheme that is actually rendered. It has no way to reason about a palette that was never painted. My pipeline loaded the page once, in whatever state Chromium starts in, and that state is prefers-color-scheme: light.

That part is well known, and the fix is one line:

const context = await browser.newContext({ colorScheme: "dark" });

So I did that, re-ran, and the run was still green.

Reason two: axe does not call this a violation

Here is the probe I wrote when I stopped believing the output. Same page, same axe config, two colour schemes:

=== scheme: light ===
{ matchesDark: false,
  color: 'rgb(255, 255, 255)',
  background: 'rgb(28, 93, 63)' }
violations: []
incomplete: []

=== scheme: dark ===
{ matchesDark: true,
  color: 'rgb(108, 196, 154)',
  background: 'rgb(108, 196, 154)' }
violations: []
incomplete: [ 'color-contrast x1' ]

In dark mode the text and its background are the same value. rgb(108, 196, 154) on rgb(108, 196, 154). You cannot read it, because there is nothing to read.

and violations is empty.

It lands in incomplete.

axe is not wrong

This is the part that took me a while to accept. axe declines to call it a violation because it genuinely cannot tell the difference between:

text that is invisible because someone broke the palette, and
text that is invisible on purpose — a visually-hidden label, a fade-in that has not started, a print-only string, a decorative element.

Matching foreground to background is a legitimate hiding technique. axe reports what it can prove and hands the rest to a human. That is the right call for a linter.

It is the wrong call for my pipeline, because my pipeline asserts on violations and throws incomplete away. Almost every pipeline does. Look at your own reporter and count how many of them you have ever read.

So the failure mode is asymmetric in the worst direction: the more severe the contrast defect, the more likely it is to be classified as needing review rather than failing. A 4.3:1 ratio is a confident violation. A 1:1 ratio is a question.

What I changed

Two things, and the first one matters more than the second.

Collect incomplete, and label it distinctly. Not "treat it as a failure" — that will bury you. Label it, surface it, and look at the ones that are new.

Also: do not set resultTypes: ["violations"]. It looks like a harmless optimisation. It makes axe return a single representative node for every other bucket, so the incomplete results get silently truncated before you ever see them.

Diff across states instead of counting. A raw count of findings is a chore that nobody triages. "These 3 findings exist in dark mode and not in your baseline" is a bug report with a cause attached.

That second idea turned into a small tool: a11y-matrix. It runs axe across dark mode, reduced motion, forced colors, mobile and the 320px reflow width, changing exactly one variable at a time, and reports only what each state uniquely breaks. MIT, and it runs from npx github:henriqueyuri00/a11y-matrix <url>.

One more thing it taught me

I pointed it at my own sales page. The baseline was clean. At 320px it reported six findings: a comparison table overflowing and covering its own third column.

So I fixed it the recommended way — overflow-x: auto, tabindex="0", role="region", an aria-label — and re-ran.

Still six findings. Identical message.

Because axe emits the same string for "covered by an overlay" and "scrolled out of view inside a scroll container". The first is a defect. The second is the sanctioned way to present a wide data table at the reflow width, and it is what the spec asks you to do.

My own tool was failing a page for being fixed correctly. That is how a check earns its way out of a pipeline within a week — so it now walks the DOM, asks whether the element sits inside an ancestor that actually scrolls horizontally and that a keyboard can reach, and suppresses the finding while printing the count and the reason. Never silently.

And since that removed the only thing watching reflow, it measures reflow directly instead: document scrollWidth against viewport width. axe has no rule for WCAG 1.4.10 at all. Stating the requirement beats inferring it from a contrast side effect.

If your accessibility pipeline is green, that is worth exactly as much as the states you rendered and the buckets you read.

Five other projects have the exact same fake API key in their tests

Henrique Yuri — Mon, 10 Aug 2026 21:34:37 +0000

GitHub flagged a Google API key in my repository. It was a fixture I had made up — sequential filler after the AIza prefix, never a real credential.

What I didn't expect was the sidebar on the alert page. Under Public leaks, GitHub lists every other public repository where it has seen the same string:

…/chat/tests/test_pii_detector.py
…/scanner/core/scanCode.test.ts
…/extensions/secret-guard/redact.test.js
…/scanner/src/diagnostics/redact.test.go
…/internal/cleaner/leakscanner_test.go

Five projects. Python, TypeScript, JavaScript, Go. Different authors, different years, different ecosystems.

All of them secret detectors. All of them with my "invented" key in their test suite.

It was never invented

I wrote AIzaSyA1234567890abcdefghijklmnopqrstuv thinking I had made it up. What I had actually done was reconstruct the most obvious possible fixture: take the documented prefix, fill the remaining 35 characters with the alphabet and the digits in order. Anyone writing a Google API key rule reaches for the same shape, because it is the shape the format forces.

Given a fixed prefix and a fixed length, the space of lazy values is tiny. We all landed on the same one.

There is a nice example of this in AWS's own docs: AKIAIOSFODNN7EXAMPLE. That one is deliberate — Amazon publishes it so people have a safe value to paste. The interesting case is the accidental convergence, where nobody coordinated and everybody agreed anyway.

Why this is a real problem and not a fun fact

Scanners cannot tell. A string is a credential shape or it isn't. AIzaSy plus 35 valid characters matches, and the scanner has no way to know that this particular one has been in five test suites since 2021. So it fires. Correctly, by its own rules.

Which means every one of those repositories has, or had, an open alert like mine. Five maintainers, at minimum, have triaged this exact false positive. Probably far more — the public-leaks list only shows repositories GitHub has indexed and surfaced.

And the failure mode is asymmetric. If you assume it's fake and you're right, nothing happens. If you assume it's fake and you're wrong, you've just dismissed a live credential because it "looked like a test value". The convergence trains exactly the wrong reflex.

What I changed

Not the values — they still need to be realistic, because a fixture that doesn't match the pattern proves nothing. I changed where they exist:

const ex = (...parts) => parts.join("");

// The matcher receives the identical string. The file does not contain it.
example: ex("AIza", "SyA1234567890abcdefghijklmnopqrstuv")

Then made it a build failure rather than a habit:

const CREDENTIAL_SHAPED = new RegExp([
  "ghp_[A-Za-z0-9]{36}", "AKIA[0-9A-Z]{16}", "AIza[0-9A-Za-z_-]{35}",
  "sk_live_[0-9a-zA-Z]{24}", "xox[baprs]-[0-9]{12}", "hf_[A-Za-z0-9]{34}",
  "\\d{9}:[A-Za-z0-9_-]{35}", "eyJ[A-Za-z0-9_-]{10,}\\.eyJ"
].join("|"));

A reader pointed out I had scoped that too narrowly — it covered source files but not documentation, which is backwards, since a README is exactly where a credential-shaped string gets pasted without thinking. It covers docs now.

If you maintain anything that detects secrets

Three things worth checking, and none of them takes long:

Do whole token literals appear anywhere in your source? Including the README, the docs site, and the store listing.
Would a new contributor know not to add one? If the answer is "they'd know because everyone knows", you don't have a convention, you have a coincidence waiting to end.
Can your build fail on it? Twenty lines. It is the only version of this that survives you.

And if you get an alert for a value you're sure is fake: check the public-leaks list before you close it. Mine turned four other people's test files into the evidence.

Context: I was building LeakGuard, which checks text for credentials and personal data before you paste it into an AI chat — 50 rules, 36 providers, entirely in your browser, nothing uploaded. The irony of it being blocked for containing secrets has not been lost on anyone. Source, MIT.

I built a secret detector. GitHub blocked my push for containing secrets.

Henrique Yuri — Mon, 10 Aug 2026 19:48:22 +0000

Every rule in a secret scanner needs a fixture — a value that proves the pattern fires. Mine looked like this:

{ id: "stripe-secret", label: "Stripe secret key", severity: "critical",
  re: /\b(?:sk|rk)_live_[0-9a-zA-Z]{24,}\b/g,
  example: "sk_live_0123456789abcdefghijklmn" }

Fabricated, obviously. Sequential filler after the prefix. Not a key, never was.

GitHub push protection disagreed:

remote: — Stripe API Key —————————————————————————
remote:  locations:
remote:    - commit: 02345e9…
remote:      path: test/scan.test.js:37
remote: ! [remote rejected] main -> main (push declined due to repository rule violations)

Which is, when you think about it, the system working exactly as intended. My fixtures are realistic enough to trip a serious scanner. That is the point of them.

There's an "allow this secret" link in the rejection. I didn't click it. Here's what I did instead, and the two things I got wrong on the way.

Mistake 1: assuming push protection is the whole check

I fixed the four types it flagged — Stripe live, Stripe test, Slack, Mailgun — by assembling the values from fragments so no whole token-shaped literal exists in the source:

const ex = (...parts) => parts.join("");

example: ex("sk_", "live_0123456789abcdefghijklmn")

The runtime value is byte-identical. The literal in the file is not.

Push succeeded. Ten minutes later, an email: Google API Key and Telegram Bot Token detected.

Push protection and secret scanning are different systems with different coverage. The first blocks a subset of high-confidence provider types at push time. The second scans the repository afterwards and covers more. Clearing one tells you nothing about the other, and I had quietly assumed it did.

So I stopped playing whack-a-mole and applied the fragment treatment to every realistic fixture in the corpus at once.

Mistake 2: fixing it by hand and calling it done

A convention that lives only in someone's memory is a convention that dies at the next commit. So the build now fails on any credential-shaped literal in the source:

const CREDENTIAL_SHAPED = new RegExp([
  "ghp_[A-Za-z0-9]{36}", "AKIA[0-9A-Z]{16}", "AIza[0-9A-Za-z_-]{35}",
  "sk_live_[0-9a-zA-Z]{24}", "xox[baprs]-[0-9]{12}", "hf_[A-Za-z0-9]{34}",
  "\\d{9}:[A-Za-z0-9_-]{35}", "eyJ[A-Za-z0-9_-]{10,}\\.eyJ"
].join("|"));

I verified the guard by injecting a violation. The build failed with the exact file and line. A check you haven't seen fail is a check you don't have.

The actual hard part isn't detection

While I'm here — the reason I was building this at all is that most paste guards are unusable, and it isn't because they miss things. It's because they cry wolf.

Match [A-Za-z0-9]{32} and call it a secret, and you flag UUIDs, git SHAs, minified bundles and build numbers. After the third false alarm the user stops reading the warning, which is strictly worse than having no warning.

Three things fixed most of it:

Prefixes beat entropy. A rule keyed on a documented token format is near-zero false positive. Entropy alone is a coin flip on base64. I kept entropy, but only as a secondary gate on assignment-shaped lines.

Validate where an algorithm exists. Card numbers get Luhn. Brazilian CPF and CNPJ have check digits — verify them. Roughly 99% of random 11-digit strings fail CPF's check digits, so a naive \d{11} flags every order number in your logs and a validated one flags none.

Recognise placeholders by vocabulary, not by shape. This one took two attempts. My first fix rejected anything shaped like word-word-word, which killed your-api-key-here nicely and also killed correct-horse-battery-staple — a real password. Vocabulary works: random credentials do not contain the word "your".

And a small trap inside that: \b treats underscore as a word character, so \bchange_?me\b never matches CHANGE_ME_PLEASE. Normalising separators to spaces before matching fixed it — and then broke the multi-word patterns, which were still expecting [-_]. Two bugs, one line apart.

The suite that matters

The false-positive suite in this project is larger than the detection suite, deliberately. Detection tests prove the tool does something. False-positive tests prove it is worth leaving switched on.

FALSE POSITIVES — the cases that make people uninstall
  PASS  UUID v4
  PASS  git commit SHA
  PASS  semver + build number
  PASS  invalid card (Luhn)
  PASS  random 11 digits
  PASS  invalid CPF
  PASS  placeholder secret

PRECISION — placeholders die, real passphrases survive
  PASS  'your-api-key-here'
  PASS  'CHANGE_ME_PLEASE'
  PASS  real passphrase kept

The thing I was building is LeakGuard — it checks text for credentials and personal data before you paste it into an AI chat. 50 rules across 36 providers. The web version runs entirely in your browser: no upload, no account, no network requests at all, which you can confirm from an empty network tab. Source, MIT.

A customer asked for our VPAT. Here's what that actually means

Henrique Yuri — Mon, 10 Aug 2026 13:24:08 +0000

A prospect's procurement team sends over a security-and-compliance questionnaire. Most of it you can answer. Then there's a line that says:

Please provide your VPAT or Accessibility Conformance Report.

If your reaction is "our what?", this is for you. It was mine too.

What is actually being asked

VPAT® is a template — the Voluntary Product Accessibility Template, maintained by the Information Technology Industry Council. Fill it in and the completed document is called an ACR, an Accessibility Conformance Report.

The document is a table. One row per accessibility success criterion, and for each row you declare one of four things:

Level	Means
Supports	You tested it and found no defects in scope
Partially Supports	Met in most of the product, with at least one known defect
Does Not Support	The majority of the relevant functionality fails
Not Applicable	The criterion genuinely cannot apply to this product

Plus a Remarks column, which is the part that actually matters. More on that below.

Which criteria? Depends on the edition. The three common ones:

WCAG edition — the 50 WCAG 2.1 Level A and AA success criteria.
Section 508 edition — for selling to US federal agencies.
EU edition — maps to EN 301 549, the harmonised European standard, whose Chapter 9 incorporates WCAG 2.1 Level AA in full.

For a web application sold internationally, the WCAG/EU editions cover the substance, because all three rest on the same 50 criteria.

Why you're suddenly being asked

Two things converged.

Procurement made it standard. Per the Seventh Annual State of Digital Accessibility Report, around three-quarters of organisations now require proof of accessibility for most digital purchases, and roughly a third require it for every purchase. Your buyer isn't being difficult — someone in their organisation made it a gate.

The European Accessibility Act became enforceable. Enforcement started in June 2025, and 2026 is the first full year national authorities are actively supervising. Buyers in the EU manage that risk the way buyers always do: by pushing requirements down the supply chain, into contracts with vendors like you.

You might not be in scope, and that changes nothing

Worth knowing, because most vendors selling into this space will not tell you: the EAA binds service providers with more than 10 employees or more than €2 million turnover. A four-person SaaS under €2m very likely isn't in scope as a service provider.

And it doesn't help you at all, because an exemption from the regulation is not an exemption from your customer's questionnaire. The buyer asking for your ACR isn't enforcing the EAA against you. They're managing their own obligation through a contract. That's contract law, and no exemption touches it.

If a deal is blocked, it's blocked either way.

What it costs to get one

The ITI template itself: free. It's a blank document you can download today.
A consultancy to run the audit and complete it: commonly $1,850–$3,100 for a small product — roughly $350 for completing the report on top of $1,500–$2,750 for the audit.
Doing it yourself: a few days of someone's time, and the discipline to be honest.

The expensive part was never the paperwork. It's knowing whether you actually conform.

The thing everyone gets wrong

The instinct is to make the report look good. Fifty rows of Supports, ship it, close the deal.

That instinct is exactly backwards, for a reason worth internalising: experienced reviewers read the Remarks column far more carefully than the conformance level. A clean sweep across an entire real product is rare enough that it reads as evidence the testing never happened. It invites the follow-up question you cannot answer.

Compare:

❌ Supports. "Mostly compliant, we follow best practices."

✅ Partially Supports. "Tested across sign-up, dashboard and billing with NVDA 2025.1 + Firefox. All native controls expose correct name, role and state. Two defects: the saved-view combobox does not expose aria-expanded, and the plan toggle does not expose a checked state. Tracked as ACME-4471/4472, remediation targeted 2026-10-15."

The second one takes four minutes to write and is the difference between a report that closes a deal and one that generates a call you'll lose. It says: we tested, we know where we stand, we have a plan. That is what procurement is buying.

Also worth knowing: a missing or inaccurate accessibility statement is separately punishable in some member states — France provides for fines up to €25,000 per year. An inflated report isn't just a bad look; it's exposure.

Two things to say no to

Overlays. If a vendor promises conformance from a script tag, walk. Overlays do not produce conformance with WCAG or EN 301 549, they've drawn litigation of their own, and buyers' accessibility teams increasingly treat their presence as a negative signal in exactly the procurement review you're trying to pass.

A green automated scan as your evidence. Run axe or Lighthouse — you should. But axe prints its own disclaimer on every run: "only 20% to 50% of all accessibility issues can automatically be detected. Manual testing is always required." The criteria that fail most in real audits (info and relationships, name/role/value, focus visibility, keyboard operation) are precisely the ones a scanner cannot judge. A methods section listing only automated tooling will be read as untested.

The honest minimum

If you want to produce a defensible report yourself:

Scope it in writing. List the flows in scope and the ones out, with a reason for each exclusion. Anything a customer touches is in scope.
Automated sweep. axe across representative pages — and on states, not just routes: open the modal, trigger the validation error, then scan.
Keyboard-only pass. Put the mouse away and complete every primary flow. This alone finds the most serious defects you have.
One screen reader session. NVDA + Firefox, or VoiceOver + Safari. You need about six commands, not fluency.
Zoom and reflow. 200% zoom, and 320 CSS pixels wide.
Write it down — including a dated evidence log of what was tested, on what, by whom. A conformance claim you cannot reconstruct is one you cannot defend.

Then answer the fifty rows honestly, and give every non-conforming one a real remark and a real date.

I built a free tool for step 6: ACR Builder walks the 50 WCAG 2.1 A/AA criteria and exports the report. It deliberately blocks you from filing a non-conforming criterion without an explanation, and warns you when all fifty come out as Supports. Runs entirely in your browser — no account, no upload, nothing leaves your machine. MIT licensed. Not legal advice; whether you're in scope of any of this is a question for a lawyer.

I have since written this up as a standalone reference: what a customer actually means when they ask for your VPAT — the four VPAT 2.5 editions and which applies, what each conformance level signals, and why the Remarks column is the one that gets read. There are companion notes on whether the European Accessibility Act applies to you at all and on why an accessibility statement has to be based on an actual evaluation.

Your axe run is green and your dark mode has 1.04:1 contrast

Henrique Yuri — Mon, 10 Aug 2026 03:24:46 +0000

I shipped a page that reported zero axe violations. It had button text at a contrast ratio of 1.04:1 — which is, for practical purposes, invisible text.

The scan wasn't broken. It was answering a narrower question than I thought I was asking.

The bug

I had a theme system built the ordinary way. Tokens on :root, overridden in a prefers-color-scheme media query, and overridden again by an explicit [data-theme] attribute so a manual toggle wins in both directions.

Buttons came in two flavours: a solid primary and a bordered secondary.

.btn      { background: var(--accent); color: var(--panel); }
.btn.sec  { background: transparent;   color: var(--ink);   }

In dark mode the accent goes light green, so white-on-accent stops working. I patched it the way you patch things at 1am:

:root[data-theme=dark] .btn { color: #10241b }
@media (prefers-color-scheme: dark) {
  :root:not([data-theme=light]) .btn { color: #10241b }
}

Now count the specificity.

Selector	Specificity
`.btn.sec`	0,2,0
`:root[data-theme=dark] .btn`	0,3,0
`:root:not([data-theme=light]) .btn`	0,3,0

:not() doesn't add specificity of its own, but its argument does. So :root (0,1,0) + [data-theme=light] (0,1,0) + .btn (0,1,0) lands at 0,3,0.

My theme patch outranks the component modifier. In dark mode, every secondary button — transparent background, sitting on a #1a1c1f panel — got painted #10241b. Dark green on near-black. 1.04:1.

The nasty part is that this class of bug is invisible in review. The rule looks correct. It is correct, for the buttons it was written for. It just also matched buttons it was never meant to touch, in one theme only.

Why the scan didn't catch it

axe-core evaluates the DOM as currently rendered. It reads computed styles, and computed styles resolve exactly one colour scheme: whichever one the browser is in right now.

So npx axe https://example.com is not "does this page pass contrast." It's "does this page pass contrast in the scheme this headless browser happened to boot in." If your CI runs light and your bug is dark, you get a green check that means nothing.

I didn't want to take my own word for this, so I re-introduced the bug and ran axe against the same page twice, changing only the theme:

const runIn = async theme => {
  document.documentElement.setAttribute('data-theme', theme);
  await new Promise(r => requestAnimationFrame(() => requestAnimationFrame(r)));
  const { violations } = await axe.run(document, {
    runOnly: { type: 'tag', values: ['wcag2a','wcag2aa','wcag21a','wcag21aa'] }
  });
  return violations;
};

Same page. Same axe 4.12.1. Same second.

light → 0 violations
dark  → 2 violations
        color-contrast 1.13 (#10241b on #131416)
        color-contrast 1.04 (#10241b on #1a1c1f)

One run says ship it. The other says two elements are unreadable.

The fix, and the better fix

The immediate fix is to stop overriding the colour and start swapping the token. The override created a specificity contest; a token has none to win.

:root                             { --accent:#1c5d3f; --on-accent:#fff;    }
@media (prefers-color-scheme:dark){ :root:not([data-theme=light]) {
                                    --accent:#6cc49a; --on-accent:#10241b; } }
:root[data-theme=dark]            { --accent:#6cc49a; --on-accent:#10241b; }

.btn     { background: var(--accent); color: var(--on-accent); }
.btn.sec { background: transparent;   color: var(--ink);       }

.btn.sec now wins cleanly, in every theme, because nothing is competing with it. The general rule: let themes change values, not selectors. The moment a theme block starts naming components, you've entered a specificity war you will eventually lose in exactly one of your themes.

Testing both schemes in CI

page.emulateMedia() is the piece most setups are missing:

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

const ROUTES  = ['/', '/pricing', '/app/dashboard'];
const SCHEMES = ['light', 'dark'];

for (const colorScheme of SCHEMES) {
  for (const route of ROUTES) {
    test(`a11y ${colorScheme} ${route}`, async ({ page }) => {
      await page.emulateMedia({ colorScheme });
      await page.goto(route);
      const { violations } = await new AxeBuilder({ page })
        .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
        .analyze();
      expect(violations).toEqual([]);
    });
  }
}

Two caveats worth knowing, because they bit me:

emulateMedia only drives prefers-color-scheme. If you also ship a manual toggle that writes data-theme or a class, that path is a different code path and needs its own case. Set the attribute explicitly before analysing.

Analyse states, not just routes. Contrast bugs love disabled buttons, validation errors, toasts, empty states and open modals — none of which exist in the DOM when the page first loads. Open the thing, then scan.

The wider point

Automated accessibility testing is worth doing and it is worth being clear-eyed about. The axe CLI prints its own disclaimer every run:

"only 20% to 50% of all accessibility issues can automatically be detected. Manual testing is always required."

I'd add a corollary from this bug: within that 20–50%, a scan only covers the rendered state you gave it. Every scheme, every breakpoint, every interaction state you don't render is a state you didn't test. A green check is evidence about one configuration, not a certificate.

I found this while building ACR Builder — a free, offline, client-side tool that walks the 50 WCAG 2.1 A/AA success criteria and exports a conformance report, for teams whose enterprise customers have started asking for accessibility documentation. Auditing it against itself turned up five real defects, this being the most embarrassing. MIT licensed, no signup, nothing leaves your browser. Source.

DEV Community: Henrique Yuri

How to test dark mode accessibility in CI with Playwright and axe-core

The one-line part

Trap 1: the worst contrast defects are not in violations

Trap 2: a strict CSP blocks the scan entirely

Trap 3: raw counts are useless, deltas are not

Trap 4: real pages are not deterministic

What this actually finds

If you would rather not wire it up

I scanned 70 well-known sites in seven browser states. The default one missed something on 68% of them.

The result

The control is the part that makes this a study

What exposed the findings

I got the counting wrong first, and the error was a factor of three

What this is not

Reproduce it

Does the legal duty change anything?

Text at 1:1 contrast is not an axe violation. It is incomplete.

Reason one: the headless browser boots in light mode

Reason two: axe does not call this a violation

axe is not wrong

What I changed

One more thing it taught me

Five other projects have the exact same fake API key in their tests

It was never invented

Why this is a real problem and not a fun fact

What I changed

If you maintain anything that detects secrets

I built a secret detector. GitHub blocked my push for containing secrets.

Mistake 1: assuming push protection is the whole check

Mistake 2: fixing it by hand and calling it done

The actual hard part isn't detection

The suite that matters

A customer asked for our VPAT. Here's what that actually means

What is actually being asked

Why you're suddenly being asked

You might not be in scope, and that changes nothing

What it costs to get one

The thing everyone gets wrong

Two things to say no to

The honest minimum

Your axe run is green and your dark mode has 1.04:1 contrast

The bug

Why the scan didn't catch it

The fix, and the better fix

Testing both schemes in CI

The wider point

Trap 1: the worst contrast defects are not in `violations`