Every automated accessibility check I have ever seen loads the page once, in whatever state the headless browser boots in: light colour scheme, no motion preference, forced colors off, desktop viewport.
That is one user. I wanted to know what it costs, so I measured it across 70 public homepages — W3C, MDN, React, Vue, Kubernetes, GitHub, Wikipedia, the BBC, a handful of design systems, and four sectors carrying an explicit EU legal accessibility duty: government portals, banking, passenger transport and e-commerce.
(This started as 36 mostly-technical sites. I extended it afterwards, and the numbers below are the larger run — the update is described at the end.)
Each site was loaded seven times. One run is the baseline: light, desktop, no preferences — what an ordinary pipeline tests. The other six each change exactly one variable away from it. A finding counts only if it exists in a non-baseline state and not in the baseline.
The result
70 attempted. 68 loaded — Allegro and Air France did not, and are excluded rather than counted as clean. Of those 68, 62 produced byte-identical output across two identical runs, and only those 62 are counted.
| Had a finding the baseline never surfaced | 42 / 62 — 68% |
| Had one axe classes as a violation, not "needs review" | 26 / 62 — 42% |
| Distinct findings missed by the baseline | 348 |
| Median per site | 1 |
The first pass, on 36 mostly-technical sites, returned 71% and 47%. Nearly doubling the sample and adding regulated sectors moved it to 68% and 42%. The finding replicated, which is worth more than either number by itself.
The median is the honest headline, not the 348. The distribution is badly skewed: half the sites have one or none, and a handful carry most of the volume — Vercel (72), Tailwind CSS (52), Radix UI (40), Wikipedia (31). Quoting the total alone would imply everyone is sitting on dozens. They are not.
Eleven sites were clean in all seven states: W3C, WebAIM, MDN, Playwright, Primer, GOV.UK, Service-Public (FR), Rijksoverheid (NL), Suomi.fi (FI), Fnac, SNCF Connect. Several either write the standard, teach it, or are bound by a public-sector duty and appear to be meeting it.
The control is the part that makes this a study
Real sites carry carousels, rotating promos, A/B tests and lazy-loaded media. Two identical loads do not necessarily produce identical output. Without measuring that, "unique to dark mode" might just mean a different hero image rendered.
So I ran the same state twice, changing nothing, on all 36 sites:
146 findings appeared and 156 disappeared — concentrated on six sites. The other 62 were byte-identical.
Those six are excluded from every figure above rather than quietly left in the denominator, and this is the part that earned the control its place. One shopping site produced 715 apparent findings and a churn of +122/−125 between two identical loads. That is a rotating product carousel, not a defect.
Excluding it moved its whole sector's total from 729 findings to 14. Had I skipped the control, that one number would have been the loudest thing in the study, and it would have been wrong.
What exposed the findings
| Variable | Sites | Distinct findings |
|---|---|---|
| Narrow viewport (mobile / 320px reflow) | 42 | 340 |
| Dark colour scheme | 38 | 303 |
| Dark scheme alone — no narrow state found it | 9 | 28 |
| Reduced motion | 3 | 8 |
| Forced colors | 0 | 0 |
The first two overlap heavily — a lot of sites break in dark mode and at 320px — which is why "dark scheme alone" is broken out. On those 9 sites the colour scheme is the only variable that can explain the finding.
By rule: color-contrast on 32 sites, then 9 sites whose document is genuinely wider than a 320px viewport — the WCAG 1.4.10 condition measured directly, since axe has no rule for it — then scrollable-region-focusable, link-in-text-block and button-name.
Forced colors found nothing at all on any stable site. Reporting that is the point: a negative result is a result.
I got the counting wrong first, and the error was a factor of three
Worth writing down, because the wrong way is the obvious way.
My first pass summed the per-state counts. But a defect caused by a narrow layout appears in mobile, reflow-320 and dark-mobile — three states, one element. I was counting it three times.
The tell was in the raw output: Wikipedia showed exactly 30 incomplete results in each of those three states. Not similar numbers. Identical ones.
Had I published, I would have claimed "210 findings on Vercel". The real figure is 72. Findings are now keyed by (kind, rule, element) and deduplicated across states.
The same mistake had a second half: attributing a finding to a single state when three overlapping states could equally claim it. Attribution is now by variable group.
What this is not
- Not an audit. axe detects a minority of WCAG failures — 57% by issue volume in Deque's own study, roughly a third by success criteria. Every number here is a lower bound on a subset.
-
Not a claim that any site is inaccessible. A finding is not a failure, and an
incompleteresult is explicitly a request for human review. - Not a ranking. The gap is a property of how pipelines are configured, not of how much any team cares. That is exactly why the sites that write the accessibility standards came out clean: they are testing the states.
Reproduce it
The tool is a11y-matrix — MIT, no account, no telemetry:
npx github:henriqueyuri00/a11y-matrix https://your-site.example
The study directory has the site list, both scripts, the raw per-site JSON and the control run. Every figure above comes from report.js and nowhere else, so you can disagree with the interpretation without re-running anything.
Does the legal duty change anything?
Only four sectors in the sample carry an explicit EU accessibility obligation — public sector under the Web Accessibility Directive, and banking, passenger transport and e-commerce under the European Accessibility Act, in force since 28 June 2025.
| Sector | Affected |
|---|---|
| Framework and tool docs | 13 / 16 (81%) |
| Design systems | 3 / 4 (75%) |
| EU passenger transport | 3 / 4 (75%) |
| EU banking | 3 / 4 (75%) |
| EU public sector | 8 / 12 (67%) |
| Developer platforms | 4 / 7 (57%) |
| EU e-commerce | 4 / 8 (50%) |
| Standards bodies | 2 / 4 (50%) |
The regulated sectors are not visibly better than the rest. They are also not visibly worse, and I would rather say that than reach for the more quotable claim.
To be explicit, because this is the kind of number people misquote: none of this is evidence of non-compliance by anyone. Conformance is assessed across a whole service against fifty-odd success criteria, most of which no scanner can evaluate. This looks at one page with one engine and reports differences between rendering states. What it shows is narrower: the states where these defects hide are the states an automated pipeline does not render — including at organisations that have a legal duty and watch their checks pass green.
If your accessibility pipeline is green, that is worth exactly as much as the states you rendered.
Top comments (0)