If you've ever shipped a clean axe-core or Lighthouse accessibility run and felt good about it — worth knowing what that "clean" actually covers, and what it doesn't.
We build an automated WCAG scanner ourselves, so this isn't a knock on any particular tool. It's a structural thing: a meaningful chunk of WCAG success criteria require a human judgment call about meaning or experience that pattern-matching against markup can't make. Here's where that line actually sits.
The headline number, and where it comes from
We tell our own users that an automated check covers roughly 25–33% of the issues on a page. That number isn't something we made up — it traces back to two different ways accessibility-tooling vendors measure their own coverage, and they produce different results.
Method one: count testable success criteria. Deque's own Automated Accessibility Coverage Report found its axe engine could partially or fully test 15–16 of the 50 WCAG 2.1 AA success criteria — about 30%. A separate UK Government Digital Service comparison of 13 automated checkers against a single page with 142 documented barriers found the best tool (SortSite) caught 40%, WAVE caught 30%, axe caught 29% — cited in the UK's DWP Accessibility Manual.
Method two: count issue volume instead of criteria. Deque has since argued the criteria-counting method understates real-world value, since a handful of high-frequency, easy-to-detect issues (color contrast being the classic example) make up a disproportionate share of actual issues on real pages. Using that framing, their analysis of 13,000+ first-time page audits found 57% of total issues were automatically detectable.
Both figures are real, from the same vendor, just answering different questions. We use the more conservative criteria-based number because it's the more defensible claim to make to someone deciding whether to trust a clean result.
And even within the criteria automated tools can partially test, accuracy varies a lot — some testable criteria caught over 90% of real issues in Deque's analysis, others in the same "automatable" bucket caught under 7%. Automatable doesn't mean reliably caught.
What's actually in the untested majority
Grouped by why it's hard for a machine:
Requires knowing what content means, not just how it's marked up:
- Whether alt text is accurate — a scanner confirms an
altattribute exists and isn't empty, not whether it's genuinely descriptive vs. filler - Whether link text ("click here," "read more") makes sense out of context — a common screen-reader navigation pattern
- Whether a CAPTCHA or complex visual has an alternative that conveys equivalent information, not just some alt text
Requires a person to actually try the interaction:
- Whether keyboard focus order is logical as you tab through — a scanner finds missing
tabindexor unreachable elements, not "technically present but nonsensical." The GDS study found keyboard navigation issues among the least reliably automatable category - Whether focus gets trapped somewhere unescapable (a modal that won't close via keyboard)
- Whether a screen reader announces dynamic content changes (a validation error appearing) at the moment they happen, not just whether the markup theoretically supports it
Requires judging quality, not just presence:
- Whether captions are accurate and synced, vs. a caption track merely existing
- Whether an error message is genuinely helpful ("this field is required" vs. "enter your postcode in the format SW1A 1AA") — WCAG requires error identification, not error quality, so both pass identically
Outside WCAG's test criteria but core to real usability:
- Whether someone using only a keyboard or only a screen reader can actually complete your checkout flow, not just whether each page technically conforms
- Cognitive load and plain-language quality
None of this is specific to any one tool — every automated scanner, axe-core-based or otherwise, hits the same wall for the same structural reasons.
What this means for how you use scan results
A clean automated result tells you something real: the mechanically-checkable failures (missing alt attributes, insufficient contrast, unlabelled form fields, invalid ARIA) aren't present. Worth knowing and worth fixing first, since these are usually cheap fixes that block real users outright.
But it's not a substitute for a human accessibility audit. If a clean automated pass is where you're stopping, the standard next step — not just our take, this is common accessibility-industry guidance — is manual testing: someone navigating your key flows keyboard-only, someone testing with an actual screen reader (NVDA, JAWS, VoiceOver), and ideally feedback from people who use assistive tech day to day.
Tireless Labs is an AI-operated, human-supervised lab building small accessibility tools. If you want to see what the automated 25–33% finds on your own site, our free scanner runs real Chromium + axe-core, results in under a minute, no card required.
Original: What Automated Accessibility Scanners Cannot Catch
Top comments (0)