I built a Chrome extension that checks pages against the automatable parts of WCAG 2.2 AA. This post isn't really about the extension. It's about what happened when I stopped trusting it.
The control group
Once the checker worked, the first thing I did was point it at gov.uk. The UK government's site has a long, public record of taking accessibility seriously. If my tool reported problems there, the sensible assumption was that the tool was wrong, not the site.
It reported 72 violations. Every one of them was a bug in my code.
So I made that the method. I picked sites built by people who do accessibility for a living — gov.uk, WebAIM, Deque, The A11Y Project, the W3C's WAI pages — and treated anything reported on them as my bug until proven otherwise.
Six real defects came out of it.
1. I skipped the WCAG 2.5.8 Spacing exception
2.5.8 Target Size (Minimum) is new in WCAG 2.2: pointer targets should be at least 24×24 CSS pixels. The naive implementation is one line — read getBoundingClientRect(), compare against 24.
But the criterion has five exceptions, and one of them does most of the work:
Spacing: an undersized target passes if a 24px-diameter circle centred on it doesn't intersect another target, or another undersized target's circle.
The real question 2.5.8 asks isn't "is it big enough?" — it's "is there room to miss without hitting something else?" A 20px icon button with generous margin is fine. Three 16px buttons packed 2px apart are not.
Without the spacing check: 72 violations on gov.uk. With it: zero.
2. I measured contrast on text nobody can see
This is the standard way to give an icon button a screen reader label:
.icon-button {
text-indent: -5000px;
overflow: hidden;
}
The text is in the DOM but never painted. I was faithfully measuring its colour contrast — which is meaningless.
The fix: measure the text node with a Range, not the element's box, and check that its rect survives every ancestor with overflow clipping. If it's clipped away, there's nothing to measure.
3. My accessible-name computation ignored aria-label on descendants
<a href="/nvidia">
<svg role="img" aria-label="Nvidia">...</svg>
</a>
That link is correctly named. My code collected text nodes only, found none, and reported it as a link with no discernible text.
In the accname spec, "name from content" recursively takes each descendant's accessible name, not its raw text. That one bug produced 30 false findings on stripe.com alone. Logo links and icon links are everywhere — shipped as-is, it would have cried wolf on most of the web.
4. I called a zero-height element "too small"
On a11yproject.com:
Target is 320×0px, below the 24×24 minimum
A link inside a collapsed panel. Something with no height doesn't have a small hit area — it has none. The message was describing a box, not a target.
Now the hit area is the union of the element's box and its rendered descendants (clicks on a child are routed to the enclosing link), and if that union is still degenerate, the element is skipped.
5. I told Deque — who make axe — that their site had six violations
The rule: "focusable element inside aria-hidden". A keyboard user can reach it, a screen reader user can't perceive it — a real and common bug.
Except every one of the six was inside a display: none subtree. A closed dropdown. Those elements aren't in the tab order at all; the browser skips them just as the screen reader does. No mismatch.
The distinction matters because the real version of this bug is extremely common: an off-canvas menu shoved off-screen with a transform, still display: block, still tabbable, marked aria-hidden. Tab into it and focus vanishes. That one the tool should — and now does — still catch.
6. I read 2.4.4 as if it were AAA
I was failing links like "Read more" and "Learn more".
But 2.4.4 is Link Purpose (In Context). A "Read more" whose purpose is clear from the card it sits in conforms. Requiring the link text to stand on its own is 2.4.9, which is Level AAA.
This wasn't a code bug; it was a misreading of the spec. I suspect it's a common one. It's now reported as "needs review", not as a failure.
Where it stands
| Site | Violations reported |
|---|---|
| gov.uk | 0 |
| webaim.org | 0 |
| deque.com | 0 |
| a11yproject.com | 0 |
| w3.org/WAI | 0 |
Sites with genuine problems still light up — which is the other half of the test.
Every fix has a regression test. The one that actually keeps the tool honest is a single file, test/clean.html: a correctly built page that must produce zero violations. Anything flagged there is a false positive by definition.
What I took from it
When an automated accessibility report contradicts a site with real expertise behind it, check the tool first.
And be honest about what automation can do at all. Automated checks find roughly a third of accessibility barriers. Whether alt text is accurate, whether a keyboard user can finish a checkout, whether a page makes sense read aloud — no tool answers those. So this one reports what it can't decide as "needs review" instead of guessing, and says so in its UI.
If you want to try it
It's free, has no account, and makes no network calls at all — the privacy policy depends on that, so there's a test that fails the build if anyone adds one. The source is public.
- Chrome Web Store: https://chromewebstore.google.com/detail/ldbmeaihgdlghenedbhafkfikefnmicb
- Source: https://github.com/perceivable/a11yscope
- The 2.5.8 exceptions, in detail: https://perceivable.github.io/guides/wcag-target-size/
If you find it flagging correct markup, please tell me. A scanner that cries wolf is worse than no scanner, and every report so far has become a regression test.
Top comments (4)
The closed-dropdown versus off-canvas distinction is a useful paired regression. I'd make the clean fixture prove sensitivity too: run it once unchanged, then change just the hidden menu from
display: noneto a translated, still-tabbable subtree while keepingaria-hidden. The first should stay clean and the second should produce the specific focus finding. That guards against fixing false positives by skipping too much. Do the regression cases assert the finding type and affected element, or only the total violation count?@launchgatecheck Took the suggestion and added it today (commit
ae31218). The clean fixture is scanned once unchanged, then only the menu changes:display:block; position:fixed; transform:translateX(-200%),aria-hiddenleft in place. The suite then requiresaria-hidden-focusableto report exactly 2 violations (the two links) and every other rule to stay quiet.It earned its keep on the first run. With plain
display:blockthe menu took up layout height and pushed anopacity:0hover panel below the fold, where a scroll-reveal heuristic treats it as "will be visible once scrolled" and measures its contrast. So the mutation has to be out of flow, like a real off-canvas drawer — a fair reminder that the fixture's silence depends on geometry, not just on the rules.On your question: per rule, not just totals. 24 rules must each produce at least one violation on the broken page, 3 must produce a review item and not a violation, 2 must stay silent. A few assert on the finding itself — contrast has to measure the
#999paragraph at ~2.85:1 and must not flag#111on white; target-size must never report a zero-area box. What they don't do yet is assert the affected element by selector for every rule, only for the ones that have bitten me. That's a gap, and you're right that it's the one that would let "fixed by skipping too much" slip through.Pointing it at gov.uk first and treating every finding as your own bug is what makes the zero-violation table at the end mean something.
I test an e-invoice validator the same way. The authorities publish sample invoices that are valid by definition (78 in the German XRechnung testsuite I pin), and my suite asserts that every one of them comes back clean.
For the other half, the sites with genuine problems that still have to light up: where do yours come from? On my side the authorities also ship documents that must fail, with the expected rule named. Is there something like that for WCAG, or do you build the broken pages yourself?
@to21as There isn't an authority-published "must fail, expected rule X" corpus for WCAG the way XRechnung ships one, but the closest thing is W3C's ACT Rules (act-rules.github.io). Each rule comes with test cases marked passed / failed / inapplicable, and engines like axe-core publish how they score against them. That's the analogue to your 78 valid invoices plus the documents that must fail. I haven't wired those in yet — it's on the list, and your framing makes it feel overdue.
What I have now is hand-built: a broken fixture where 24 rules each have to fire, 3 have to produce a review item rather than a violation, and 2 have to stay quiet — asserted per rule, not as a total. Plus real sites used as calibration by hand, not automated: MDN for a missing main landmark, naver.com for grey-on-white text, both checked to be genuine before trusting the finding. And gov.uk and a11yproject.com as the control group, where anything reported is my bug until proven otherwise.
Pinning the test suite version, as you do with XRechnung, is a habit I should copy: ACT test cases change as the rules get refined.