Median change to the underlying markup: zero — in every one of six runs, across three samples drawn at different times.
That sentence needs about two thousand words of context before it means anything, including the three occasions where I nearly published a number that was wrong. Here they are.
Start with what it does not measure: whether a widget helps anyone. It measures whether a page's markup changes when the widget runs. A tool with a genuinely useful toolbar — bigger text, a reading mode, a contrast switch — scores zero here and may still be worth having. The full limitations are below, and they are load-bearing.
The comparison that cannot work
Accessibility overlay widgets attach a script to your site and are sold as a way to improve its accessibility. The obvious way to test that is to measure sites that have one against sites that don't.
I did that first. It produced a clean, statistically significant result pointing in an interesting direction, and it is worthless.
In a pool of 233 German Shopify storefronts, the 17 stores with an overlay installed had a median of 25 violation nodes; the 216 without had 14. The overlay group carried 78.6% more, and the difference clears the usual bar (Mann–Whitney, z = 2.48, p = 0.013). Read that as a fact about which stores buy a widget, not about what a widget does. Group A is 17 stores and a median over 17 is fragile; the international contour, with 65 stores against 71, shows no statistically detectable difference (5.7%, z = −0.90, p = 0.369). Nor can I say why the German groups differ. Overlay stores do carry heavier pages, a median of 3,068 DOM nodes against 2,714, but that is 13% more markup against 78.6% more violations, so page weight accounts for a small part of the gap at best. The conclusion of this article rests on the paired test in the next section, not on either of these numbers.
The problem is structural. Nobody installs an accessibility widget at random. You go looking for one because you have a problem, or because a lawyer's letter arrived, or because your category is one where this gets noticed. The stores that bought a widget are not a control group for the stores that didn't — they are the stores that had a reason. That comparison can never separate the effect of the tool from the reason someone reached for it, no matter how many sites you throw at it or how small the p-value gets.
I could have published the 78.6%. It is a real number, it is significant, and it flatters a thesis people enjoy. It is also uninterpretable, and dressing it up with a p-value would only have made it more convincing than it deserves.
The comparison that does work
Measure the same page twice, a minute apart, and change exactly one thing: whether the browser is allowed to load the overlay's script.
ON → load the page as served, wait, run axe-core
OFF → load the page again, abort every request to the overlay's CDN, wait, run axe-core
Same URL. Same viewport, 1440×900. Same locale and Accept-Language. Same settle delay. Same axe-core version and the same rule set. Same network position — a German exit node throughout — seconds apart, so the same CDN edge and the same storefront state. Fresh browser context each time, so nothing carries over.
The difference between those two numbers is what the widget does to the page. There is no selection left to worry about, because the store is its own control.
I ran this on 56 distinct stores — a German sample of 16, an international sample of 20, and a fresh sample of 20 that had never been measured, drawn deliberately after the first results existed so that the finding had a chance to fail against data it wasn't derived from. Two of the three samples were measured twice, and one vendor had to be measured again after a bug described below, which is how 56 stores produce 92 store-runs and 179 page pairs.
What came out
Nodes removed by the widget, per run. Read "removed" as OFF minus ON: positive means the widget took violations away.
| run | pairs | stores | nodes removed | of | share | median per store |
|---|---|---|---|---|---|---|
| German | 14 | 7 | 12 | 468 | 2.6% | 0.0% |
| English, run 1 | 35 | 18 | 106 | 1,920 | 5.5% | 0.0% |
| English, run 2 | 35 | 18 | 99 | 1,921 | 5.2% | 0.0% |
| Fresh, run 1 | 37 | 19 | 41 | 1,343 | 3.1% | 0.0% |
| Fresh, run 2 | 35 | 18 | 17 | 1,215 | 1.4% | 0.0% |
| Accessibly, re-measured | 23 | 12 | −3 | 905 | −0.3% | 0.0% |
| all six | 179 | 56 | 272 | 7,772 | 3.5% | 0.0% |
The widgets in the sample were UserWay (23 stores), accessiBe (15), Accessibly (12), EqualWeb (4) and AudioEye (2). Results are not broken down by product: the split is uneven and no per-vendor number here would survive its own sample size. Nothing in these figures should be attached to any one named product. The Accessibly row exists only because that vendor had to be re-measured, not to single it out.
Two summary figures, and they disagree on purpose.
3.5% is node-weighted: every violation node counts equally, so one large storefront with hundreds of findings can carry a whole run. 0.0% is the median across stores: every store counts once, regardless of size. The median came out at zero in all six runs.
The rest of the shape:
- 56 of 92 store-run observations showed no change whatsoever — 61%.
- Per-run share ranged from −0.3% to 5.5%. That spread, on samples this size, is the honest measure of how much a single run can wander.
- In every run some stores measured worse with the widget enabled than with it blocked — 1, 3, 2, 2, 4 and 2 stores respectively. That is mostly run-to-run noise rather than a widget effect. Across the two English runs, 3 different stores came out worse and only 2 did so both times; across the two fresh runs, 4 different stores and again only 2 both times.
In one sentence: on most of these storefronts, blocking the widget's script changed nothing an automated checker could see, and where something did change it was small and did not reproduce reliably between runs on the same stores.
The three times I nearly published something false
All three were caught by re-running measurements I had already written down as results. All three push the number in the flattering direction. If you repeat this work, you will hit them.
Lazy loading
An early run showed one German storefront dropping from 46 violation nodes to 21 — a 54% improvement, exactly the kind of number that ends up on a slide.
It was not the widget. The two measurements used different settle delays, 5 seconds and 20 seconds. Almost the entire difference is a single rule: image-alt fell from 31 to 7, while the page kept growing between the two measurements. The storefront's own lazy-loading was still working at 5 seconds, and the placeholder images it had not yet swapped out were being counted as images with no alt text. What I had measured was how long I was willing to wait.
Re-run as an actual pair — widget allowed and widget blocked, both at the same delay: 46 and 46.
This is why the settle delay is not a detail you tune for convenience. Any asymmetry in how long you wait becomes a measurement of page weight wearing an accessibility result's clothes.
Pages that never rendered
The fresh sample nearly ended the project. Run 1 gave −17 nodes; run 2, on the identical sample under identical settings, gave +116. A result that flips sign between consecutive runs is not a result.
Per store, though — before any exclusion — 16 of 20 were stable within three nodes, and three stores produced 92% of the variance. One of them returned a single violation node on the widget-enabled side against 56 on the blocked side: the page had not rendered at all, and it had no <title>. That one blank page contributed +98 of that run's +116.
So I wrote down a validity rule — before checking which stores it would remove:
A pair is discarded if one side has ≤ 3 violation nodes while the other has ≥ 15, or if the two sides disagree about whether the page had a
<title>.
It removes 4 pairs across the whole study. With it applied, the fresh sample gave 3.1% and 1.4% — inside the range the original samples had already produced. The finding survived contact with data it had not been built on.
I mention the order — rule first, then look — because doing it the other way round is the whole game. A threshold chosen after you have seen which points it excludes is not a threshold, it is an opinion with arithmetic attached.
A vendor that was never actually switched off
This is the one that should have killed the study, and the only reason it did not is that I checked a field I had been collecting and never using.
The OFF side works by aborting every request whose URL contains one of a list of overlay hosts. My list contained accessiblyapp.com. Accessibly serves its script from cdn.accessibly.app. Those two strings do not match, so for every pair of every run, that vendor's widget loaded normally on both sides. Thirty pairs that I was counting as ablations were the same page measured twice.
Nothing looked wrong. Those pairs produced small differences clustered around zero — which is exactly what the study concluded — so the bug was holding the headline up rather than knocking it down. A result that agrees with you is the hardest kind to audit.
What exposed it: ablation.mjs records widgetInDom on the ON side, whether the widget's own button or panel is present after the delay. Comparing that field between ON and OFF, Accessibly never disappeared — 30 pairs out of 30. Then a second surprise: after fixing the host list, it still never disappeared, because accessibly-trigger and accessibly-config are rendered into the page by the Shopify app itself and sit there whether the script loads or not. widgetInDom is a vendor-dependent proxy and cannot be used as a validity filter — a conclusion I reached only after briefly believing the opposite. What settles it is the request log: with the corrected host list the script request is aborted, and that is the thing being manipulated.
The 30 bad pairs are excluded and those 12 stores were measured again with the list fixed. Properly switched off, Accessibly removes −3 nodes across 23 pairs — three nodes worse with the widget running. The median per store is 0.0%, like everything else.
The full accounting, since a denominator you cannot reconstruct is not worth much:
| pairs | |
|---|---|
| attempted | 192 |
| lost — one side hit the 40 s navigation timeout | 3 |
| discarded by the validity rule | 3 |
| excluded — the block never engaged for this vendor | 30 |
| re-measured with the host list corrected (24 attempted, 1 discarded) | +23 |
| analysed | 179 |
The lesson is not "check your regexes". It is that a manipulation has to be verified, not assumed. I had written in my own method notes that a pair where the widget never ran is not testing anything, and then never enforced it.
Half the problem is one rule
Across the wider survey — 729 pages, 367 distinct stores, 18,383 violation nodes — the findings sort like this. (233 of those stores sit in the German contour and 136 in the international one; that sums to 369 rather than 367 because two stores appear in both.) All of these counts are lower bounds: axe-core decides only the part of WCAG a machine can decide, and one node is one element failing one rule, not a measure of how much it hurts.
| bucket | nodes | share |
|---|---|---|
| deterministic fix, no meaning to invent | 3,037 | 16.5% |
| detection deterministic, fix is generated text (alt, link name) | 5,824 | 31.7% |
| fixable but the visual design changes (contrast, target size) | 9,219 | 50.1% |
| needs a human | 303 | 1.6% |
One rule, color-contrast, is 50.1% of everything found. Every confident claim of the form "N% of accessibility problems can be fixed automatically" is, underneath, a claim about how you treat contrast — and contrast cannot be fixed without changing what the site looks like, which is a decision belonging to whoever owns the brand.
I ran a separate test on that: if a handful of colour pairs generate most of a store's contrast findings, contrast is fixable centrally in a theme rather than node by node. Across 990 contrast nodes in 30 stores, the median store needs 2 colour pairs to cover 80% of its contrast findings — promising. But the share of stores clearing the coverage bar I had set in advance came out at 39.6% against a 40% threshold.
That is four nodes out of 990. I left the threshold where it was and recorded the result as undecided. A test that lands this close hasn't told you the answer is no; it has told you your sample is too small to ask.
One more thing about those contrast numbers: 132 of the 990 nodes (13.3%), across five stores, have a computed contrast ratio below 1.1 — text almost exactly the colour of its background. That is the signature of an element that is invisible for some unrelated reason, not of unreadable text. I did not inspect them visually, and they are included in the totals above.
For completeness: how common are these widgets
Not very. In a straight scan of German Shopify storefronts, 16 of 886 had one — 1.8%. In a global sample, 370 of 5,326 — 6.9%.
Which raises a fair objection to the numbers further up: 17 of the 233 measured German stores had an overlay, and that is 7.3%, not 1.8%. Both are correct, and the gap is deliberate. At 1.8% a random draw of 233 stores would contain about four overlay stores, which is not enough to compare anything. So overlay stores were deliberately over-sampled into the measured set — roughly four times their natural rate.
That is fine for the ablation, where each store is its own control and the sample's composition is irrelevant. It is one more reason the store-against-store comparison in the first section should not be read as representative of anything.
Limitations
Read this section before quoting any number above. Several of these are large enough to change what you should conclude.
This measures DOM repair and nothing else. The ablation answers exactly one question: does the source markup contain fewer machine-detectable violations when the widget runs? It does not answer whether a widget helps anyone. An overlay can provide a genuinely useful toolbar — bigger text, a reading mode, a contrast switch — and leave the underlying markup untouched. Such a product scores 0.0% here and may still be valuable to a real person. Nothing in this article should be read as a claim that any of these tools does not work.
Automated testing catches a minority of WCAG. axe-core checks the success criteria a machine can decide. Whether a label actually describes its field, whether reading order makes sense, whether a custom widget is operable by keyboard — none of that is in these numbers. A page with zero violations can be unusable; a page with many can be fine.
A node is not a severity. A missing alt on a decorative icon and a missing name on the checkout button both count as one.
Nobody tried to buy anything with a screen reader. These are scanner counts, not user research. That is a different study and a more important one.
Checkout was never measured. Two pages per store: homepage and one product page. Shopify hosts checkout, and testing someone's checkout without permission is not something I was willing to do — so the single most important page in a store is absent from every number here.
The German contour is now very small. Most of the German ablation sample ran Accessibly, so removing the unblocked pairs cut it from 16 stores to 7. Treat that row as an indication, not a country-level result.
One network position, one moment. All measurements ran from a German exit node between 5 and 8 August 2026, in headless Chromium with no profile or prior consent state. Geo-routing, CDN edges and consent banners all change what a scanner sees. This matters for the survey figures. It matters much less for the ablation, because both sides of every pair were measured from the same position seconds apart — that is the point of pairing.
Blocking a CDN is not uninstalling. Aborting requests to the overlay's host is the closest thing to a controlled off-switch available from outside a site, but a page could in principle behave differently when a script fails to load than when it was never referenced. And, as the third episode above shows, the block only works if your host list is right — which is now something the code checks rather than assumes.
Small samples. 92 store-runs, 179 pairs, three samples. The spread between runs is the honest error bar, and it is why the headline is a median across stores rather than any single run's total.
None of this is legal advice or a compliance assessment. Nothing here establishes whether any site meets any legal requirement in any jurisdiction. Conformance is a judgement about a whole product, and a scanner is not competent to make it.
What is not published
No store domains, no raw HTML, no screenshots. These stores did not agree to be measured, and a list of named sites with their violation counts is a list of targets, not a methods note.
The cost is real and worth stating plainly: you cannot check my arithmetic against my inputs. You can only re-run the method on a sample you draw yourself. The code, the exact versions, the delays, the sampling procedure and the validity rule are all published so that this is possible:
https://github.com/VanyaNeytrino/overlay-ablation
Run it twice on the same sample before you believe any number it gives you — and check that the widget was actually switched off before you believe it did nothing.
Measured with axe-core 4.12.1 via @axe-core/playwright, Playwright 1.62.1, Chromium 151.0.7922.34, tags wcag2a wcag2aa wcag21a wcag21aa, viewport 1440×900, 12,000 ms settle for ablation and 5,000 ms for the survey.
Top comments (0)