DEV Community

Kynth
Kynth

Posted on

Auditing a 40,000-page government site for WCAG without auditing 40,000 pages

#a11y #webdev #javascript #testing

A city of 30,000 people has more web pages than it has employees. Agendas back to 2009, a permit portal, three departments that each bought their own CMS, and 12,000 PDFs. The DOJ's ADA Title II rule gives that city a hard date to be at WCAG 2.1 AA, and the person responsible is usually one IT generalist who also fixes the printers.

You cannot run a human audit on 40,000 pages. You also cannot just run axe-core over a crawl and call the output an audit — that produces a 900,000-row CSV that proves nothing and fixes nothing. Here's the approach that actually works.

Pages don't vary. Templates do.

The insight that makes this tractable: a 40,000-page municipal site is usually 40–80 templates. Every agenda page fails the same way. If the events template has a table without a <caption>, you have one bug, not 4,000.

So the first pass is clustering by DOM skeleton, not by URL:

const fingerprint = (dom) => {
  const skeleton = [...dom.querySelectorAll('*')]
    .filter(el => !isContentLeaf(el))          // drop <p>, text-only <span>, <li> bodies
    .map(el => `${el.tagName}.${stableClasses(el.className)}`)
    .join('>')
  return sha256(skeleton)
}

// stableClasses drops anything that looks generated:
//   /\d/ -> hashed utility classes, CSS-modules suffixes, per-post ids
const clusters = groupBy(crawledPages, p => fingerprint(p.dom))
Enter fullscreen mode Exit fullscreen mode

Two knobs matter more than the hashing:

isContentLeaf is the whole ballgame. Include too much and every page is its own cluster. Include too little and the permit form collapses into the same bucket as a press release. Tune it until cluster count stabilizes across two crawls a week apart — that's your signal the fingerprint is keyed on structure and not on content.

One sample per cluster is not enough. Take three: shortest, median, and longest page by content length. The long tail is where the embedded YouTube video, the nested data table, and the 2011 Flash-era markup live. Same template, wildly different violations. This one change roughly doubled real findings for us.

40,000 pages → ~60 clusters → ~180 audited pages. That's a week of human review, not a year.

Automated scans prove about a third of it

This is the part most "compliance scanners" quietly skip. axe-core reliably detects roughly 30–40% of WCAG failures. It cannot tell you whether alt text is accurate, whether focus order is logical, or whether an error message is understandable. A tool that reports "98% compliant" from a scan is reporting a number it did not measure.

So every success criterion gets a three-state ledger entry, per template:

  • machine_verified — a rule mapped to that criterion ran and passed
  • human_verified — a reviewer performed the documented procedure (keyboard-only pass, screen-reader pass, contrast spot-check)
  • not_applicable — with a written reason, because "N/A" without a reason is the single most common thing that gets a conformance claim thrown out

A criterion is only conformant if it's covered by one of the first two. The ledger makes the gap visible instead of hiding it behind a percentage.

Evidence has to be frozen

An audit finding is worthless six months later if the page has changed. Every finding stores a content-addressed evidence bundle: raw HTML, a full-page screenshot, the axe JSON, the response headers, an RFC 3161-style timestamp, and a SHA-256 over all of it. Findings reference the hash, never the live URL.

That also gives you cheap remediation tracking — re-crawl, re-fingerprint, diff the hashes. Templates whose fingerprint changed get re-audited. Everything else carries forward.

PDFs are where the real liability is

Municipal sites are PDF delivery systems wearing a website. Check for a /StructTreeRoot (untagged = fail), a document title, a language tag, and whether the text layer is real text or a scan. Cluster those too — the "agenda" generator produces one failure mode across 4,000 documents, and fixing the generator fixes all of them.

What comes out

Not a dashboard. A binder: scope and methodology, the sampling justification, the criterion ledger, per-finding evidence with hashes, a remediation plan with owners and dates, and the conformance statement. That's the artifact that answers a complaint.

That's how we built CivicBinder.

MVP in a Month — $12,000, ~4 weeks. Empty repo to a shipped product doing real work, like the one above. See the work and what it costs.

Top comments (0)