In October I audited OpenMRS O3, an open source electronic health record, against WCAG 2.2 level AA. I ran it locally from the official Docker images, pinned to version 3.7.1, and scanned eight screens with axe-core 4.13.0: login, location picker, the service queues home, patient search, patient registration, the patient chart, the vitals form and appointments.
axe reported ten distinct problems. Then I put the mouse away and went through the same screens with the keyboard, 200% zoom, a 320-pixel viewport and two screen readers. That found seven more failures. axe had not reported any of them, although it had flagged one of the elements for a different reason.
This article is about that gap: what the scanner saw, what it could not see and why, and how I still run axe on every pull request without the suite turning into noise. The full audit, with the raw scan, the evidence screenshots and the source verification, is public in accessibility-audits. The CI code comes from playwright-qa-showcase.
What axe found
I ran the WCAG 2.0, 2.1 and 2.2 A and AA rule tags. Per screen, the counts look like this:
| Screen | Violations (rules) | Checks axe could not decide |
|---|---|---|
| Login | 0 | 1 |
| Location picker | 1 | 0 |
| Home (service queues) | 3 | 15 |
| Patient search | 4 | 14 |
| Patient registration | 4 | 15 |
| Patient chart | 6 | 54 |
| Vitals form | 5 | 52 |
| Appointments | 5 | 14 |
Added up, that is 28 rule hits, which overstates the problem. Nineteen of them involve two global components, the help button and the user menu, which show up on almost every screen. Grouped by cause, they come down to ten findings under five success criteria: 1.3.1, 1.4.1, 1.4.3, 2.5.8 and 4.1.2. A few examples:
- The floating help button has no accessible name, so a screen reader announces only "button".
- The user menu puts
<div>elements directly inside a<ul>, which breaks the list semantics on six of the eight screens. - The date placeholders on the registration form have a contrast ratio of 1.72:1, against a minimum of 4.5:1.
- "Vitals history" is 16 pixels tall, below the 24-pixel minimum target size that WCAG 2.2 added.
All of these are real, and axe found them in seconds. When I checked the findings against the current OpenMRS source, two of the ten were already fixed upstream, including the help button. This is what automated scanning is good at, and it costs nothing to run again after every change.
The third column matters too. axe marks a check as "incomplete" when it cannot decide by itself. On these eight screens, that was 165 checks: 76 for color contrast (mostly text over the colored alert bands in the vitals tables), 87 for ARIA attributes and two others. A scan that passes with 165 undecided checks has not shown that the page is accessible. It has handed you a to-do list.
What it missed
The manual pass ran in Firefox and Chrome on Linux (keyboard, zoom, responsive mode, DOM inspection), with Orca on Linux and TalkBack on an Android phone as screen readers. I did not test NVDA, JAWS or VoiceOver in this round, and the report says so.
These are the seven failures, grouped by the reason a scanner cannot catch them.
The page had to change first. axe looks at the DOM at the moment you call it. Two failures only happen after an interaction:
-
2.4.3 Focus Order. The vitals, biometrics and conditions tables are paginated. When you change the page with the keyboard, the table re-renders, the page selector (a
<select>) is recreated with a newid, anddocument.activeElementdrops from the selector to<body>. A keyboard or screen reader user loses their place. - 2.4.7 Focus Visible. In the date-of-birth calendar, Tab moves focus into the grid of days and the arrow keys change the date, but the focused day has no visible style. The date in the field changed from 06/02/1959 to 26/03/1959 while I watched nothing move on screen.
The viewport had to change. I scanned at 1280×800. Both of these failures appear only in a different layout:
- 2.4.11 Focus Not Obscured (Minimum), new in WCAG 2.2. At 200% zoom, the chart's actions become a fixed bar at the bottom of the screen. Tabbing through the vitals card puts focus behind that bar, first partly and then completely hidden. The browser does not scroll, because as far as it knows the element is already in view.
- 1.4.10 Reflow. At 320 pixels wide, the registration form keeps its two-column layout and gets cut off on the right: half-labels, a clipped "No" button, an unreadable date field. There is no horizontal scrollbar to reach the rest, so registering a patient at that width is impossible.
Something had to be announced. In the global patient search, the "N search results" counter updates on every keystroke, but it is a plain <p> with no role="status" or aria-live anywhere above it. Sighted users see the count. Screen reader users have to leave the field to find out whether the search found anyone. That is 4.1.3 Status Messages, and no static scan can hear what is not announced.
No rule exists for it. Each result in that same search is a link that contains a button ("Active Visit"). Interactive content inside a link is invalid HTML, and screen readers handle it inconsistently. I wondered whether axe missed it only because I never scanned the open search panel, so I tested a minimal page with a <button> inside an <a> against axe-core 4.13.0. It passed. axe's nested-interactive rule only applies to roles whose children are presentational, such as button or checkbox, and in axe's ARIA table the link role does not have that flag.
The element was flagged, for a different reason. The header of the vitals card is a div with role="button" and tabindex="0" that contains a link and a button. axe did flag it, as nested interactive controls. What axe could not tell is what happens when you reach it with Tab: focus lands on it with no visible indicator, Enter does nothing, and Space scrolls the page. It is a dead tab stop that announces itself as a button. The manual finding (2.4.7 and 4.1.2) is the one a keyboard user actually feels.
So the seven failures are four about focus, two about semantics and announcements, and one about reflow. The focus failures share a pattern: OpenMRS works with the keyboard almost everywhere, but it does not always show where the focus is. One consistent :focus-visible style and scroll-padding on the fixed bars would fix most of them.
Running axe in CI without drowning in it
None of this makes axe less useful. It makes it a regression guard, not an audit. I use it that way in my showcase suite, which scans four pages of SauceDemo on every pull request.
The first decision is which rules to run:
export const AXE_TAGS = ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22aa', 'best-practice'];
export async function runAxe(page: Page) {
return new AxeBuilder({ page }).withTags(AXE_TAGS).analyze();
}
I include best-practice, which is advisory and not a WCAG failure. SauceDemo has no WCAG A or AA violations on these pages, so without the best-practice rules the mechanism below would have nothing real to work on.
The second decision is what to do with the violations you already know about. If the suite fails on all of them, people learn to ignore the red. If you skip the rules that fire, a new problem under the same rule goes unnoticed. I keep a baseline file with the violations I knowingly accept, one entry per rule per page, each with a reason and a date:
"login": [
{
"ruleId": "page-has-heading-one",
"reason": "Best practice: the page has no level-one heading; the 'Swag Labs' logo is a plain div.",
"since": "2026-09"
},
{
"ruleId": "region",
"reason": "Best practice: the logo text sits outside any landmark region.",
"since": "2026-09"
}
],
The reason is required. An entry that says only "known issue" does not tell the next person whether it is still acceptable. The file is validated with zod when the suite loads it, so an empty reason, a malformed date or a duplicated rule on the same page fails before any test runs.
The comparison is a pure function, with its own unit tests:
export function diffAgainstBaseline<T extends ViolationLike>(
pageKey: string,
violations: readonly T[],
baseline: Baseline,
): BaselineDiff<T> {
const entries = baseline[pageKey] ?? [];
const entryByRule = new Map(entries.map((entry) => [entry.ruleId, entry]));
const firedRules = new Set(violations.map((violation) => violation.id));
const diff: BaselineDiff<T> = {
newViolations: [],
stillPresent: [],
resolved: entries.filter((entry) => !firedRules.has(entry.ruleId)),
};
for (const violation of violations) {
const entry = entryByRule.get(violation.id);
if (entry) {
diff.stillPresent.push({ violation, entry });
} else {
diff.newViolations.push(violation);
}
}
return diff;
}
Three outcomes come out of it. A violation that is not in the baseline fails the test. A known one passes and becomes an annotation in the HTML report. A baseline entry whose rule stopped firing is annotated as stale, so someone removes it in a pull request instead of letting the file rot. A Playwright fixture runs the scan, attaches the raw axe JSON and a short markdown summary to every test, and makes the assertion:
// Only rule ids are compared; the details go in the message.
expect(
diff.newViolations.map((violation) => violation.id),
formatNewViolations(pageKey, diff.newViolations),
).toEqual([]);
Each test then reads as one line, await auditPage('login'), after the test has navigated to the page.
I chose two trade-offs on purpose. There is no "update baseline" mode: every entry is written by hand and goes through review, because an automatic update is how a real failure gets accepted without anyone reading it. And the baseline is keyed by rule per page, not by element selector. That keeps the file small and stable when the markup changes, but it means a second element breaking an already-baselined rule on the same page is not flagged. Keying by selector would catch it, at the cost of an entry that breaks whenever the markup changes.
The latest run, on 16 November 2026 with axe-core 4.13.0: no new violations on the four pages, five best-practice findings in the baseline (the missing level-one heading on all four pages and content outside a landmark on the login page), and three color contrast checks on the inventory page that axe could not decide. Those three go on the manual list, like the 165 on OpenMRS.
Narrowing the gap
Some of the seven manual failures can become automated checks once a human has found them. The cheapest step is to scan states, not only URLs. The scanner I used for the audit takes an optional selector to click before the scan, which opens a panel or a dialog first:
for (const { name, url, waitFor, click } of pages) {
// SPAs that poll never reach "networkidle": wait for the page's own selector plus a short render pause.
await page.goto(url, { waitUntil: 'load' });
if (waitFor) await page.locator(waitFor).first().waitFor({ timeout: 60_000 });
await page.waitForTimeout(2_000);
if (click) {
await page.locator(click).first().click();
await page.waitForTimeout(2_000);
}
const result = await new AxeBuilder({ page }).withTags(TAGS).analyze();
// ...write result to disk
}
That is how the vitals form got scanned at all: it is a side panel that only exists after clicking "Add".
The next step is to write a plain Playwright test for the behavior a manual finding describes. The focus order failure is a good candidate: change the table page with the keyboard and assert that the focused element is still the page selector. The reflow and focus-not-obscured failures need the same flows run at 320 pixels and at 200% zoom. I have not written those tests yet. When I do, each one will start from a finding a person made with a keyboard, not from a scanner rule.
What does not automate is the judgment. Whether the counter needs to be announced, whether the focus indicator is visible enough, whether a cut-off form is still usable: those calls need someone who uses the page the way a disabled user would.
How I split the work now
On every pull request, axe runs with the baseline, and a new violation blocks the merge. It is fast, and it keeps fixed problems from coming back.
Before a release, or when a team is about to sign an accessibility conformance report, someone does the manual pass: keyboard only through the main flows, 200% zoom, 320 pixels wide, and at least one screen reader. On OpenMRS that pass produced seven of the seventeen findings in the report, and every one of the four focus failures. If a product only has the scanner, those are the failures its keyboard users meet first.
Top comments (0)