DEV Community: Michael Fairchild

AI-generated accessibility, an update — frontier models still fail, but skills change the game

Michael Fairchild — Thu, 21 May 2026 14:40:00 +0000

A few months ago I shared early results from the A11y LLM Eval project, a benchmark that measures how accessibly LLMs generate UI code. The previous post showed that LLMs default to inaccessible code, explicit accessibility instructions can dramatically change that, and manual testing is still essential.

The latest report is out, with new models, a redesigned test scope, and a brand new mechanic: skills. Two things stand out:

The newest frontier models (GPT‑5.5, Claude Opus 4.7, Gemini 3.1 Pro Preview, Claude Haiku 4.5, and others) still fail accessibility checks by default.
A well-written skill can produce the highest pass rates we've measured. Skills can even let a weak-baseline model outperform the leaders, though they can cost more tokens to run.

Note that the pass rate reflects only this harness's automated checks (a curated set of axe-core WCAG rules plus hand-written assertions per test case). Automated testing can detect only a subset of accessibility issues: 100% here means the sample passed every check that was run, not that the page is WCAG conformant or fully accessible.

TL;DR

Default ("control") accessibility is still bad: average pass rate is 12%, with GPT‑5.4 Mini leading at 25%. Newer does not mean more accessible.
Custom instructions still pay off. The Basic instruction set lifts pass rates by +48.5pp to 60%.
Skills go further. The Building Accessible UI skill, run as a two-turn Generate then Review workflow, hits 86% pass rate (+74.6pp). The top performer is Gemini 3.1 Pro Preview, a model that scored only 8% on control.
The skill's review turn costs about 5.5× the input tokens of control. Quality is not free.

What's new in this report

Results are now fully agentic. The previous report called LLM APIs directly with a single prompt and response. This report drives each evaluation through the GitHub Copilot SDK as an actual agent, with tool use, multi-turn reasoning, and the same instruction and skill loading mechanics that Copilot agents use in production. The numbers below reflect how these models behave when wrapped in an agent loop, not in a one-shot API call. It also means the new "skills" variant is only possible because we're running through an agent runtime.
8 models evaluated across 32 prompt cases (1,280 control samples). The model lineup is mostly fresh: GPT‑5.4, GPT‑5.4 Mini, GPT‑5.5, Claude Opus 4.7, Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Pro Preview, and Gemini 3 Flash Preview.
Instruction sets tracked are now Basic and Minimal. The Detailed expert-level instruction set from the previous run isn't in this report.
Skills, reusable task-specific guidance packages, were added as a new mechanic. The first skill evaluated is Building Accessible UI. More on this below.
A new variant token and pass-rate snapshot lets you compare quality against token cost across control, instructions, and skill turns.

Control: frontier models, same accessibility problem

The headline finding from the previous post was that LLMs default to producing inaccessible code. With the latest models in hand, that hasn't changed.

A few observations:

GPT still leads on control, but the top score is well below the 41% GPT‑5.2 posted previously. The prompt set has changed and the assertions are stricter, so the numbers aren't directly comparable, but the conclusion is the same: nobody is shipping accessible code by default.
Claude Haiku 4.5 sits at 3%, with an average of nearly 6 WCAG failures per sample. Sonnet 4.6 and Gemini 3 Flash Preview aren't far behind.
The hardest test case, Shopping Home Page (React, Dark theme), produced a 0% pass rate with 15.55 average WCAG failures across all models. Component density compounds the problem fast.

The training-data hypothesis from the previous post still seems to fit. The open web is overwhelmingly inaccessible, so models trained on it inherit those patterns regardless of how capable they are at code in general.

Instruction sets: still the cheapest win

Custom instructions are still the fastest thing a team can ship to improve accessibility.

The Basic instruction file produces nearly 5× the control pass rate while only increasing average input tokens by roughly 50%. Even the one-line Minimal instruction ("All output MUST be accessible.") more than triples it.

If you do nothing else, ship an instruction file. The basic instructions are a good starting point to customize for your team's stack and design system.

Skills: the new mechanic

The biggest change in this report is the introduction of skills. Where instruction sets are always-on guidance loaded into the agent's context for every task, a skill is a reusable, task-specific package that bundles guidance, examples, supporting files, scripts, and a tool-use workflow. The agent loads the skill only when it's relevant, and within a skill, only loads the slice it needs for the task at hand.

This matters because it changes two things at once: what guidance the model sees and when it sees it. Skills can carry far more detail than an instruction file without flooding the context window, and the two-turn Generate then Review pattern gives the model a structured second look at its own output before it's done. Together that's why skills outperform instructions in this report.

The first skill evaluated is Building Accessible UI. It is purpose built to:

Activate when the agent is creating UI, so generated code is more accessible by default.
Contain expert-level guidance and checklists for many components and patterns.
Only pull the guidance for the specific component or pattern being built, to limit token impact on the context window.
Run a two-turn workflow: Generate the UI, then Review it against the skill's checklist and fix issues.

Variant	Pass rate	Delta vs. control
`Building Accessible UI`, Generate (turn 1)	82%	+70.4pp
`Building Accessible UI`, Review (turn 2)	86%	+74.6pp

A few observations:

The best model under the skill is Gemini 3.1 Pro Preview, the same model that scored 8% on control. With the right scaffolding, a weak baseline can outperform the leaders.
The review turn pulls its weight. Asking the agent to self-check against the skill's checklist adds 5.6pp on top of an already strong first turn, closer to how a human accessibility reviewer actually works.
Skills don't dominate the prompt. Only 14 to 18% of input tokens come from the skill itself, compared to 100% for instruction sets. Most of the context window is still free for the actual task.

The cost question

Skills win on quality, but they aren't free.

The skill's review turn averages roughly 5.5× the input tokens and 2.7× the API calls of control. At scale, that's a meaningful budget impact.

A practical split:

Instruction sets are broad, always-on guardrails. Cheap, simple to ship, and the best ratio of accessibility improvement to tokens spent. The downside is that the token impact of custom instructions can add up quick if your project has instructions for multiple domains, such as accessibility, security, content, etc. Use these as the default for any team but keep them short.
Skills are focused, procedural guidance for higher-stakes work, or when you have the budget.

Recommendations

The advice from the previous post still holds, with one new lever:

Ship a project-tailored instruction file today. Start from the basic instructions and customize for your stack, design system, and component library.
Add a skill for high-stakes UI work if your token budget allows. The two-turn Generate then Review pattern materially improves outcomes.
Bake automated accessibility checks into CI/CD and block PRs on regressions.
Keep manual testing by humans, including people with disabilities. None of these tools (automated checks, instructions, or skills) can cover all accessibility requirements.

Closing

The trajectory hasn't changed. AI keeps scaling how much UI we ship, and the open-web training data keeps scaling inaccessible patterns alongside it. Frontier models alone aren't going to fix this; the latest results from Claude 4.7, Gemini 3.1, and GPT‑5.5 make that clear.

The toolbox has changed, though. Instructions still pay off in minutes. Skills are a new lever, and a powerful one: capable of taking an 8% baseline model to 86% on these checks. The work now is to pick the right tool for the right task, enforce it in CI, and keep humans in the loop where it matters most.

See the full report and the a11y-llm-eval repository on GitHub.

Embedding Accessibility into AI based software development

Michael Fairchild — Thu, 12 Mar 2026 16:07:21 +0000

At CSUN-AT 2026, I spoke with my colleague Mallika Meiyappan on Embedding Accessibility into AI based software development. Here are some key take aways.

AI is causing rapid transformation across the entire development lifecycle. It's embedded in design tools, developer workflows, content creation, and user experiences. This speed and scale increase both productivity and risk of scaling accessibility issues.

Unless accessibility is intentionally built into AI powered workflows, we risk scaling accessibility barriers as fast as we scale productivity.

TL;DR

AI is scaling development speed—and accessibility problems if accessibility isn’t intentionally built into AI workflows.
LLMs generate poorly accessible code by default, largely because they’re trained on web code where most sites already have accessibility issues.
Explicit accessibility instructions dramatically improve results, with structured guidance pushing some models from near-zero to over 90% pass rates.
Teams should embed accessibility into AI tooling and pipelines, using custom instructions, CI/CD checks, and continued manual testing.

LLMs don't do a great job of generating accessible code

At Microsoft, I built an evaluation tool to benchmark how well LLMS produce accessible code. That tool is available at github.com/microsoft/a11y-llm-eval. The tool contains a test suite of prompts to generate pages and common components, then evaluates the resulting code against the axe-core automated scanner via playwright. Axe-core is great, but it's a generic testing tool and can't test keyboard behaviors or know to expect certain semantics or other behaviors. Because of this, each prompt has an additional suite of custom tests that go beyond what axe-core can test.

That being said, it's important to note that these tests do not fully evaluate WCAG or guarantee fully accessible results. Manual testing is still essential.

The prompts do not contain anything about accessibility. This is done to establish a baseline/control for how well the LLMs produce accessible code by default, without explicit prompts for accessible code.

The results paint a pretty bleak picture. View the most recent report.

GPT 5.2 takes the lead with 41% passing.
The top 3 models are all GPT models.
The rest of the models score zero or near zero, including Gemini 3 pro, Grok 4 Fast Non-Reasoning, Gemini 3 Flash Preview, DeepSeek V3.2, Claude Haiku 4.5, Claude Sonnet 4.5, and Claude Opus 4.6.
This results in an average score of about 10% across all models.

Why are results so bad?

It's difficult to know for sure, but what we do know is that about 95% of websites have accessibility issues. So it's safe to assume that these models are being trained on code that is inaccessible, and thus producing results that are inaccessible.

So why is GPT so much better? I'm not sure, but my guess is that they are training on a higher quality data set that has more accessible code than what you would find generally in the wild.

What can devs do to improve results?

This is where custom instructions for accessibility come into play. Custom instructions are files (usually .md files) that enable you to define common guidelines and rules that automatically influence how AI generates code. Visual Studio Code has some great documentation on this. If set up correctly, the agent will automatically use these instructions for all prompts.

As part of the LLM-Eval project, I've benchmarked 3 different custom instruction files for accessibility.

Minimal: just says "All output MUST be accessible." This alone, resulted in a 18 percentage point jump.
Basic: says "All output MUST be accessible. Use semantic HTML first; only use ARIA when necessary, and ensure full keyboard support. Conform to WCAG 2.2 Level AA." This resulted in a 37 percentage point jump.
Detailed: is full-on expert level guidance. This resulted in a 48 percentage point jump.

So just mentioning the word "accessibility" has a huge impact in results.

But if you look closer, with the detailed instructions set:

Some models, like GPT scored over 90%.
Other models only saw marginal improvements.
Still other models, kept scoring zero (looking at you Grok).

So what instructions should I use?

I've published the detailed instructions at the Awesome Copilot project. This is a great place to start.

But these instructions are still generic. It's best to customize your instructions to fit your specific project. GitHub has great guidance on this. Here are some tips:

Define team and project specific workflows, tools, standards, design systems, and component libraries.
Use precise language, like MUST, MUST NOT, SHOULD, and SHOULD NOT.
Use lists to format your instructions when possible. LLMs love structure.
Ask an agent to optimize your instructions. This can be very helpful.
DO NOT paste entire standards or guidelines like WCAG or ARIA in your instructions. This will often result in worse code.
DO NOT put critical resources behind links - agents will not follow these links.

What about other aspects of software development?

AI is having a huge impact on all aspects of software development. Here are some insights and opportunities:

Research

Change: AI is being leveraged to speed up product and UX research. "Synthetic users" are AI bots that pretend to be users and give feedback and insights on ideas and designs. Additionally, AI is analyzing more data than ever and identifying trends that result in new features or changes.

Opportunity: "Synthetic users" can be used to help provide quick feedback on accessibility too, but they cannot replace lived experiences and insights from people with disabilities. It may also be possible to leverage AI to help detect accessibility issues from customer feedback and data insights - but we need to be careful and mindful of privacy.

Design

Change: Designers are being asked to use AI now more then ever. AI is being used for rapid prototyping, and some designers are moving away from static designs to vibe coded prototypes. Speed is a huge pressure, and it's common for designers and developers to work in parallel, rather than a classic hand off from design to engineering. We are even seeing a desire for designers to contribute directly to production code, but this has yet to become a reality.

Opportunity: AI can be leveraged to help designers annotate for accessibility quickly and accurately, as well as review their designs and annotations. Additionally, designers can leverage custom instructions for accessibility to improve their vibe coded prototypes.

Testing

Change: Development is happening a much larger scale than ever before, and testing is struggling to keep up.

Opportunity: Leverage AI to facilitate and assist in testing. Clear policy and quality gates are now more important than ever and need to be consistently enforced. Ensuring that accessibility is baked into the CI/CD pipeline and blocks pull requests is essential. Manual testing by humans remains essential.