DEV Community: Clyde Garret

A Practical Guide to Smoke Testing: Scenarios, Metrics, and Common Mistakes

Clyde Garret — Fri, 26 Jun 2026 12:15:43 +0000

There’s a lot of confusion in the testing community regarding what smoke testing is, and whether it is really all that different from sanity testing and regression testing.

Having a mental image helps clear that confusion. The name “smoke testing” can be traced back to hardware engineering, where powering on a new circuit board was considered successful if no "magic smoke" escaped from it. Many also point to plumbing, where smoke was pumped through newly installed pipes to detect leaks before water was introduced in them.

Therefore, smoke testing is the first checkpoint after a new build is deployed. I’ll cover the practical aspects of smoke testing, including example use cases and key metrics, in this guide.

What Is Smoke Testing?

Smoke testing is a quick set of tests executed immediately after deploying a new build to verify that the application is stable enough for further testing. Rather than validating every feature, it checks the application's critical workflows, core integrations, and environment health.

A smoke test answers one question: can the QA team confidently proceed with deeper testing, or should the build be rejected?

Objectives of Smoke Testing

A good smoke test is designed to achieve four objectives:

Validate build stability: Confirm that the application is functional after deployment.
Catch critical failures early: Detect issues such as deployment errors, broken dependencies, configuration problems, or unavailable services before extensive testing begins.
Save testing effort: Prevent QA teams from wasting hours testing a broken build.
Provide rapid feedback: Give developers immediate visibility into release-blocking issues so fixes can begin without delay.

Think of smoke testing as an early quality gate. Passing a smoke test doesn't mean the application is bug-free; it simply means the build is healthy enough for comprehensive functional, integration, and regression testing.

Smoke Testing vs. Sanity Testing vs. Regression Testing

While these terms are sometimes conflated, they serve different purposes.

Smoke Testing	Sanity Testing	Regression Testing
Verifies the overall build is stable right after deployment.	Verifies that a specific bug fix or newly implemented feature behaves as expected.	Verifies that recent changes haven't broken existing functionality.
Broad but shallow coverage.	Narrow but deeper validation.	Broad and comprehensive coverage.
Runs immediately after every build deployment.	Runs after targeted code changes or bug fixes.	Runs before releases or whenever significant code changes occur.
Goal: Decide whether testing should continue.	Goal: Verify the intended change works.	Goal: Ensure existing features continue to work correctly.

A typical workflow looks like this:

Code changes → Build & Deploy → Smoke Test → (Pass) → Sanity Test (if validating specific fixes) → Full Functional & Regression Testing → Release

Suppose a new build of an e-commerce application is deployed.

A smoke test verifies that users can open the site, log in, search for products, add an item to the cart, and reach the checkout page.
A sanity test verifies that a recently fixed coupon calculation bug now applies the correct discount.
A regression test checks the entire purchasing workflow along with user accounts, payments, inventory, order history, notifications, and other related functionality to ensure nothing else has been affected.

The Smoke Test Scenarios I Prioritize

Your smoke suite should contain only the workflows that determine whether the application is fundamentally usable. For a web application, these are the checks I prioritize:

Application availability: Verify the application launches successfully after deployment.
Authentication: Log in with a valid user account.
Navigation: Open key pages and ensure they load without errors.
Search: Perform a basic search and verify relevant results are returned.
Core business flow: Complete the application's primary user journey, such as placing an order, submitting a form, or booking an appointment.
Critical APIs: Verify essential APIs return successful responses.
Database connectivity: Confirm critical data can be retrieved and saved.
Third-party integrations (if applicable): Ensure services such as payment gateways, authentication providers, or email services are reachable.
Error monitoring: Verify no critical exceptions, crashes, or HTTP 5xx errors occur during execution.

Avoid adding edge cases, negative scenarios, boundary testing, UI polish checks, or exhaustive validations to your smoke suite.

Smoke Testing Best Practices

A smoke test should be fast, reliable, and focused on answering one question: Is this build stable enough for further testing? Follow these best practices:

Automate the suite: Integrate smoke tests into your CI/CD pipeline so they run automatically after every deployment.
Keep execution under 15 minutes: A smoke suite should provide rapid feedback. If it takes an hour to run, it’s probably bloated and needs to be assessed.
Focus on critical workflows: Validate only the application's core user journeys and essential infrastructure.
Define clear pass/fail criteria: Every smoke test should have a binary outcome. If a critical workflow fails, reject the build and stop further testing.
Run in a production-like environment: Test against realistic infrastructure and configurations to catch deployment and environment issues early.
Use stable test data: Keep test accounts and datasets consistent to avoid false failures.
Maintain the suite regularly: As the application evolves, add newly critical workflows and remove obsolete ones to keep the suite lean and relevant.

How to Select Smoke Test Scenarios

A useful rule of thumb is this: if the failure of a scenario would cause you to reject the build, it belongs in your smoke test suite.

There's no universal smoke test suite. The right scenarios depend on your application's purpose, architecture, and the workflows users rely on most. Whether you're executing tests manually or building an automated suite, use the same selection criteria.

Identify the critical user journeys. List the core workflows without which the application isn't usable. For example: E-commerce: Login → Search product → Add to cart → Checkout. Social media: Login → View feed → Create a post → Logout. Banking: Login → View account balance → Transfer funds.
Include critical dependencies. If your application depends on APIs, databases, authentication services, payment gateways, or other third-party integrations, include a basic check to ensure they're functioning.
Prioritize high-impact features. If you're still unsure, utilize product analytics tools and historical bug reports to see which features generate the highest traffic, and those that would immediately block users if they failed.

Common Smoke Testing Mistakes (With Examples and Fixes)

While smoke testing seems straightforward, teams fall into a few common traps. Here are the four missteps I see most often:

Mistake #1: Turning Smoke Tests into Regression Tests

The biggest mistake is trying to validate everything. For example, a QA team adds 80 checkout scenarios to the smoke suite, including invalid coupon codes, expired cards, tax calculations, and shipping combinations.

Result: The suite takes over two hours to complete. Developers wait longer for feedback, and smoke testing no longer serves as an early quality gate. Besides, the chances of test flakiness due to timeouts can go up.

Fix: Keep only the critical happy path:

User logs in → Searches for a product → Adds it to the cart → Completes checkout.

Mistake #2: Running Smoke Tests in an Unstable Environment

Sometimes the application is fine, but there are issues with the environment. For instance, a banking application might be deployed to staging without the database schema being updated. This will lead to the smoke test failing while loading account balances. A simple environment health check before executing smoke tests will eliminate many false failures.

Result: QA rejects a perfectly good build and developers spend hours debugging an environment issue instead of an application defect.

Fix: Before running smoke tests, validate that:

The correct build is deployed.
Required services are running.
Database migrations have completed.
Third-party dependencies are available.
Test data is in a known state.

Mistake #3: Writing Brittle Automation

A smoke suite should be resilient to cosmetic UI changes while still detecting functional failures. Let’s look at an example: a login test locates a button using a CSS class. This is the automation code:

await page.locator('.btn-primary-blue').click();

Then the frontend team changes the styling, and the class name changes. In such a case, the smoke suite will fail even though login still works.

Fix: Always use stable locators designed for automation.

await page.getByRole('button', { name: 'Sign in' }).click();
// or
await page.getByTestId('login-button').click();

Mistake #4: Ignoring "Flaky" Smoke Test Failures

A smoke suite is only valuable if the team trusts its results. If flaky tests are routinely ignored, it stops being an effective release gate.

Let’s take an instance where an automated smoke test fails during user registration, but a manual retry succeeds, leading the engineer to override the failure and approve the build anyway. Later, regression testing reveals the registration service fails roughly 30% of the time due to an intermittent backend issue.

Result: The team spends hours testing a build that should have been rejected immediately.

Fix: Don't dismiss smoke test failures without investigation.

Capture screenshots, logs, and API responses.
Retry only if your pipeline has a defined retry policy.
If failures continue, treat the build as unstable until the root cause is understood.

Measuring Smoke Testing Effectiveness: Key Metrics

Not every team needs to track every metric. Keep it simple, so that tracking isn’t too much of a hassle. Here are some important metrics QA teams often track:

Smoke Test Execution Time: Measures how quickly the team gets deployment feedback. Aim for 5–15 minutes. If execution time keeps increasing, your smoke suite is likely becoming a regression suite.
Build Failure Rate: Measures the percentage of builds rejected because smoke tests found critical issues. A persistently high rate often indicates unstable deployments or weak pre-merge validation.
False Failure Rate: Tracks failures caused by flaky automation or unstable environments rather than genuine defects. High false positives reduce trust in the smoke suite.
Escaped Critical Defects: Counts production issues that should have been caught by smoke testing. Every escaped defect should trigger a review of smoke test coverage.
Mean Time to Detect (MTTD): Measures how quickly smoke tests identify a broken deployment after release. Lower detection times reduce developer context switching and accelerate fixes.

Which metrics should you prioritize? They should match your team’s challenges. For example, if you deploy multiple times a day (CI/CD), prioritize Execution Time and MTTD. If your smoke suite frequently fails for non-functional reasons, focus on False Failure Rate to improve automation reliability.

Also, don't track a metric unless someone owns improving it. Every metric should have a clear owner and a defined action when it moves in the wrong direction; otherwise, it becomes another dashboard that nobody uses.

Conclusion

One of the most impactful changes in my team was redefining what belonged in our smoke tests. We stopped treating smoke testing as a mini regression cycle and instead focused on a handful of critical workflows, which led to much better coordination with developers.

As delivery pipelines become faster and AI increasingly assists with generating test cases and maintaining automation, the role of smoke testing will become even more important. AI can help create scripts, identify impacted workflows, and reduce maintenance effort, but it can't decide what is truly business-critical. That remains a testing strategy decision.

Cypress vs Selenium: Which One Should Your Team Choose?

Clyde Garret — Mon, 15 Jun 2026 07:00:00 +0000

Cypress or Selenium are frameworks that are widely used for web application testing, yet teams often arrive at very different conclusions after evaluating them.

So what makes this decision so challenging?

At a glance, Cypress and Selenium appear to solve the same problem, but once you look beneath the surface, you'll find important differences in architecture, setup, browser support, and testing workflows. Those differences can have a significant impact on how quickly your team builds, runs, and maintains automated tests.

What Is Cypress?

Cypress is a modern end-to-end testing framework built specifically for web applications. It was designed with developers and QA engineers in mind, offering a streamlined setup process and a developer-friendly testing experience.

Unlike traditional automation frameworks, Cypress runs directly within the browser and operates alongside the application being tested. This architecture allows it to provide fast execution, automatic waiting, and real-time debugging capabilities.

Advantages of Cypress

Easy setup and configuration.
Fast test execution.
Built-in automatic waiting mechanisms.
Excellent debugging and developer experience.
Rich documentation and intuitive API.

Limitations of Cypress

Limited support for programming languages outside JavaScript and TypeScript.
Historically focused on Chromium-based browsers, though browser support has expanded over time.
Not ideal for testing multiple browser tabs or windows.
Less flexible for highly complex enterprise automation scenarios.

It is often favored by teams looking for quick feedback and a simpler testing workflow.

What Is Selenium?

Selenium is one of the most established browser automation frameworks in the industry. It has been a foundational part of web test automation for more than a decade and supports a wide range of browsers, operating systems, and programming languages.

Unlike Cypress, Selenium interacts with browsers through WebDriver, which allows it to automate browser actions externally rather than running inside the browser itself.

This architecture gives Selenium exceptional flexibility and broad ecosystem support.

Advantages of Selenium

Supports multiple programming languages, including Java, Python, C#, JavaScript, and Ruby.
Broad browser compatibility.
Large ecosystem and community support.
Suitable for complex enterprise automation projects.
Works well with distributed and large-scale testing environments.

Limitations of Selenium

Steeper learning curve.
More setup and configuration effort.
Synchronization issues often require explicit waits and additional handling.
Debugging can be more challenging for new users.

Cypress vs Selenium: Core Differences at a Glance

Feature	Cypress	Selenium
Architecture	Runs inside the browser	Uses WebDriver to control browsers
Language Support	JavaScript, TypeScript	Java, Python, C#, JavaScript, Ruby and more
Browser Support	Modern browsers	Extensive browser support
Setup Complexity	Simple	Moderate to high
Execution Speed	Generally faster	Depends on configuration
Automatic Waiting	Built-in	Requires explicit handling
Debugging Experience	Strong	More manual
Enterprise Flexibility	Moderate	High
Community Maturity	Growing rapidly	Very mature

While both tools solve the same problem, their architectural differences influence everything from execution speed and debugging to browser support and framework flexibility.

When to Choose Selenium Over Cypress

Selenium is often the better choice when:

Multi-language support is a requirement across teams or projects.
Cross-browser and cross-platform testing is a critical part of the testing strategy.
You maintain large-scale enterprise automation frameworks.
Tests need to run across distributed environments, grids, or device clouds.
Your organization already has significant investment in existing Selenium-based infrastructure.

Selenium is particularly valuable in environments where flexibility, scalability, and ecosystem maturity are more important than ease of setup.

When to Choose Cypress Over Selenium

Cypress is often the stronger choice when:

Your team primarily works with JavaScript or TypeScript.
Fast feedback cycles and rapid test execution are priorities.
You want to minimize framework setup and maintenance effort.
Developer experience and debugging play an important role in day-to-day testing.
The focus is on testing modern web applications rather than complex enterprise environments.

For teams starting a new web automation initiative, Cypress can simplify test creation and reduce operational overhead.

Which One Should Your Team Choose?

The right choice depends less on the tool itself and more on your team's priorities, existing investments, and long-term testing goals.

If you're building a new automation framework for a modern web application, Cypress can help you to get started quickly. Its streamlined setup, developer-friendly workflow, and built-in testing capabilities make it particularly attractive for JavaScript-centric teams that value fast feedback and ease of maintenance.

Selenium, on the other hand, is often the stronger choice for organizations with broader testing requirements. Its support for multiple programming languages, extensive browser coverage, and mature ecosystem make it well-suited for enterprise environments and large-scale automation initiatives.

Before making a decision, consider:

Team skill set: Is your team primarily JavaScript-focused, or does it work across multiple languages?
Application landscape: Are you testing a modern web application or a diverse set of browsers and environments?
Existing investments: Do you already have frameworks, libraries, and processes built around Selenium?
Long-term scalability: Will your automation strategy need to expand across teams, platforms, and projects?

Ultimately, neither framework is universally better. The most effective choice is the one that aligns with your team's technical requirements, workflow preferences, and future automation goals.

What the Testing Community Actually Says About Both?

The discussion around Cypress and Selenium has evolved significantly over the last few years.

Across many community platforms like Reddit, Stack Overflow, GitHub, and testing communities, the focus has shifted from finding a single winner to choosing the right tool for the job.

Some common themes emerge consistently:

Cypress is often praised for its developer experience, intuitive API, and debugging capabilities.
Selenium continues to be valued for its flexibility, language support, browser coverage, and mature ecosystem.
Enterprise teams often remain invested in Selenium because of existing frameworks, tooling, and long-term automation strategies.
Playwright has become a major part of the conversation, with many teams viewing it as a modern alternative that combines broad browser support with a developer-friendly testing experience.

The broader community consensus is that framework selection should be driven by project requirements, team expertise, and long-term maintenance goals rather than popularity alone.

What Is Performance Testing? A Complete Guide for QA Teams

Clyde Garret — Sat, 13 Jun 2026 10:36:00 +0000

In 2018, a major ticketing platform crashed during a high-profile concert sale. Millions of users rushed to buy tickets at the same time, overwhelming the system. Customers were locked out, transactions failed, and the company faced a lot of backlash.

The thing is, performance failures rarely happen during normal traffic. They happen during product launches, holiday sales, viral marketing campaigns, or unexpected traffic spikes, which is precisely when systems need to perform at their best.

Nowadays, speed is the baseline expectation from users. Yet many QA teams still treat performance testing as a late-stage or one-time activity. In this article, I will explain what the fundamental issue is with the way teams approach performance testing, and how to implement it in a fail-safe way.

What Is Performance Testing?

Performance testing is the process of evaluating how an application responds under different levels of workload, user activity, and system stress. The goal is to assess not only the speed, but the application's responsiveness, stability, reliability, and scalability under realistic conditions.

The testing ensures that an application meets its Service Level Agreements (SLAs) regarding response times and uptime. Unlike functional testing, which verifies whether features work correctly, performance testing examines whether those features continue working effectively when demand increases.

The purpose of performance testing is to:

Identify bottlenecks: Uncover slow database queries, memory leaks, inefficient APIs, and infrastructure constraints.
Validate scalability: Measure how effectively the application handles growth in data volumes.
Ensure stability: Verify that the system remains responsive during sudden demand spikes.
Support capacity planning: Provide data that helps teams optimize server sizing and cloud resources.

Types of Performance Testing

Performance testing is an umbrella term that includes several specialized testing approaches.

Load Testing

Load testing measures how a system performs under expected levels of user activity. The objective is to verify that response times, error rates, and resource utilization remain within acceptable thresholds during normal operating conditions.

For example, an e-commerce platform expecting 10,000 concurrent users during a weekend sale would run load tests simulating that traffic level.

Load testing is the most common form of performance testing.

Stress Testing

Stress testing pushes a system beyond its expected limits to determine when and how it fails.

Suppose an application is designed to support 20,000 concurrent users. A stress test may gradually increase traffic to 30,000, 40,000, or even 50,000 users, to figure out the breaking point, and also evaluate recovery behavior after failure.

Spike Testing

Spike testing evaluates how systems handle unexpected traffic increases, such as the earlier ticketing platform example I mentioned. The goal is to identify weaknesses in autoscaling, caching strategies, database connections, and infrastructure provisioning.

The way to do it is to abruptly ramp up the user load to the peak point. Do not increase it gradually, as you want to simulate an instant surge. Track critical metrics like CPU usage, memory utilization, response times, and error rates to see if the system crashes or scales effectively.

Soak Testing

Soak testing, also called endurance testing, measures performance over extended periods: days instead of minutes or hours. This is done because systems generally pass short-term load tests but fail after prolonged operation.

For example, a financial application might sustain normal traffic for 72 hours to determine whether resource exhaustion gradually degrades performance.

Scalability Testing

Scalability testing examines how efficiently an application grows as resources or workload increase. To do this, I might double the number of servers and measure whether throughput also doubles.

Scalability testing becomes vital especially for rapidly growing organizations.

Load Testing vs Performance Testing

From my experience, I see many people think load testing and performance testing are one and the same. Load testing is only one component of a much broader performance testing strategy.

Performance testing is an umbrella term: it helps me evaluate how an application behaves under various operating conditions. As detailed in the previous section, performance testing is more of a strategy encompassing different types of testing, each with a different goal. The overarching goal is to understand where bottlenecks exist and how the application will perform as usage patterns change.

Load testing, on the other hand, answers a specific question: can the application handle the expected volume of users without performance degradation?

Run performance testing when you have made major updates to your core architecture, shifted to a new cloud provider, or need to map out general system vulnerabilities without knowing where things might fail.

Run a load test when you are preparing for a targeted traffic event, such as a Black Friday sale, and you need to verify that your infrastructure can handle the predicted user spike.

Key Metrics in Performance Testing

The most useful performance metrics, in my view, are the ones that reveal how real users experience the application under load and what is happening behind the scenes when performance starts to degrade.

Here are the metrics I think are most important to monitor:

Response Time Percentiles (P95/P99): Measure how quickly most requests complete and highlight worst-case user experiences. Focusing only on average response times can be misleading because averages often hide slow or failed requests. A P99 response time of 2 seconds means 99% of requests finished within 2 seconds.
Throughput (Requests or Transactions per Second): Indicates how much work the system can process successfully over a given period.
Error Rate: Tracks failed requests and application errors as traffic increases, helping identify stability issues before they become outages.
Resource Utilization: Monitors CPU, memory, disk, and network usage to uncover infrastructure bottlenecks and capacity constraints.
Latency Consistency: Measures how stable response times remain during a test. Large variations signal underlying performance problems even when average response times look acceptable.

How to Do Performance Testing: A Practical Walkthrough

Proper performance testing begins with a well-defined test plan. One mistake I've often seen in less mature teams (and something I’ve been guilty of in the past!) is focusing heavily on the testing tool, scripts, or performance metrics without first understanding the underlying business use cases.
Here's the approach I follow:

Step 1: Start with real user behavior
Before writing a single test, identify the workflows that matter most to the business. For an e-commerce application, that might be product search, add-to-cart, and checkout. For a SaaS platform, it could be login, dashboard access, and report generation.

The goal is to test how users actually interact with the system.

Step 2: Understand expected traffic patterns
Next, determine how much load the application is expected to handle. If the application is already in production, analyze traffic metrics and usage logs. If it's a new application, work with product and engineering teams to estimate expected user volumes.

For example, you might learn that:

Login accounts for 40% of requests
Search accounts for 30%
Checkout accounts for 10%
Other actions account for the remaining 20% These percentages help create realistic workloads.

Step 3: Define performance targets
Before running tests, establish clear success criteria, such as P95 response time below 500 ms, error rate below 1%, support for 5,000 concurrent users, and CPU utilization below 80%.

Step 4: Build realistic test scenarios
Rather than testing individual APIs independently, simulate complete user journeys whenever possible. For example, the journey from login, to searching for products, to completing a checkout is a complete user journey.

This approach provides a more accurate picture of how the application behaves under real-world conditions.

Step 5: Run tests gradually
Start with expected load, then increase complexity.

A typical progression is:

Baseline load test to validate normal traffic
Stress test to identify breaking points
Spike test to evaluate sudden traffic surges
Soak test to uncover memory leaks and resource exhaustion

A staged approach gives far more reliable results.

Step 6: Monitor the entire system
Performance testing is not just about response times. During execution, monitor application logs, databases, CPU usage, memory consumption, network traffic, and error rates.

When performance degrades, these metrics reveal the root cause faster than test reports.

Step 7: Analyze, optimize, and repeat
The first test run is not the final answer. Use results to identify bottlenecks, implement improvements, and rerun tests to measure the impact.

In my view, performance testing is less about finding a specific number of virtual users and more about understanding system behavior.

Performance Testing Tools

Here are some of the popular performance testing tools I’ve implemented during my career, and what my review is.

Apache JMeter: JMeter is one of the most battle-tested performance testing tools available. It supports load, stress, spike, and endurance testing across web applications, APIs, databases, and messaging systems. While its GUI makes test creation accessible to beginners, I’d typically execute tests in non-GUI mode for large-scale runs. JMeter is valuable for organizations that need a mature ecosystem of plugins and community resources.

BrowserStack Load Testing: It is designed for teams that want to run performance tests without managing their own infrastructure. As it combines browser-level and API load testing in a single cloud platform, I can evaluate both frontend user experience and backend performance during the same test run. I also like the ability to reuse existing functional tests as load scenarios and integrate them directly into CI/CD pipelines.

Gatling takes a test-as-code approach that aligns well with modern workflows. Its lightweight architecture allows it to simulate large numbers of concurrent users efficiently, and I like the built-in reporting very much. I’d pick Gatling as a strong choice for developer-led teams.

Grafana k6 is emerging as one of the most popular modern performance testing tools because it combines simplicity with strong CI/CD integration. Tests are written in JavaScript or TypeScript, making them approachable for developers and automation engineers alike. For organizations practicing DevOps and continuous delivery, I think k6 makes it very smooth and easy to integrate performance testing into development workflows.

Final Thoughts

Performance testing is sometimes viewed as a final checkpoint before release, which is an outdated way of working. Teams should treat performance testing as an ongoing practice that runs throughout the development lifecycle. More importantly, QA should recognize that performance testing is about understanding system behavior as much as it is about handling system traffic.

Another takeaway for me has been that performance testing is not a QA-only activity. Product managers and engineering leads play a role in defining what success looks like. If there is a lack of alignment on user expectations and business requirements, then there will be a gap in the performance testing strategy as well.

Open-Source Alternatives to WAVE for Mobile Application Accessibility Testing

Clyde Garret — Mon, 18 May 2026 13:57:28 +0000

WAVE is a great tool for web accessibility testing, but it does not support native Android or iOS apps. Mobile accessibility testing depends on platform-specific APIs, screen readers like TalkBack and VoiceOver, focus order, touch targets, and real-device behavior that browser-based scanners cannot validate.

That creates a major gap in mobile QA workflows. Teams often validate accessibility on the web layer while native apps still ship with issues that impact real users.

In this article, I’m covering some of the best open-source alternatives to WAVE for mobile application accessibility testing, including their strengths, limitations, and where they fit into modern QA pipelines.

What are the Key Requirements for a good Mobile Application Accessibility Testing Tool?

It should support WCAG 2.x/AA, ADA/EAA standards, and native Android/iOS accessibility APIs.
It must validate using actual TalkBack and VoiceOver behavior instead of simulated screen readers.
It should detect missing labels, incorrect roles, broken focus order, and inaccessible UI elements.
It must verify color contrast and touch target sizes across different screen densities.
It should be able to test on real Android and iOS devices to catch OS-specific accessibility behavior.
It should integrate with Appium, Espresso, and XCUITest pipelines for automated accessibility checks.
Finally, it must report with clarity; on the issue, context, screenshots, and remediation guidance developers can act on quickly.

Why Look for Open-Source Alternatives to WAVE?

The main reason to look for an alternative is simple: WAVE is designed for web accessibility testing, not native mobile applications. It works at the browser level, which means it cannot validate accessibility behavior inside Android or iOS apps.

That becomes a problem when teams need to test things like:

TalkBack and VoiceOver behavior
Focus order and gesture navigation
Native UI labels and accessibility roles
Touch target sizing across devices

This is where open-source mobile accessibility tools become useful. They integrate directly with native testing frameworks like Appium, Espresso, and XCUITest and help teams catch common accessibility issues earlier in the development cycle without adding commercial tooling upfront.

Best Open-Source Alternatives to WAVE for Mobile Application Accessibility Testing

Let’s explore the open source WAVE alternatives for mobile application accessibility testing:

Android Accessibility Test Framework (ATF)

Android Accessibility Test Framework (ATF) is Google’s open-source accessibility testing library for Android apps. It integrates directly with Espresso tests and automatically checks for common accessibility issues during UI test execution. Teams typically use it to catch accessibility regressions early without adding a separate testing workflow.

Pros

Integrates directly with Espresso-based Android test suites
Detects issues like missing labels, low contrast, and small touch targets
Works inside existing UI automation flows without additional app instrumentation

Cons

Coverage is limited to the screens and flows already covered by tests
Supports only Android applications
Can generate false positives for intentionally hidden or decorative UI elements

My Thoughts: ATF works well for Android teams already using Espresso who want lightweight accessibility checks inside existing automation pipelines. It is useful for regression coverage, but it does not replace manual accessibility validation or real screen reader testing.

Espresso Accessibility Checks

Espresso Accessibility Checks are built into Android’s Espresso testing framework and use the same underlying engine as ATF. Once enabled, they automatically run accessibility validations during UI test execution, making them a lightweight option for adding accessibility checks to existing Android automation suites.

Pros

Requires minimal setup for teams already using Espresso
Runs accessibility validations automatically during UI test execution
Detects common issues like missing labels and touch target violations early in development

Cons

Limited to Android applications
Only catches issues that can be detected programmatically during test execution
Does not validate screen reader announcements, navigation flow, or interaction quality

My Final Verdict: Espresso Accessibility Checks are useful for Android teams that want quick accessibility regression checks inside existing Espresso workflows. They work well as a baseline automation layer, but manual testing and separate iOS tooling are still required for broader accessibility coverage.

Accessibility Scanner

Accessibility Scanner is a free Android app from Google that analyzes screens for accessibility issues directly on a physical device. It scans the active screen, highlights violations visually, and suggests fixes without requiring code changes or automation setup.

Pros

Requires no setup or instrumentation
Easy to use for testers, designers, and non-developers
Useful for quick accessibility reviews on real Android devices

Cons

Fully manual testing workflow with no automation support
Does not integrate with CI/CD pipelines or regression suites
Limited to Android applications and active screens only

My thoughts on this: Accessibility Scanner works well for exploratory accessibility checks during development and design reviews. It is useful for spotting obvious issues quickly, but it does not scale for automated regression testing or continuous accessibility monitoring.

Accessibility Inspector

Accessibility Inspector is Apple’s built-in accessibility inspection tool available through Xcode. It lets teams inspect the accessibility tree of iOS apps, review element properties, and preview how content is announced through VoiceOver on simulators or physical devices.

Pros

Included with Xcode with no additional installation required
Helps verify labels, traits, hints, and accessibility hierarchy visually
Useful for debugging accessibility issues during active iOS development

Cons

Limited to iOS applications
Entirely manual with no automated testing support
Does not generate reports or integrate with CI/CD pipelines

My Verdict: Accessibility Inspector works well for validating accessibility metadata and debugging individual iOS screens during development. For continuous accessibility testing or regression coverage, teams still need automated tooling like XCUITest alongside it.

XCUITest

XCUITest is Apple’s native UI testing framework for iOS applications. While it is not a dedicated accessibility testing tool, teams can use it to write accessibility-focused assertions around labels, traits, and element states as part of existing iOS UI automation suites.

Pros

Built directly into the Xcode testing ecosystem
Supports automated checks for accessibility labels, traits, and element states
Fits naturally into existing iOS UI test pipelines written in Swift or Objective-C

Cons

Limited to iOS applications
Accessibility coverage depends entirely on the test logic teams maintain
Does not detect contrast, layout, or screen reader flow issues automatically

My Verdict: XCUITest works well for teams already maintaining iOS automation suites and looking to add accessibility assertions into existing workflows. It is useful for accessibility regression checks, but broader accessibility validation still requires manual testing and additional tooling.

Appium Accessibility Plugin

Appium Accessibility Plugin extends Appium with accessibility audit capabilities for mobile applications. It adds accessibility scan commands directly into existing Appium automation workflows, allowing teams to run accessibility checks alongside functional mobile tests across Android and iOS.

Pros

Supports both Android and iOS inside a single Appium workflow
Generates accessibility reports directly from existing mobile automation scripts
Fits into existing cross-platform CI/CD pipelines without separate tooling

Cons

Requires an existing Appium setup and additional configuration
Accessibility coverage is limited to the flows exercised during testing
Issues like screen reader flow, animations, and interaction quality still require manual validation

My thoughts: Appium Accessibility Plugin works well for teams already using Appium who want accessibility checks inside the same mobile automation pipeline. It is useful for automated regression coverage, but human review is still necessary for broader accessibility validation.

Challenges of Open-Source Mobile Application Accessibility Testing tools and How I Overcome Them

Most open-source accessibility tools work well in isolation. The problem starts when teams try to build a complete mobile accessibility testing workflow around them. This usually leads to problems like:

Separate tooling for Android and iOS accessibility testing
Manual validation still required for TalkBack and VoiceOver behavior
Fragmented reporting across multiple tools and workflows
Additional CI/CD setup and maintenance effort
Scalability issues as testing needs grow across platforms

That is usually where commercial tools start making more sense. They reduce the operational overhead by combining accessibility scans, real-device testing, reporting, and workflow integration into a single platform. The difference is not just coverage. It is the amount of manual effort required to keep accessibility testing consistent as applications scale.

The three commercial tools I keep coming back to are BrowserStack App A11y, Evinced, and Level Access Mobile Testing Suite.

BrowserStack App A11y works well for teams that want accessibility testing on real Android and iOS devices without maintaining separate infrastructure.
Evinced is useful for teams looking for deeper automated detection and centralized accessibility reporting.
Level Access Mobile Testing Suite is better suited for teams focused on structured accessibility compliance and large-scale testing programs.

Final Thoughts
From what I’ve seen, most teams do not ignore accessibility intentionally. The bigger issue is that mobile accessibility testing still feels fragmented, especially once native Android and iOS workflows enter the picture. It is easy to assume coverage exists because some automated checks are already running, while important usability gaps still go untested.

Open-source tools are still worth using, especially early on. They help teams start building accessibility checks directly into development workflows instead of treating accessibility as a last-minute audit. The important part is understanding where automation stops and where deeper testing, real devices, and human validation still matter.