Recently I was asked about cross browser / device testing solutions such as Browserstack, Sauce Labs or Lambdatest.
Having used one of these tools in the past, and having researched the others, I have formed an opinion on these solutions and the grid approach to cross browser testing.
There are only 2 main code bases for browsers, and the versions are not a critical factor anymore
Browsers are converging into 2 main choices; Chromium-based and Firefox. All Chromium based browsers behave near identically in testing. The versions are kept up to date automatically and tested rigorously. Our value can be in testing the canary/beta versions of these browsers to gain confidence that things will be fine when browser updates happen.
The OS does not matter on desktop, mobile browsers boil down to viewport sizes as opposed to device types
There was a time these did matter, especially in desktop development and a while ago in mobile development. In web development, this parameter is not critical, and if it becomes so there are effortless built-in solutions such as Github Actions build matrix.
So what is the meta solution today? How do we get away from the grid approach but still have a confidence in our products?
Given that the main parameters of concern today are the browser type and the viewport, the below 3 strategies can be harmonized to gain the maximum value with minimal effort.
Combinatorial Testing can be applied to CI, considering browsers, deployments and a subsets of tests. Check out this slide deck (scroll down with down arrow), it showcases how we applied theoretical math to reduce 72 CI test config combinations to 5.
Combined (unit+ e2e) code coverage can be utilized to decide on a lean subset of tests that covers the main workflows and at the same time hits a good percentage of the source code.
Visual AI testing can be used to not only cover browser & viewport concerns but also ensure that there are no visual and/or CSS regressions in the app. In this example of a run with Percy.io, 1 test covers 2 browsers x 3 viewports.
Top comments (3)
Good post! Btw, I started becoming a huge fan of visual regression tests with Cypress.
Just wanted to point out that we still have three major browser engines, don't forget about Safari. Even though Webkit and Chromium have the same origins, Safari seems to be slowly becoming the IE of the modern age. We find many issues with Safari that tends to do things slightly different - so it's always good to have someone check the web app on a Mac as well.
Thankfully Safari is coming soon on their roadmap docs.cypress.io/guides/references/...
Nice point of view on handling different types of testing in current landscape where we have so many different tools and ways of testing.