We are nearing towards the end of 2019, where we are witnessing the introduction of more aligned JavaScript engines from major browser vendors. Which often strikes a major question in the back of our heads as web-developers or web-testers, and that is, whether cross browser testing is still relevant? If all the major browser would move towards a standardized process while configuring their JavaScript engines or browser engines then the chances of browser compatibility issues are bound to decrease right? But does that mean that we can simply ignore cross browser testing?
Well my friend, if you are thinking about it that way then you are wrong! There is still a relevance of browsers compatibility testing across different browsers.
We recently had a discussion with Joe Colantonio, a virtuoso, in the field of test automation and currently heading full time the podcast and discussion platform TestGuild. Our Director of Product, and Growth, Mudit Singh, went live on a podcast with Joe to address the questions around the relevance of browser compatibility testing in today’s world.
This article is a verbatim excerpt of the discussion, and focuses on the points that were addressed during our conversation with Joe!! That way, you can have a thorough idea of how pivotal cross browser testing is for your web-applications in the time yet to come.
“So I guess the First Question I have is around Cross Browser Testing, is it still relevant?”
That’s the most frequently asked question and the answer is Yes!! To understand whether cross browser testing is still important, we first need to dwell upon the history of browser development itself, so there is a term that people use nowadays and it’s called browser war. On April 4, 1994, the first browser version was launched and it was named Netscape. Netscape was being widely used until it met its first competitor in August 1995, which came to be known as the Internet Explorer.
Introduced by Microsoft, the Internet Explorer turned out to be a favourite very quickly and that triggered the first-ever browser war. And for the next decade, in spite of the introduction of the Opera browser in 1996, IE enjoyed the dominant market share.
In 2003, Apple’s Safari was launched, leading to demise for Netscape as now every Macintosh system who had previously offered a Netscape browser was now going to deliver the Safari browser.
In 2004, Mozilla Firefox was launched as an open-source browser, and in 2008, Google Chrome came into the picture as well. Since then, Internet Explorer struggled to retain the market share as it once held. Internet Explorer didn’t follow the W3C guidelines, so the websites developed during the dominance of IE started rendering weirdly in the newly introduced browser who happened to follow the W3C guidelines. This led to the inception of browser compatibility issues. Internet Explorer being beaten by modern browsers turned out to be a problematic browser for web developers and resulted in a mundane existence. We have talked about the death of IE with great details in one of our recent articles, make sure you give it a read.
In 2015, Microsoft dropped the support for Internet Explorer as they introduced a new browser called Edge. However, Edge has ever since struggled to gain relevant browser market share as a majority of Internet Explorer continue to use it even today, even though the browser has been ditched from any update in the last 4 years. Now, Microsoft is looking to bring a Chromium-based Edge browser and the expectations from that one is a little higher, from my end.
It is amazing to realize that Chrome which was introduced to the browser war field, so late in 2008, ended up dominating the browser market share in just 4 years, as it dethroned Internet Explorer from the top in around May-July 2012, and has been the leading browser ever since. On the other hand, Internet Explorer continues to fall.
So there came multiple players in the market and each has their own different technology that they’re using in the backend each has their different motivations of developing the same piece of technology called browsers that helps you to access any website over the internet.
Right now if you look at different browsers such as Google Chrome, Mozilla Firefox, Safari, Opera, and more. You will realize that each of these browsers uses a totally different technology in the backend, for example, Google Chrome uses a browser engine called Blink and JavaScript engine called V8. On the other hand, Mozilla Firefox uses a different browser engine called quantum and JavaScript engine called SpiderMonkey. As each browser has a different engine working in the backend to render the same piece of an HTML, CSS and JavaScript code, then the results may come out to be totally different in different browsers.
Browser Engines: The Crux Of Cross Browser Compatibility
“How does one test for that thing, is it more than visual validation or could you give us an example of a solution to get around that particular issue with running tests on different browsers versions and devices?”
Right so the major pain point with Cross Browser Testing is that you would have to test the same piece of code for your website or web application across all the different browsers, operating systems and mobile devices. And as much as we may wish to avoid the browser compatibility issues, unfortunately, there is no workaround for this. Even if your developer team happen to follow the best guidelines at present by taking all of the well-supported web elements for your web application. There are chances that any one of those web elements may end up a stray from the browser support on a later date. And figuring out the ripple effect of that can be very exhausting.
So it is recommended to perform cross browser testing in every release cycle, on all of the browsers, browser versions, and operating systems. If not all then you can start off with the browsers that matter the most to your business and based on that you can define your cross browser testing strategy.
Each browser has a different demographic, so you can differentiate based on the target audience. Your website or web app that you’re going to build, would have differences in the targeted audience on the basis of the device they’re using.
Which Browsers Are Important For Your Cross Browser Testing?
So for example, if it’s a banking application, banking software which is used by mainly can say bank employees then it would be a target for desktop browsers. However, if you’re building a news website or social media channel then you know most of the users would be using it on mobiles. So basically, you can priorities your testing based on your audience. There are a number of tricks which you can do.
First, is to obviously realize the targeted audience. You can refer to it as Predictive Analysis, where you are predicting which type of audience will come and based on that you are creating your cross browser testing strategy.
The other one would be using the Reactive Strategy. For example, you would have the Google Analytics code configured in your Website or Web-application. You can leverage that tag to get information from Google Analytics about the browsers, operating system even browser versions, and devices that are being used by your website visitors to access your website.
You can find it inside Google Analytics by going to the Technology section in the User Section based.
On that basis, you can define a strategy and build your cross browser testing matrix indicating the most important browsers, browser versions and devices on which you need to test your web application for browser compatibility.
After realizing the most important browsers for your business, and developing a matrix. The next step is to figure out the execution part. What would be the best approach to test on all these browsers?
Earlier what people used to do was that they would create an in-house device lab. If you are a big enterprise then you can purchase multiple different devices or multiple desktop.
So for example, say you want to test on Safari browsers for which you would need a Mac device, either laptop or tab or maybe both. Similarly, if you want to test on the Edge browser versions then you would need a Windows device. If your targeted audience uses both Edge and Safari browsers then you would require various Mac devices and Window devices. This way, you would have to invest a considerable amount of figure over different hardware.
This is an expensive method, agreed! But if you have a very limited number of browsers that you want to test on, then this is the fastest ideal solution for you. However, if you have to test on a huge list of browsers and browser version then the best solution right now would be to opt for a cloud-based approach.
This is where LambdaTest as a cross browser testing cloud can be a perfect choice for you. With LambdaTest, you can test across 2000+ real browsers running on real operating systems, hosted on the cloud. So you can completely eliminate the hassle of investing and maintaining an in-house device lab as you can go for a cloud-based solution with zero-downtime. That is not all though, you can also enjoy the benefits of third-party integrations with tools for bug-tracking, project management, instant messaging, and continuous integration.
“I definitely agree! You don’t want to spend your time as a tester, worrying about a lab when you have solutions like LambdaTest. A pushback though, what happens if I wish to use Selenium Grid or a Docker, or maybe use alternatives like Zalenium?
What are the Pros and Cons if someone is going for a Selenium Grid Type Approach?
Selenium Grid is definitely a great way to set up an automation system and test on different browsers. So let’s start with the cloud-based approach as well.There are multiple ways that you can create a cloud-based infrastructure for yourself.
You can go to a service provider like LambdaTest which have built-in machines that are ready to fire up, ready to scale up, allowing you to run your test on them directly.
Another approach would be to create and set up this kind of infrastructure for yourself. So you go to cloud-based infrastructure provider , for example, AWS(Amazon Web Services), Rackspace or Digital Ocean. Or you can even go to a Bare Metal Host Server provider , and then use containers, dockers, and Zalenium itself, to create a Grid of your own.
The challenges with cross browser testing using a Grid setup of your own would be around the limited number of browsers over which you can perform cross browser testing. But that is not all, there are further limitations on that as well. For example, mac machines if you want to test on safari then you need an online Mac solution provider and they are totally different than Windows or Linux solution providers.
So Mac only works with authorized Mac resellers and are very costly to even lease on different resellers. You can go to MacStadium and ask them to lease you some of the Mac devices online.
Another challenge with cross browser testing using this approach is to maintain the browser versions so Google Chrome has a very fast release cycle every 2-3 months they come out with a new browser version, Firefox is toe to toe on that context. Now, every time a new version is introduced, you would have to change your Docker image to update this change in the browser. You would also have to maintain your browser infrastructure every time.
This drill would also be required whenever a new operating system is launched so, for example, Mac is going to launch their new operating system named Catalina. Now, every time when a new operating system is launched you have to update your all the infrastructure to match the new updates.
There is another challenge with cross browser testing using a Grid setup on your own infrastructure comes in the execution of the test itself. A cloud-based server will usually have a little bit of latency because there is a cloud network involved. Which means that every command thread will take a little bit longer to go from your local machine to the cloud infrastructure and then execute back.
So test thread wise execution takes a little bit longer. However, the advantage here is that you can leverage parallel testing. With parallel testing, you can create multiple different machines at the same time, and you can test on these machines, simultaneously. This will reduce your test execution time by many folds.
Let us take an example, say you have 100 tests and each test takes 1 minute to execute. That means it would take 100 minutes to execute your whole test suite in a sequential manner. However, if you have 10 different machines and you are leveraging them for parallel testing then your whole suit will complete within 10 minutes so you cut down your test series execution by a factor of 10 by simply increasing the number of machines over which you want to test on and this again increases a new added work for the DevOps team that they have to maintain test infrastructure of ten different types of machines they have to scale up those machines, scale down they have to optimize the performance they have to clean these machines every time the test is completed.
This is an added overhead related to the DevOps team but it is said by the fact that you now have a better cross browser testing coverage under better test execution time.
“Do you find that often with Companies with bigger businesses which happen to deal with certain browser versions, that are older and not even maintained by the browser vendor anymore, and if so, are those still versions that you support yourself as a company?”
Right, exactly! That is one of the main reasons as well. Many people ask me the questions around Internet Explorer browser versions, they ask us about why do we have such browsers on board? Why should they include them in their test cycles considering how archaic they are?
They ask such questions because they believe that Internet Explorer is a long-forgotten road and nobody uses it anymore. However, if you check the available stats online, you will find that around 1-2% of people are still using Internet Explorer.
Now, say that is only a 1% market share then it may not be a big deal if your traffic is around 100 people coming to your website. However, if a million people are coming to your website daily, weekly, monthly, or yearly then that one per cent amount to 100K. Losing your website in front of 100K visitors is like failing to achieve 100 thousand dollars of revenue. And as they realize it, then Internet Explorer suddenly becomes very important for them.
Cross Browser Testing on Internet Explorer browser which is not even provided in the latest Windows infrastructure becomes a challenge. You will not find Internet Explorer 10 anywhere in any of the latest Windows machines so you have to go back and you have to search on the internet to find older versions and Internet Explorer is easier because they had very few browser versions. But what about Firefox?
Which started with version 4 and at present has Firefox 76. There is a 60 to 70 different browser versions being involved now, and you have to collect and maintain them in your cross browser testing checklist to make sure that you are covering all the types of users that may come to your website.
“So I guess that is one extreme example is IE, but I guess another extreme I have been seeing is, more and more automation tools have come out that just support Chrome, and not even support the other browser, so have you been seeing this in the market? If so, any pros and cons for Chrome really dominating this market as of now?”
Right, So most of them use Chrome and my guess would be that the reason behind, lies with the browser market share for Google Chrome, i.e. above 70%. So quite a lot of automation tools bank upon this fact that Chrome is the most important browser for you! And in some ways, this is one of the main reasons why Selenium still doesn’t offer a 100% coverage, it is the most dominantly used test automation framework but their others as well in this test automation framework space that is giving Selenium a challenge in cross browser testing automation.
JavaScript based testing tools like Cypress, Zest, Mocha, and more who work directly with Chrome. For example, if you look at Cypress, it does not offer browser compatibility testing capability and this is why Selenium is still one of the most preferred automation testing frameworks, even in big enterprises.
I don’t mean to criticize Cypress, there are pros to that tag along. Cyprus has its own users, It is built from the ground depth to help a content developer in there testing. Right, it extremely fast when we compare it to Selenium. It uses JavaScript which makes it comfortable for the front-end developers to understand. It works, you can say it has really fast feedback and turn-around time. Debugging using Cypress is a little bit easier as compared to Selenium. However, if cross browser compatibility or browser compatibility testing is important for you. You would have to eventually make a move to a platform like Selenium which covers a broader scope of a number of browsers, in fact, nearly every major browser is compatible with Selenium.
**“So, I understand the reason why Cyprus works for Chrome and I think it is a good solution for certain folks but the other issue is, if you are using Selenium and you are running it against Chrome, and then you run it against Firefox, the results might be different. So you have to do strange hacks.
I understand it would be great to spin up all your tests against all the browsers and test all the things but at the same time, I have seen some functionality like hover-over work for Chrome than does in Firefox. So if someone is just using Selenium, are there any workarounds for that, or any best practices you have seen around in that area?”**
So, usually there are a few hacks involved to make sure that your website works in all different browsers and similarly, that goes with the testing itself. There are a few functionalities in Selenium which works perfectly with Chrome browsers, however, when you run the same code in Firefox, you may have to tweak that code a little bit.
This is something that I believe the Selenium community is still working upon, and bit-by-bit this problem will be solved in itself when the difference among browsers will kind of close around. However, there are other libraries that can help you with this task. Libraries like Modernizr and Babel JS, can be used to define how your website works on different browsers. But again! It is the responsibility of the tester to test and make sure that code for the website is working fine with different browsers.
“So, for best practices do you recommend people to run their entire test suites against all the browsers, or do you usually recommend a lot of majorities to test against Chrome and then have maybe 10 high-level end-to-end tests against all the other browsers? Is that something you can get into?”
Right, so as an automation testing platform we are usually integrated into the complete CI/CD development pipeline of the tool. We commit to continuous testing approach so every time a code is committed to the pipeline, an automatic test job is triggered using the Jenkins or any other CI/CD tool and run the entire test suite over that updated code.
So people usually prioritize browsers like Google Chrome, Safari, or Mozilla Firefox considering they are the most important ones. But then again, as I said earlier, you have to know the cross browser compatibility matrix for your business.
You must realize which browser versions are bringing you the most traffic, and from which device or operating system?
Based on that you can decide on the particular browsers for which you want to delay the test execution and the ones on which you want to hurry it.
So the most common browsers to encounter in almost every browser compatibility matrix are going to be browser versions of Chrome, Safari, and Firefox. However, IE and Edge, are also noted among the matrix and with a less comparative time execution as compared to major browsers.
“Besides cross browser functionality which is a given benefit for using an online provider. Are there any other benefits you see that people don’t think about or know about or any other feature by using an online cloud provider, that you may not even think about?”
Ofcourse, there is you can say a significant cause involved as well when you set up your own Grid. For example, if I want to set up my own Grid and run successful tests using the AWS and I want them to run on a decent pace.
I would have to go with a machine that has around 8-16 GB Ram with an SSD hard drive. These are usually what we call c5d 2x large machine and they cost at least $0.752 per hour.
Now, if we calculate on this basic, I want to run 100 tests using this machine and I want to run these tests for around 12 hours every day, over the period for 30-40 days during the 6 months of the development process. Let’s calculate how much would that be?
Here’s the costing of 101 (100 Selenium Node VMs and 1 Hub VM) machines with 12 hours per day uptime.
c5d.2xlarge(8Core CPU, 16GB Memory, 200NVMe SSD) costs $0.752 per Hour
Monthly cost = 0.752*12*30*101 = $27342.72 Per Month
Now, let me tell you about LambdaTest. Each machine provided by LambdaTest has 8Core CPU with minimum 16GB of RAM memory, with NVMe SSD to further boost test execution time.
And to run your tests over 100 parallel sessions, the pricing for LambdaTest would be $7900 Per Month.
Wondering about the significant difference in cost?
Let me explain why cloud-based provider turns out to be a cheaper cost overall.
Basically, a cloud-based provider offers a shared infrastructure. So once you perform a test execution over a machine, then those machines are been cleaned-up and then they are handed over to other users, which builds a community of users who are using that shared infrastructure.
On a shared basis you can say that the test margin of maintaining that infrastructure goes down, so in the long term these cloud-based test service providers come up cheaper.
But wait, there is more!
The other aspect is if you want to run a test over 100 different machines, you have to maintain those 100 different machines too! This would require a significant resource and bandwidth consumption.
So running 100 different machines will require to use different tools for configuring and supporting your test environments. These tools would involve names like VMware, Container, Docker, you would have to create templates of all these 100 machines. You would have to maintain those templates and you have to upgrade those template too!
Then you would have to create scripts that will automatically scale up those machines and one that accessing is them they would automatically clean machine so that in the next testing cycle, so you would have to clear all the cookies history, and cached data.
You can avoid this hectic drill by opting for LambdaTest as we will provide you with a well-maintained infrastructure on the cloud. LambdaTest will help you eliminate the need for infrastructure development, infrastructure maintenance, infrastructure scaling in-house.
We are also eliminating the need for having a dedicated DevOps team just for testing so that you can focus more on building code with better quality, resulting to faster shipping of products after validating them for cross browser testing.
“You are the first person I have spoken to since Selenium 4 has been released and I was just curious to know how a provider even handle that? Are there any new features in Selenium 4 that you think are going to impact maybe cross browser testing or testing in general?”
Right, so if you are using Selenium right now, running the new release of Selenium 4 won’t impact you that much because the new Selenium 4 will be backward compatible with all the selenium code that you have already written. However, if you want to start off by building new infrastructure, new Selenium code, new Selenium suites from now, you would have to take into account the new protocols, the new grammar that is there, involved in writing the code.
Selenium is basically made up of four different aspects.
- Selenium client libraries
- Browser drivers
- Browser
- JSON wire protocol – A collection of APIs that sends the instruction to browser driver and those browser drivers run the browsers and execute all of the instructions on the browser.
These instructions of API has changed a little bit. Since the last 4-5 years, people have been working on standardizing these API sets. And, we are now using W3C protocols, so the set of instructions sent back to the browser drivers are a little bit different. There are newly added capabilities as well.
For example, there is a new thing called the Action APIs. Earlier, it was a challenge to run multiple actions at the same time. So for mobile devices which have 2 touch gestures like zoom-in or zoom-out so these kinds of gesture would earlier not be possible. However, new action APIs will help you perform multiple different test actions at the same time and this is what is changing in the new Selenium 4.
Another change that is worth mentioning would be the depreciation. so earlier so all the deprecated API that were there on Selenium, they used to give a warning but now they will provide you errors so you would have to make sure that your Selenium code is updated to at least 3.4 or 3.X or 4.0 standards. If you are using very old versions then you may have to change a few of the APIs. But apart from that everything works perfectly fine.
Selenium 4 would be completely compatible with all Selenium 3.X version and you can use them as a test.
“Before we go, if there is one piece of actual advice that you can give someone to help them with their automation or cross browser testing efforts? Let us know the best way to find or contact you or learn more about LambdaTest!”
The best advice I would want to give to every enterprise clients is to build a browser coverage bit-by-bit. Start up with Google Chrome, start up with Firefox, Safari and then see how many users are coming to your platform and based on those statistics. You can increase your browser coverage gradually. There is a way to automate each and every aspect of the test chain but you cannot overlook the fact that human testers are still the best eyes there is, so you cannot overlook the manual or exploratory testing part. You would have to make sure that you as a developer are giving at least 3-4 hours of week on making sure that your application is working as a cross browser compatible web application, every day and night.
If you want to know more about LambdaTest, you can check our website. We have an in-app/website chat support and are super active in responding to any of your queries. So if there are any issues using LambdaTest or you wish to know more about cross browser testing or automation testing using Selenium then just give us a shout.
Thank You, Joe, For This Amazing Podcast!
We would like to thank Joe Colantonio for having this thrilling and wiseful podcast with us. We are sure that the readers and the listeners got plenty to learn around cross browser testing implementation. If you like this podcast then don’t forget to subscribe to TestGuild for such exciting sessions. Cheers and happy testing! 🙂
Top comments (0)