While there is a huge demand and need to run Selenium Test Automation, the experts always suggest not to automate every possible test. Exhaustive Testing is not possible, and Automating everything is not sustainable.
Let us consider an example of Selenium WebDriver in this article and analyse what Selenium test scenarios are not greatly supported with it when used as a stand-alone tool. We will also see what all Selenium test scenarios are not best for automation.
Automated functional testing tests helps to ensure that your web app works as it was intended to. Learn more about functional tests, and how automating them can give you a faster release cycle!
Selenium WebDriver
Selenium WebDriver is a web automation tool, primarily works best across different browsers, browser versions, and desktop platforms (e.g. Windows, macOS, etc.). . However, if we want to automate specific test scenarios that were initially not intended to be supported with Selenium, we would be doing something wrong.
The agenda of this article is to create awareness of what Selenium test scenarios are NOT to be automated using Selenium WebDriver as a stand-alone tool.
1. Windows apps Testing
As Selenium WebDriver is specifically used for automated testing of web applications, we cannot use it for windows based applications.
For example, if we want to automate tests for a Windows application, such as a native ‘Calculator’, it is not possible via Selenium. We cannot even launch the Calculator app in the windows machine using Selenium or Appium commands.
Hence, Microsoft has been implementing and working on automating tests on Windows-based applications, which were otherwise not possible in Selenium and Appium for the last few years.
Today, Microsoft’s WinAppDriver works to a great extent when combined with the Selenium and Appium test framework on some of the native built-in apps of Windows 10 applications.
WinAppDriver sounds promising on Windows Apps Testing, yet there is a minor limitation here. It requires a C# codebase environment such as Visual Studio to run the first-level automation test scenarios, running on Windows apps before configuring it on a CI/CD tool like Jenkins.
Popular tools such as Sikuli and AutoIT are widely used to test applications of some Windows native components. that are often found integrated to an existing Selenium testing framework. This practice, in general, is not very helpful hence not much recommended for extensive usage.
We also have HP’s UFT, which is licensed and used in some windows-based application testing purposes. One needs to know Visual Basic (VB) scripting, as the tool does not have a diverse language support feature like Selenium.
2. Mac apps Testing
Similar to windows applications, Mac apps cannot be considered as Selenium test scenarios. However, Appium’s Mac driver can be integrated into an existing Selenium Test Framework to support a certain extent of basic automation on Mac applications.
As Appium Mac Driver is a layer written on top of XCUITest Framework, it would also be a better idea to go with Apple’s XCUITest Framework..
3. Native apps Testing for Mobiles/Tablets
Selenium automation testing is not suited for testing native mobile applications. Appium, when combined with Selenium, works fine on mobile applications.
Appium is a layer on top of UIAutomator, Espresso, and XCUITest frameworks. Hence, would be a better approach to go for the latter tools to achieve better efficiency and testability of maximum test scenarios.
4. Smart TV apps Testing
Selenium does not support testing of apps developed for Smart Televisions (TVs) It would most probably continue to be unsupported in the future, as Selenium test scenarios are meant for automated cross browser testing.
5. WatchOS apps Testing
As of date, automating tests developed for smartwatches (e.g. iOS, Tizen, etc.) is not possible with Selenium automation framework. There are hardly any options to realize this testing.
Undoubtedly, the same cannot be automated using Selenium anytime soon and shall remain unsupportive for some time to come.
6. Device / Hardware / Embed Testing
Selenium cannot be used to automate testing on Customised/Embedded Hardware..
Also, we always need to remember that if manual testing takes less time and effort, then automating those flows would be a bad idea, as automation test scenarios are usually considered for saving time, effort, and increasing reliability.
7. Captcha Verification
‘Captcha’ stands for “Completely Automated Public Turing test to tell Computers and Humans Apart”. Captcha is one of the challenge-response tests widely used in many websites to verify if the accessing end-user is a bot or an actual human being.
The standard practice while running automated test scenarios for Captcha, is to
Disable Captcha in test environments OR
Adding a hook to allow the tests to bypass the Captcha OR
Making Captcha field as an optional field ONLY in test environment, so that if the Captcha step is skipped, test remains unimpacted
However, Selenium test automation cannot be used to perform Captcha verification. These tests can only be realized by using one of the above practices that helps in bypassing (or disabling) Captcha.
Captcha has always been tricky for testers to automate & is possible under certain scenarios here a video for you to understand it better.
Through this usability testing tutorial, you will learn how usability testing is a great way to discover unexpected bugs, find what is unnecessary or unused before going any further, and have unbiased opinions from an outsider.
8. Barcode / QR Code Verification
We cannot read barcodes or QR codes using Selenium Web Driver. Barcodes and QR codes can be automated, but in reality, do not serve the real-time purpose to automation.
We have to integrate third-party APIs to the existing Selenium test scenarios for realizing automation of barcode/QR code verification. . However, it’s best if the test is performed in a manual manner as it yields higher accuracy.
9. Visual testing
Visual Testing (also referred to as Screenshot testing) is aimed to check the GUI and ensure that it is displayed as intended. You can capture full page screenshots with Selenium as well as capture screenshots of WebElements on the page.
However, visual Testing that primarily includes “Bitmap comparison” is significantly difficult to perform with Selenium Web Driver.
In a nutshell, capturing screenshots is not a hassle in Selenium Automation Testing but performing a ‘visual comparison’ of the images can be a tedious process.
10. 2FA — Two Factor Authentication
While it is possible to automate Two Factor Authentication mechanisms such as OTP validation, push notifications, etc. they are quite tedious and complicated to integrate with Selenium WebDriver. Only a limited part of the test process can be automated using the tool to run automated tests.
You can validate the OTP indirectly by tweaking and looking at the page content displayed to the end-users on the AUT, post submission of the OTP.
However, most organisations in the industry choose not to automate OTP submission & verification flow and prefer following one of the below test steps:
Disabling 2FA for specific test credentials that are used for running automated tests.
Disabling 2FA in Test Environments completely when automation tests are run.
Disabling tests based on test machine IPs (It may not be a good approach in larger organisations where test machines are randomly chosen from a huge list of machines).
11. Performance testing
Performance testing using Selenium Webdriver is not recommended. It is not like you cannot create Selenium test scenarios to run performance tests but Selenium is not optimized for the purpose of performance testing. Hence the entire exercise might not yield favorable results.
Well, WebDriver tests are subjected to many points of external and internal fragility which are beyond your control (e.g. browser startup speed, speed of HTTP servers, the response of third-party servers that host JavaScript or CSS, and the instrumentation penalty of the WebDriver implementation itself).
Instead, you can leverage tools that help you with front-end performance testing that provide accurate results. Tools like Google Lighthouse, LT Browser can help you run performance tests faster and more efficiently.
12. Security Testing
Web application security (or website security) is considered a top-most priority by enterprises since it protects the web app and other online services against different security threats. Hence, most enterprises have a dedicated team of Cyber Security experts who use their own set of tools and technologies to perform security testing.
Again, Selenium is not even counted in one of the tools through which security testing can be performed.Most in-demand tools used by cyber experts are JSky, Owasp Zap, Burp suite, etc.
13. API Testing
API testing is usually performed with tools like Postman, RestAssured, etc. . Selenium WebDriver is usually not used when testing at the API-level. .
It would be best to treat API testing as a separate testing type and run them separately in a different package/suite for achieving better visibility and efficiency.
14. Link spidering / Application Crawling
Conducting App Crawling helps find the broken links easily at no extra cost and time. This helps to find out if the app is behaving weirdly.
Many a time, link spidering is done in much easier steps and it hardly requires much time or effort to write the test scripts for the execution of the same.
While there are several easier ways of conducting link spidering tests in apps, it is suggested not to use Selenium WebDriver to crawl into all the links in the apps. The primary reason is that tasks like finding broken links with Selenium WebDriver are suited when there are less number of links on the page (or website).
Hence, you should choose tools for link spidering based on your requirements.
15. Email Integrations and Social Network Logins
Another case where you should not automate Selenium test scenarios through UI is email and social media logins. Ideally doing this isn’t considered one of the best practices for automation. It is recommended to check the usage guidelines when performing automation else it might lead to serious consequences (including account shutdown).
By integrating the above apps on the UI test layer might lead to test flakiness along with increased test maintenance.
Hence, it is encouraged to use the API layers. By using the login and emailing features, the chances of breaking the API are comparatively lower. The end result is lower test flakiness and faster test execution.
16. Audio / Video streaming
Consuming video content on YouTube and popular OTT platforms have become a norm. However, top-notch user experience can only be dished out if the product is thoroughly tested by the product development and testing teams.
Testing or automating audio/video streaming on a large scale is still considered one of the biggest challenges. There are tweaks of trying to test it remotely by passing the URLs in sites such as StreamTest but the test result may not be very reliable or assuring to count on.
While we are reaching a point to solve this problem via AI with better validation and verification of video quality to the end-users, it is not possible to automate with Selenium test scenarios as of date.
17. Screen Sharing platforms
As remote working became the new normal since 2020, several online meeting platforms such as Zoom, Google Meet, etc. witnessed a surge in demand.
With such tools in use on a large scale, testing and automating user test scenarios becomes much more critical. However, on closer inspection of the web pages of such platforms, we notice that tools like Selenium might not bring much-needed efficiency to the testing approach.
Front end of most meeting apps do not meet the basic automatable criteria and might require tools more sophisticated than Selenium.
As a result, many of these platforms collect feedback from their end-users to optimize the product features as per the feedback.
18. Image Verification
Selenium Webdriver alone cannot be used for comparing expected and actual images. Third-party tools like AShot when integrated with existing Selenium Test Framework can help in achieving that task. However, the integration provides the flexibility to compare images to a minimal extent (i.e. comparison accuracy might not be that great)
Although AShot cannot be used wholeheartedly across all the tests, it could be a boon in testing a few critical scenarios.
Several open-source snapshot testing tools such as Uber’s snapshot test framework for iOS and Facebook’s Screenshot test for Android are popularly used across mobile-first projects.
19. Layout Testing
Selenium can capture effective screenshots and save them in desired locations, as per our convenience across tests. AI-based comparison of the screenshots against expected screenshots is still not a built-in feature of Selenium.
One particular tool which is quite popular among Layout testing tools is Galen Framework. It is widely used in responsive web design testing of websites and apps.
Tools like LT Browser make responsive testing faster and easier by providing 45+ screen resolutions for different mobiles, desktops, and tablets. Using LT Browser you can streamline UI/UX development, test content placement, layout alignments, and assess visually heavy websites on different screens side-by-side. Further, you can generate performance reports of your websites and simulate network conditions to see how your website looks and behaves.
20. File Upload and Verification
We can go ahead and run Selenium test scenarios in the web application where Selenium Webdriver can be used. For example, you can automate to mimic the action of a user clicking on the upload button (which can upload content such as an image, file or a video from the local system).
However, post successful uploading action, we cannot validate using Selenium to verify if the file is successfully uploaded and displayed in the UI. We also need to rely on third-party tools like AutoIT to upload the contents from the local system for the Windows machine.
And post upload, we may have to integrate it with another image verification tool to test that the image is successfully uploaded. This might sound impractical but is doable only if a business wants to automate the flow at any cost. Interestingly, human Intervention cannot be avoided if tests are flaky and unreliable.
Do you know how to upload and download a file in Selenium? Watch this video to learn how to upload and download files in Selenium WebDriver using different techniques.
21. File Download and Verification
While Selenium test scenarios can be used in performing the action to download a file/content from the web to the local system, it cannot validate the ongoing progress of the downloading content in the Downloads efficiently. The only way to verify if the file is downloaded is by comparing the number of items in the downloaded folder (before and after the download process).
The usage of this feature is still limited and needs further improvements to help and conduct effective Selenium Test Automation.
One simpler way is to use the HtmlUnit driver to download any attachments if one can’t avoid skipping it. Also, verifying if the downloaded content matches with the expected contents of the design specifications also requires human intervention to a certain extent.
22. Exploratory Testing
The speciality of Exploratory Testing is completely based on human experience, instincts, and observation while exploring the app as an end-user.
Ideally, Exploratory Testing should not be automated using any Selenium test scenarios or any other automation tool. While exploratory testing is a recommended testing step in any release cycle, we should avoid trying to automate steps from Exploratory testing. It is best to perform exploratory testing using a manual approach under any given situation.
23. Usability Testing and Accessibility Testing
Identifying tools and resources is critical in testing the app’s right UX and functional features from the end-users perspective. However, performing Usability Testing at scale can turn out to be a costly and time-consuming exercise.
It does not matter how bug-free a built app is, and how efficient and fast the app performs if at the end of app release, the end-users find the app difficult to use or find is unusable due to poor UI/UX. Factors like efficiency, reliability, and impressive UI/UX go a long way in building a winning app experience.
Accessibility Testing is a specialised form of testing that makes sure that the Application Under Test (AUT) is fit for use by the differently abled audience. Hence, accessibility testing is treated as one of the forms of usability testing.
This testing type is also not the right tool to be used with Selenium WebDriver to automate Selenium test scenarios. Instead, you should opt for tools like Wave, Taw etc. This is a special form of testing, hence it is recommended to go for tools that test the applications particularly from an accessibility point of view.
24. Libraries and Frameworks Testing
Most libraries and frameworks are functional in nature and require strict functional testing to be performed on them to achieve efficient and reliable test coverage.
As Selenium Webdriver is a UI testing tool for Websites (or web apps), it would not be a good idea to use it for testing the built-in UI-less libraries and frameworks. Hence, you should look for better tools to test libraries and frameworks.
Most teams may not rely on any third-party tools for simplicity. Most teams would like to automate tests and the developed source code itself using the development language and framework itself with minimal usage of testing frameworks such as Junit, Nunit, TestNG etc.
Black box testing? Don’t worry; we will be covering what is Blackbox testing, merits, demerits, types & techniques.
Wrapping it up!
Well these are the 25 scenarios where performing test automation won’t be helpful to you. However there are chances that in some cases it might work, but you can refer to this blog and understand the reasons why it’s advisable to not automate these test scenarios. Perform your free end to end automation testing at a blazing speed!!!
Also, we need to constantly remind ourselves that automation is to help us save time, money and efforts. But one needs to have an understanding of Selenium’s core usage is browser compatibility testing.
Suppose any automated script time or execution time is exceeding the time taken to run manually. In that case, the same may not be the best scenario to automate in the real world, unless the automated work reduces the time in the long run effectively or helps business reach the market fast with quicker & reliable releases. You just cannot use Selenium everywhere!
Just how exploratory testing is impossible, automating everything is also a bad move in many cases. Prioritising what tests to automate first and how quickly to integrate the created automated tests to CI-CD pipelines to add value in the development lifecycle is way more important than automating each and every use case in the project. I hope this article helps you understand why you shouldn’t automate everything using Selenium. Feel free to reach out if you have any questions or feedback.
Happy Testing!
Top comments (0)