DEV Community: Brian Anderson

Top 10 Continuous Testing Tools | 2021 Updated

Brian Anderson — Tue, 27 Jul 2021 08:12:13 +0000

Many companies have favored continuous testing for its role in accelerating productivity while maintaining strong code quality. Nonetheless, choosing the right continuous testing tool is not an easy task. The reason being testers need to evaluate not only the tools’ supported test types and programming languages but also their accessibility and CI/CD ecosystem support.

Therefore, we have chosen the top 10 continuous test tools based on the above requirements. We also considered other factors such as special features that would benefit the stakeholders in conducting continuous testing as well. Without any further ado, let’s begin.

Selenium

Website: http://www.seleniumhq.org/
License: Open-source

The first entry on our list is a classic — Selenium. Built as an open-source framework, Selenium is available on Windows, Mac, and Linux operating systems and can run tests on various browsers(Chrome, Firefox, Safari just to name a few) in addition to web application testing.

The solution, especially when used by capable testers, boasts a diverse range of automated web testing and continuous testing. It also supports parallel testing and its test scripts are transferable across different environments. On the other hand, it can be challenging to use Selenium’s CI/CD tools if you don’t possess great programming skills or just get to know the solution’s functionalities.

Appium

Website: http://appium.io/
License: Open-source

Next on our list is Appium, a mobile application testing tool available for IOS, Android, and cloud service providers.

The tool allows for users to write, upload, execute and view test results directly on the cloud as more and more cloud providers have allowed Appium to function on their infrastructure. Besides, the tool features cross-platform testing capabilities while also allowing users to conduct tests on simulators and emulators.

Much like Selenium, Appium is the best tool for experienced teams to fully integrate the tool into the CI/CD Pipeline. As you will find out in this list, we have listed several options with a more shallow learning curve to accommodate users without deep knowledge of CI tools or their relative framework.

Katalon Studio

Website: https://www.katalon.com/
License: Free & Commercial

Katalon Studio is a free test automation tool for APIs, web applications, desktop apps, and mobile applications. It is available on Windows, macOS, and Linux platforms.

Katalon's complete collection of informative reports, real-time monitoring, and an instant feedback mechanism will detect any emergent issues in the CI/CD pipeline and DevOps support. The solution is compatible with popular continuous integration platforms such as Jenkins, Bamboo, Azure, CircleCI, and Jira Agile. As a result, Katalon facilitates team performance.

For users that struggle to adopt Selenium or Appium for their continuous testing purposes, Katalon also offers out-of-the-box continuous testing components, all without having to write test scripts or configure from scratch. Additionally, Katalon Studio also features sophisticated extensions for professionals. The plugins available in the Katalon Store can assist you in extending automation capabilities and simply integrating with other CI/CD applications. Moreover, they provide their free resource to learn Automation Testing in Katalon Academy

Watir

Website: http://watir.com/
License: Open-source

Watir is a web application test solution available on Windows that supports cross-browser testing with Chrome, Firefox, and Opera.

While Watir is limited to only one OS and can only run simplistic tests, it enables data-driven testing as well as headless browser testing, both of which are required for CI/CD pipeline testing. Watir is also compatible with Behavioral Driven Development (BDD) tools and frameworks such as Cucumber, RSpect, and Test/Unit.

As such, your team can still achieve effective continuous testing throughout the development lifecycle by combining Watir with BDD frameworks and connecting with CI technologies (e.g. Jenkins).

Testsigma

Website: https://testsigma.com/
License: Commercial

Enterprises with limited testing resources or complex testing requirements may rely on the solution's continuous testing infrastructure for devices, operating systems, and browsers. Moreover, Testsigma is a cloud-based software that composes tests in English, which helps manual testers learn automation by flattening the learning curve. The solution is compatible with web, mobile, and API platforms.

For continuous testing adoption, Testsigma supports and integrates popular CI/CD tools such as Jira, Slack, and Jenkins. More notably, Testsigma uses AI to assist in shift-left testing and test case maintenance to highlight possible dangers from continuous change, allowing users to take quick action.

Eggplant

Website: https://eggplant.io/
License: Commercial

Besides Katalon Studio, Eggplant is another easy-to-adopt testing solution. It features a special test automation approach — Applications Under Test (AUTs) which display images for ease of viewing in place of raw test scripts.

Moreover, Eggplant Eggplant a test lab, a cloud device management solution functions round the clock for continuous testing and deployment. Furthermore, Eggplant enables connection with the CI/CD ecosystem such as Jenkins, Bamboo, CA Agile Requirements Designers, and others. Finally, Eggplant also boasts a record and playback tool — Undo, that can help testers notice and respond to defects more efficiently.

Unified Functional Tester

Website: https://software.microfocus.com/software/uft
License: Commercial

Micro Focus Unified Functional Tester (UFT) is a forerunner that appeared at the pinnacle of test automation trends.

UFT offers automated and continuous testing for online, mobile, API, hybrid, Robotics Process Automation (RPA), and enterprise applications. It has image-based identification driven by built-in OCR (ABBYY, Google Tesseract) that mimics how people recognize items.

The combination of UFT ecosystem components such as source control systems (Git, SVN), and CI/CD technologies facilitates continuous testing (e.g. Jenkins, Bamboo, Microsoft TFS).
UFT's programming language is VBScript, which may be difficult for manual testers. As a result, before choosing a tool, consider analyzing its existing scripting capabilities.

Rational Functional Tester

Website: https://www.ibm.com/marketplace/rational-functional-tester
License: Commercial

Rational Functional Tester (RFT) offers functional, data-driven, API, performance, and regression testing. RFT includes many cooperation modes, including storyboard testing for visualizing testing methods and scripting for more complicated projects.

The program also allows for continuous testing and the execution of tests in the CI/CD pipeline. However, because the tool was released a long time ago, it comes with out-of-date technology.

As a result, the setup procedure might be difficult, especially for inexperienced users, because the tool demands significant technical understanding in DevOps. If properly incorporated into the CI/CD pipeline, nonetheless, the remainder of the operation may be rapidly completed.

TestComplete

Website: https://smartbear.com/product/testcomplete/overview/
License: Commercial

TestComplete is a test automation solution for desktop, online, and mobile apps. Python, Javascript, and VBScript are among the scripting languages supported by the program. TestComplete can run keyword and data-driven tests. It can also use AI capabilities for dynamic test item selection and management. If the AUT's user interface changes, TestComplete may identify and update the tests automatically.

For API testing, the program enables third-party unit test frameworks namely TestNG, Selenium Webdriver, and SOAP UI. All of them will help you improve your test coverage.
With plugins that connect with standard CI/CD systems such as Jenkins, GIT, and Zephyr, TestComplete offers open integration with the CI/CD ecosystem so you may also create new plugins to work with the current system.

Tosca

Website: https://www.tricentis.com/
License: Commercial

For business stakeholders that want to approach the testing process, Tosca is the solution for you. Tosca enables enterprise applications, online, mobile, APIs, and business intelligence testing. The solution may also provide certain important test cases for the team to swiftly identify and focus down the application's most problematic regions.

Tosca provides a platform that connects with DevOps and Application Lifecycle Management (ALM) systems as a continuous and DevOps testing solution. Tosca provides virtual services to assist its users in simulating the test environments that would be encountered in production, allowing the team to enhance test coverage and assure product quality.

Conclusion

With the right tools, achieving continuous testing will no doubt ensure your company’s quality product rollout at speed. Nonetheless, the right tool in question may vary based not only on their features and CI/CD system support but a myriad of other factors such as your teams’ technical know-how and dynamics.

This is the case for the entries that we have selected. One might boast special functions that require specific CI knowledge to properly use, while others that are easy to instantly employ might be limited in scope. At the same time, this list is not exhaustive. As a result, it is up to you to choose and settle for one that best suits your and your team’s needs.

A Look at Selenium and Its Alternative Solution for 2021 and Beyond

Brian Anderson — Mon, 19 Jul 2021 08:15:51 +0000

Selenium was initially published in the mid-2000s as a Web browser plugin for recording and replaying Web browser interactions. Since then, it has undoubtedly become the most used test automation tool. According to a study of the automated testing community, the framework is utilized by more than 80% of those polled.

Selenium is an open-source framework for Web-based application automation, including Selenium WebDriver, Selenium IDE, and Selenium Grid.
Selenium IDE is a browser plugin that captures and replays user interactions.
Selenium Grid enables WebDriver to simultaneously execute automation tests on many computers, browsers, and OS systems.

Selenium is frequently utilized in the software automation sector due to the benefits it gives. First, it is developer-friendly, enabling developers to build and test any programming language of their choosing. Second, the framework is a cross-browser and cross-operating system, allowing teams to perform test automation across several browsers and operating systems.

The Structure of Selenium WebDriver

The Selenium WebDriver architecture is made up of three parts: the WebDriver, the Browser Specific Drivers or Drivers, and the Browser. These components communicate bidirectionally or two-ways, issuing HTTP requests and receiving HTTP replies in RESTful services.

WebDriver includes language-specific client libraries, often known as language bindings, for creating automation scripts and executing script commands in the Browser using Drivers. Selenium can now enable scripting in various languages, including Java, C#, Python, Ruby, and JavaScript, thanks to these language bindings.
Browser Specific Drivers, also known as Drivers, are responsible for running the actual Browser via delivering WebDriver requests. Drivers can be implemented and extended by third parties. Two browser driver implementations are GeckoDriver for Firefox and ChromeDriver for Chrome.
Browser component accepts requests from the Drivers component to drive interactions on the application under test (AUT). Selenium is compatible with all major browsers, including Chrome, Firefox, Internet Explorer, Opera, and Safari.

WebDriver can operate on a separate system than the Drivers and Browser components. In this configuration, WebDriver connects with these components remotely through Selenium Server or Remote WebDriver. Selenium can now perform automated tests on many browsers and computers seamlessly at the same time.

Benefits of this structure

The architecture is disjointed. Three components connect using the HTTP protocol and RESTful Web services as the mode of communication. This property enables an element to function independently of the others. Changes and improvements to one component may be performed quickly and without impacting the others. Selenium may be platform- and language-neutral thanks to its loosely linked design.
Moreover, the architecture is accessible. Selenium WebDriver is a W3C Recommendation, which allows for many contributions from the software community.
These architectural characteristics result in a spate of significant Selenium characteristics:

Cross-platform and cross-language compatibility. Because several implementations of the Drivers component are available, users may run automation scripts on various operating systems and browsers. As long as language bindings in the WebDriver are accessible, Selenium automation scripts may be developed in various languages.
Developer friendly. Because Selenium supports various languages, developers may build automation scripts in their preferred language. They may also create and incorporate a binding library in the WebDriver component to help a new scripting language if necessary.
Open for extension. The Selenium architecture allows for language binding expansions by supporting additional test scripting languages or Drivers for new browsers.
Open for integration with a test automation framework. A new test automation framework may be constructed on top of Selenium by utilizing it as an engine for executing scripts for Web automation, as demonstrated in the figure below.

The Demand for a Selenium Alternative Solution

When viewed through the eyes of individuals who do not possess programming or sophisticated technical abilities, Selenium's advantages in assisting programming are really its drawbacks. Many testers see Selenium as a barrier to getting started with test automation due to complex programming. As a result, using Selenium directly presents a high learning curve for many testers.

Selenium also lacks means for separating automation scripts from test data, resulting in high test script maintenance costs and difficulties with test script and test data reusability. There is also a deficiency in managing and organizing test materials such as test scripts, data, and test objects.

These restrictions help to explain the rapid rise of Selenium-based products in the software test automation industry in recent years. Teams and companies want a solution that provides out-of-the-box testing capabilities without requiring them to develop their own Selenium-based framework to start their automation process.

The Structure of Selenium-based Test Frameworks

On top of Selenium, a variety of test automation frameworks have been developed. These frameworks attempt to solve several of Selenium's shortcomings, such as the absence of modules for reporting, logging, monitoring, and reusing test automation assets. Importantly, these frameworks aid in overcoming the challenge of developing and launching an automation effort with Selenium.

The diagram below presents various components of a typical test framework that uses Selenium as its test execution engine.

Many current automation solutions use this Selenium-based framework. Some examples are Katalon Studio, CrossBrowserTesting, QMetry Automation Studio, Lambda Test, and Sauce Labs. Many of these solutions go a step further by incorporating Appium as a test automation engine for mobile apps, allowing them to handle both web and mobile automation in a single tool.

Resource: A Deep Dive into Selenium, Its Alternative Solution for 2021 and Beyond
Top 3 Selenium IDE Alternative
Also Katalon are introduce how to migrate from Selenium

Are Developers as Productive as Possible When It Comes to Testing?

Brian Anderson — Tue, 26 Jan 2021 10:13:38 +0000

In this blog post, I describe the current testing trend among software developers from various data sources. Evidence suggests that a significant proportion of developers may not be as productive as possible when testing features/bug fixes, becoming increasingly important due to the shift-left movement. With that said, I also found a solution called Katalon Recorder that has made testing before handing it to QA easier. Additionally, it can help you transition into software developers in test much much more effortless.

But first, what do I mean by testing?

Developers must test their features/bug fixes on a functional level to ensure that they do not break anything obvious before handing it off to QA. Of course, there are software developers in test that specialize in building software to test software. Such testing activity happens continuously as modifications are made to the codebase because any change can potentially affect the software’s behaviors.

For example, one must set up the precondition to test a change, such as logging in the application, navigating to a particular site where the change affects. One must also test the flow affected by the change is as expected, and repeat the entire process for the next change.

When I talk about testing in this post, I mean functional testing and not unit testing. Functional testing is essential because it directly affects the user experience.

Developers are becoming more and more responsible for testing their code

According to the State of Frontend 2020 Report [1], more than 70% of frontend developers state that they are responsible for testing in their software development team, either independently or together with QA specialists. Some of them are also software developers in test who build frameworks to test software. This epitomizes the shift-left testing movement that has emerged within the past few years.

Shift-left testing is an approach where testing is performed earlier in the life-cycle. This is a shift to the left in the project timeline, hence arose the name. The rationale behind this approach is the fact that problems detected downstream are much more costly to fix, compared to those found upstream.

Developers would have already moved on to other tasks by the time a problem is discovered by QA. They will have to switch contexts to fix the problem. Research showed that context switching means high cognitive cost and low performance [2]. On the project level, this increases the risk of not meeting commitments made to stakeholders. Recognizing such downsides, teams have started adopting the shift-left approach described above. Consequently, developers become more and more responsible for testing their own code before handing it over to testers.

A significant portion of developers are still manual testing their code

Also, looking at the State of Frontend 2020 report [1], the importance of end-to-end testing is demonstrated, with more than 40% of developers performing UI tests themselves. This is not surprising, considering the importance of ensuring user experience to business success.

What kinds of tests have you performed yourself

However, a different source of data gives us an interesting twist. The Front-End Tooling Survey [3], conducted in 2019, revealed that almost 36% of developers currently do not use any tool to test their applications.

Most popular front end developers’ tools

Manual testing goes against what makes developers great at their job

The primary problem with testing manually for developers is cognitive load. Unlike QA members who specialize in such activity, developers are not trained in that domain. Of course, the exception is software developers in test. This unfamiliarity increases cognitive load, especially over the long run. Even worse, developers have to test manually again and again as they code up the feature. Doing manual testing repeatedly reduces the cognitive capacity available for software development activity. As a result, developers are not as productive as they could be.

Another explanation to his problem is the inherent bias in developers that make them good at their job: the tendency to think in terms of “How can I build it ?”. Testers are trained to have just the opposite bias: “How can I break it ?” which makes testing come more naturally for them. This doesn’t mean that developers should not test software. This simply means that testing doesn’t play well with what developers are trained for, and it further increases the cognitive load whenever they need to do it.

Manual testing goes against what makes developers great at their job

The takeaway here is that manual testing is not ideal for developers. If you are looking to be more productive, chances are you need to reduce the amount of effort involved in testing while also not compromising its quality.

Is there a better way?

I am not a software developer in test, but in my years of doing software development and testing, I realized that the force of habit is genuine. It may prevent us from even thinking of the possibility that things could be better. I have recently come across a free web extension called Katalon Recorder, which is primarily used to generate automated tests. Still, its record-playback functionality can also be used to replace my manual testing activities.

Katalon Recorder — a free extension to automate browsers

After installing the extension, you just need to open it and click on the Record button to start generating tests. Click on the Play button to execute tests. It literally could not be any easier.

I like Katalon Recorder, much more than Selenium IDE , is the lack of friction that Katalon Recorder provides. With Selenium IDE, you need to create a project, input URL, and only then you can start generating tests. Undoubtedly, the onboarding process may make sense for a different audience. As a developer, I am only concerned with creating cheap and fast tests to replace the actions that I would have to do myself otherwise. The frictionless feeling is crucial for my long-term adoption of Katalon Recorder because god knows I have to do it a lot.

Since my discovery of Katalon Recorder, I have become much more productive due to the amount of repetitive testing it saved me from. I believe that we can be much more productive if we are mindful of the activity that we engage in every day because only then will we consider how these activities can be improved. Better yet, there may already be a tool out there that can help us immediately.

Additionally, I think that Katalon Recorder is an excellent tool if you are looking to transition into software developers in test because it allows you to experience automated testing quickly and easily. Feel free to share the tools that help boost your productivity every day in the comment section below.

References

[1] State of Frontend 2020 Report

[2] Shakeri, Zahra & Noaeen, Mohammad & Zowghi, Didar & Far, Behrouz & Barker, Ken. (2018). Two Sides of the Same Coin: Software Developers’ Perceptions of Task Switching and Task Interruption. EASE’18: Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018. 175–180. 10.1145/3210459.3214170.

[3] The Front-End Tooling Survey

Top 15 Mobile Testing Tools for 2021 (Latest Update)

Brian Anderson — Tue, 28 Jul 2020 08:19:34 +0000

Best Mobile Testing Tools for 2020

With approximately 3.5 billion smartphones users and 1.35 billion tablet users, it’s no surprise that the mobile application industry is thriving with 204 billion apps downloaded in 2019, and projected to increase by 25% between 2020 and 2022.

Such volumes of demands, in addition to the constant influx of software sophistication and operating systems, give way for automated mobile testing to prosper in popularity. Automated mobile testing tools offers a win-win solution for all parties involved:

For developers, it allows the reproduction of potential issues on various devices and OS in real-time, then viewing logs, data, recording for immediate bugging and adjustment, thus improving test cases and efficiency.
For businesses, mobile app testing tools reduce costs and time-to-market, increasing profit and competitive edges.
For users, it increases quality, thus ensuring a smooth experience.

From there, we can see the importance of choosing the right automated mobile app testing tools for developers and companies. Amid such rising appeal for automation, there are hundreds of mobile testing tools for teams to choose from since technology companies are vying for market shares.

However, such availability of mobile testing tools can confuse organizations when looking for the best fit for their mobile apps. To give testing teams some clarity, I’m using my humble personal experiences in software testing to create this list of 15 best automated mobile testing tools in the industry. Teams can use this list to get some good understanding of all the available software, and compare each software’s distinct features; therefore, determine which tool best caters to their apps.

Top 5 Mobile Testing Tools for 2021

1. Appium

This is a no-brainer when it comes to choosing a testing tool for mobile application projects. As one of the most-searched mobile app testing tools, Appium has long been the leading software in the mobile testing industry.

Highlighted features:

Appium is a free, open-source mobile testing tool that supports a wide range of languages, test frameworks, and OS.
It offers various integrations with third-party tools, enabling cross-platform testing using the same API, and tests reusability between iOS, Android, and Windows.
Support all kinds of apps with no modification required, making it among the most robust and flexible mobile testing tools.

In conjunction with its advanced features, Appium requires extensive coding experience to create tests, which can be problematic when users require training before use. There’s no official support but rather a community of users sharing information.

Website: http://appium.io/

Price: Open-source

2. Katalon Studio

There’s a reason why over 65,000 companies worldwide adopted Katalon Studio as their testing solution. Utilizing the robust Appium and Selenium frameworks as its foundation, Katalon Studio is an all-in-one testing solution for companies and teams at any scale. It offers codeless testing for all testing needs (Mobile testing, Web testing, Desktop application testing, API testing) without the steep learning curves of Appium and Selenium. For mobile testing specifically, Katalon supports iOS and Android testing platforms and extends other OSs like Windows and Linux.

Highlighted features:

Supports most testing capabilities offered by Appium, but easier: simple setup; effortless test script creation using keywords, images, objects; test execution locally and remotely on real devices, simulators or custom cloud-based devices, and flexible test reusability across mobile platforms, API, and Web.
Offers powerful utilities to accelerate the automation process: Mobile Object Spy to quickly populate the project’s object repository and Mobile Recorder to generate automation scripts from recorded actions with built-in modifications and actions when recording tests.
Integration with all the commonly used third-party tools (Jira, Git, Jenkins, etc.) and mobile platforms (Sauces Lab, Lambda Test, etc. )
Comprehensive documents, tutorials and official support from both the Katalon teams and community.

Website: https://www.katalon.com/

Price: Free — $839 per year

3. Eggplant

Formally TestPlant, Eggplant is an AI-powered software test automation for desktop, web, and mobile apps. For mobile testing projects, Eggplant supports all standard platforms: Android, iOS, BlackBerry, Windows Phone, and Symbian. Although the community for Eggplant isn’t as prominent as other mobile testing tools, its recent merge with Keysight Technology hints a promising expansion may be on the horizon for Eggplant.

Highlighted features:

Test cases can run from the command line without supervision.
AI-powered tools to ensure the continuity of tests while you’re doing remoted works on Cloud.
Eggplant’s Digital Automation Intelligence platform can test any technology on any device, operating system, or browser at any layer, from the user interface (UI) to application programming interfaces (APIs) to the database.

Website: https://www.eggplantsoftware.com/

Price: Commercial — Contact Sales

4. TestComplete

TestComplete is an AI-integrated automated testing tool for UI tests in web, desktop, and mobile applications. Aside from some difficulties in updating test cases, this tool is flexible and easy-to-use with the options of both script and scriptless test generation.

Highlighted features:

Supports multiple programming languages (Python, JScript, VBScript, or JavaScript), multiple mobile platforms (iOS, Android), and third-party tools (Jenkins, GIT, etc. ).
For mobile app testing, TestComplete allows testing on both native and hybrid mobile apps on emulators, real devices or virtual machines.
User-friendly UI with supportive documents and customer services.

Website: https://smartbear.com/product/testcomplete/overview/

Price: From $6095 per year

5. Ranorex

Striving to be an all-in-one testing solution, Ranorex promises to deliver high-quality desktop, mobile, and web applications testing. It’s a natural choice for beginners with a codeless click-and-go interface, helpful wizards, and powerful automation experts with a full IDE.

Highlighted features:

Support iOS and Android testing, including native mobile apps and mobile web apps with detailed summary reports.
Run cross-device tests in parallel or use Appium WebDriver for mobile web tests.
Support limited languages but does offer powerful integration.

Website: https://www.ranorex.com/

Price: Perpetual licenses, from €2,890

6. Kobiton

Another new player in the field, Kobiton markets itself as a mobile experience platform. From building device labs, instant health-check to run manual and automated tests on native, web and hybrid Android/iOS apps with real devices, Kobiton strives to deliver seamless access to testing regardless of testers’ needs.

Highlighted features:

Built on top of the Appium open-source framework, Kobiton opens for integration with all frameworks (Appium, XCUI, Espresso, etc.)
Pioneering Intelligent Test Automation and Appium Anywhere technology allows for a codeless mobile testing experience based on one manual test script and a seamless text script adaptation across devices.
Providing the latest real, cloud-based devices, centralized testing history and data logs to support extensive collaboration between teams.

Website: https://kobiton.com/

Price: Free trial — $2004 per year

7. Monkey Talk

Monkey Talk, another open-source mobile testing tool, is a user-friendly tool that doesn’t require jailbreaking. It automates functional, real interactive tests for both Android and iOS — catering from basic smoke tests to data-driven test suites.

Highlighted features:

Testers can generate XML and HTML reports with this mobile testing tool.
Automatically take screenshots when error or failure happens.
For continuous integration needs, Monkey Talk also has inbuilt support for Jenkins and Hudson.

Website: https://github.com/alexnauda/monkeytalk

Price: Open-source

8. Robotium

Created as an extension of the Android framework, Robotium is an open-source Android-only mobile testing tool that supplies robust automatic black-box test cases for Android applications.

Highlighted features:

Unlike many other Android testing tools, Robotium also allows the automation of multiple Android activities and creates solid test cases in minimal time.
Supports Java as the main programming language.
Robust integration with Maven and Ant.
An extensive library of unit tests.

Website: https://github.com/RobotiumTech/robotium

Price: Open-source

9. iOS Driver

Another prominent open-source tool on this list, iOS Driver, is a Selenium-integrated mobile testing tool for iOS native, hybrid and mobile applications using Selenium/WebDriver API. This tool runs efficiently on an emulator rather than devices, some of its recent versions run on devices but are comparatively slower than on the emulator. As it’s built upon two different frameworks from Apple, testers should check the system requirements for compatibility.

Highlighted features:

Once compatible, testers can utilize some features of the Apple-developed X-code.
iOS Driver makes it easy for testers as they don’t need to jailbreak, change any app code or load any additional app for testing the app on devices.
iOS Driver can be used as a Selenium Grid Node; thus tests can be run parallel on the same architecture as the web, decreasing the feedback time for GUI test.

Website: https://ios-driver.github.io/ios-driver/

Price: Open-source

10. Test Io

Test IO is a leading SaaS platform for software crowd testing: the continuous testing of web and mobile applications by skilled human testers using real devices. To test your software, you need to upload your mobile app or share the website URL with test IO, then the team at test IO will securely distribute across the interested testers all over the globe. You will get the initial test results within one hour after starting the test run.

Highlighted Features:

From waterfall to continuous delivery, test IO tests when you need to: from prototype to UAT, regression, and even production.
Integration with Jira, Github, Asana, etc.
Remove QA bottleneck with on-demand, flexible testing that scales up to your needs.

Website: https://test.io/

Price: Commercial — Contact Sales

11. Selendroid

Selendroid is among the leading testing tools for Android-based hybrid and native mobile apps. It addresses most setbacks presented by other Android mobile testing apps with its powerful integration of WebDriver API and Selenium System.

Highlighted features:

An open-source framework for Android that can simultaneously interact with multiple devices, emulators, and simulators.
Powerful recording features, UI testing, and flexibility with hardware devices.
Testers are not required to alter their AUT for automation.

Website: http://selendroid.io/

Price: Free

12. Perfecto

Recognized by Forrester Wave as a 2020 strong performer, Perfecto is a cloud-based testing software that supports mobile and web testing. It delivers a hassle-free testing experience, made possible by the very extensive platforms and testing devices through which teams can perform tests on. Although the speed on mobile testing is reported to be relatively slower than web, it boosts a reputation for its collaborative capabilities with screen-sharing options for teams while testing.

Highlighted features:

Support real user simulators and large-scale remote testing.
Extensive test failure analysis with dashboard and detailed artifacts (images, videos, crash logs, vitals, etc).
Wide availability of framework and testing platforms integrations.

Website: https://www.perfecto.io

Price: From $3,600 per year

13. Testdroid

Testdroid is a cloud-based software for mobile app testing that helps developers save development costs, eliminate the unpredictable operational cost, and improve time-to-market. Now Testdroid is under development at SmartBear, a leading provider of automation solutions.

Highlighted features:

One of the best platforms to test your iOS and Android devices with different screen resolutions, OS versions, and HW platforms.
Limit the risks of testing with real devices and agile practice.
Allows remote access to more than 300 genuine gadgets running in Android.

Website: https://bitbar.com/

Price: From $1,908 per year

14. Calabash

Developed by Xamarin, Calabash is an open-sourced mobile app testing tool for iOS and Android. Although it lacks a recording function, it makes up for being one of the few mobile testing tools that support Cucumber, the foundation that enables writing codes in English.

Highlighted features:

Using Xamarin Test Cloud, testers can run test cases on more than 1000 devices with real-time feedback and validation across many different form factors, OS versions, OEM customizations, and hardware specs.
Aside from essential integration with CI/CD tool integrations, Calabash also has separate libraries for iOS and Android that allows testers to interact with native and hybrid apps programmatically.
Supports Ruby, Java, and C# as programming languages.

Website: https://github.com/calabash/calabash-android

Price: Open-source

15. 21lab

21 is a self-learning test building and analytics platform for iOS and Android apps. Incorporating major tech innovations like AI, 21 promises an almost full implementation of automation into your testing plan.

Highlighted features:

Fast and intelligent authoring for users to create automated functional and UI tests in minutes.
Automating test maintenance to ensure test reliability across all testing platforms, reduce flaky tests.
Fully SaaS, requires no installation or devices to create or execute tests.
Price: Free demo, contact Sales for more information.

Website: http://21labs.io

Price: Commercial — Contact Sales

Conclusion

By combining the two booming industries of mobile apps and automation testing, there is no doubt that the automated mobile testing industry has a bright future ahead. These 15 tools represent the most innovative, the most flexible products in the industry. With such collections of powerful distinctions and features, it’s no doubt that a few of these apps would be great candidates for your projects and companies.

With that being said, the definition of the best mobile testing tool depends on your product and team. Your testing tool choice should be a software that fits into your team, not software for your team to fit in. Remember to consider your budget, expertise, and needs, then combine with the potential development trends of your app and the mobile application industry to optimize long-term financial gain from the chosen mobile testing tool.

References:

Top Software Test Automation Software of 2020 — Gartner Peer Insights
Best Automation Testing Tools for 2021 (Top 10 reviews)
Top 10 API Testing Tools (Details & Updates Done for You!)
The original article is previously published on Medium.

Top 15 Mobile Testing Tools for 2021 (Latest Update)

Brian Anderson — Tue, 28 Jul 2020 08:19:34 +0000

Top 15 Mobile Testing Tools for 2021 (Latest Update)

Best Mobile Testing Tools for 2021

With approximately 3.8 billion smartphone users and 1.35 billion tablet users, it’s no surprise that the mobile application industry is thriving with 244.2 billion apps downloaded in 2020, and predicted to reach 258 billion app downloads globally in 2022.

For developers, it allows the reproduction of potential issues on various devices and OS in real-time, then viewing logs, data, recording for immediate bugging and adjustment, thus improving test cases and efficiency.
For businesses, mobile app testing tools reduce costs and time-to-market, increasing profit and competitive edges.
For users, it increases quality, thus ensuring a smooth experience.

Top 5 Mobile Testing Tools for 2020

1. Appium

Highlighted features:

Appium is a free, open-source mobile testing tool that supports a wide range of languages, test frameworks, and OS.
It offers various integrations with third-party tools, enabling cross-platform testing using the same API, and tests reusability between iOS, Android, and Windows.
Support all kinds of apps, making it among the most robust and flexible mobile testing tools.

Website: http://appium.io/

Price: Open-source

2. Katalon Studio

Highlighted features:

Supports most testing capabilities offered by Appium, but easier: simple setup; effortless test script creation using keywords, images, objects; test execution locally and remotely on real devices, simulators or custom cloud-based devices, and flexible test reusability across mobile platforms, API, and Web.
Offers powerful utilities to accelerate the automation process: Mobile Object Spy to quickly populate the project’s object repository and Mobile Recorder to generate automation scripts from recorded actions with built-in modifications and actions when recording tests.
Integration with all the commonly used third-party tools (Jira, Git, Jenkins, etc.) and mobile platforms (Sauces Lab, Lambda Test, etc. )
In-depth reports with AI-based analysis on Katalon TestOps — a QA orchestration platform.
Comprehensive documents, tutorials and official support from both the Katalon teams and community.

Website: https://www.katalon.com/

Price: Free — $759 per year

3. Eggplant

Highlighted features:

Test cases can run from the command line without supervision.
AI-powered tools to ensure the continuity of tests while you’re doing remoted works on Cloud.
Provides various modes of script creation workflows via SenseTalk.
Eggplant’s Digital Automation Intelligence platform can test any technology on any device, operating system, or browser at any layer, from the user interface (UI) to application programming interfaces (APIs) to the database.

Website: https://www.eggplantsoftware.com/

Price: Commercial — Contact Sales

4. TestComplete

Highlighted features:

Supports multiple programming languages (Python, JScript, VBScript, or JavaScript), multiple mobile platforms (iOS, Android), and third-party tools (Jenkins, GIT, etc. ).
For mobile app testing, TestComplete allows testing on both native and hybrid mobile apps on emulators, real devices or virtual machines.
User-friendly UI with supportive documents and customer services.
Use artificial intelligence to run the object recognition engine to speed up the creating and maintaining process.
View quality report and test analysis.

Website: https://smartbear.com/product/testcomplete/overview/

Price: From $6,095 to $9,479 per year

5. Ranorex

Highlighted features:

Support iOS and Android testing, including native mobile apps and mobile web apps with detailed summary reports.
Run cross-device tests in parallel or use Appium WebDriver for mobile web tests.
Support limited languages but does offer powerful integration.

Website: https://www.ranorex.com/

Price: From $3,590 — to $5,990 per year.

6. Kobiton

Highlighted features:

Built on top of the Appium open-source framework, Kobiton opens for integration with all frameworks (Appium, XCUI, Espresso, etc.)
Pioneering Intelligent Test Automation and Appium Anywhere technology allows for a codeless mobile testing experience based on one manual test script and a seamless text script adaptation across devices.
Providing the latest real, cloud-based devices, centralized testing history and data logs to support extensive collaboration between teams.

Website: https://kobiton.com/

Price: Free trial — $2004 per year

7. Monkey Talk

Highlighted features:

Testers can generate XML and HTML reports with this mobile testing tool.
Automatically take screenshots when error or failure happens.
For continuous integration needs, Monkey Talk also has inbuilt support for Jenkins and Hudson.

Website: https://github.com/alexnauda/monkeytalk

Price: Open-source

8. Robotium

Created as an extension of the Android framework, Robotium is an open-source Android-only mobile testing tool that supplies robust automatic black-box test cases for Android applications.

Highlighted features:

Unlike many other Android testing tools, Robotium also allows the automation of multiple Android activities and creates solid test cases in minimal time.
Supports Java as the main programming language.
Robust integration with Maven, Gradle and Ant.
An extensive library of unit tests.

Website: https://github.com/RobotiumTech/robotium

Price: Open-source

9. iOS Driver

Highlighted features:

Once compatible, testers can utilize some features of the Apple-developed X-code.
iOS Driver makes it easy for testers as they don’t need to jailbreak, change any app code or load any additional app for testing the app on devices.
iOS Driver can be used as a Selenium Grid Node; thus tests can be run parallel on the same architecture as the web, decreasing the feedback time for GUI test.

Website: https://ios-driver.github.io/ios-driver/

Price: Open-source

10. Test Io

Highlighted Features:

From waterfall to continuous delivery, test IO tests when you need to: from prototype to UAT, regression, and even production.
Integration with Jira, Github, Asana, etc.
Remove QA bottleneck with on-demand, flexible testing that scales up to your needs.
Allow viewing detailed bug reports with screenshots and videos
Supports real mobile devices, browser, OS version, IoT and wearable devices (Smartwatch, phablet, set-top box, or a smart TV)

Website: https://test.io/

Price: Commercial — Contact Sales

11. Selendroid

Highlighted features:

An open-source framework for Android that can simultaneously interact with multiple devices, emulators, and simulators.
Powerful recording features, UI testing, and flexibility with hardware devices.
Testers are not required to alter their AUT for automation.
Can be used on emulators and real devices or integrated as a node into Selenium Grid

Website: http://selendroid.io/

Price: Free

12. Perfecto

Highlighted features:

Support real user simulators and large-scale remote testing.
Extensive test failure analysis with dashboard and detailed artifacts (images, videos, crash logs, vitals, etc).
Wide availability of framework and testing platforms integrations.

Website: https://www.perfecto.io

Price: From $3,600 per year

13. Testdroid

Highlighted features:

One of the best platforms to test your iOS and Android devices with different screen resolutions, OS versions, and HW platforms.
Limit the risks of testing with real devices and agile practice.
Allows remote access to more than 300 genuine gadgets running in Android.

Website: https://bitbar.com/

Price: From $1,908 per year

14. Calabash

Highlighted features:

Using Xamarin Test Cloud, testers can run test cases on more than 1000 devices with real-time feedback and validation across many different form factors, OS versions, OEM customizations, and hardware specs.
Aside from essential integration with CI/CD tool integrations, Calabash also has separate libraries for iOS and Android that allows testers to interact with native and hybrid apps programmatically.
Supports Ruby, Java, and C# as programming languages.

Website: https://github.com/calabash/calabash-android

Price: Open-source

15. 21lab

Highlighted features:

Fast and intelligent authoring for users to create automated functional and UI tests in minutes.
Automating test maintenance to ensure test reliability across all testing platforms, reduce flaky tests.
Fully SaaS, requires no installation or devices to create or execute tests.
Price: Free demo, contact Sales for more information.

Website: http://21labs.io

Price: Commercial — Contact Sales

Conclusion

References:

Top Software Test Automation Software of 2020 — Gartner Peer Insights
Best Automation Testing Tools for 2020 (Top 10 reviews)
Top 10 API Testing Tools (Details & Updates Done for You!)

How I knockdown "Object Locator Flakiness" in automation testing.

Brian Anderson — Wed, 22 Jul 2020 04:21:29 +0000

As a Tester I have to deal with increasing new page objects and changing identifiers when the web application grows in Web UI testing. No matter if you are a manual or an automation tester, updating object locators for the Application Under Test (AUT) is a required step when changes are made. This helps ensure regression testing flows without friction, and all key functionalities are not affected.

Self-healing is a common solution provided by automated testing tools made to solve the challenge of constantly changing the UI of web applications. Self-healing capability offers a host of benefits for testers and their team: it takes less time and effort to ensure all the functional tests run smoothly and avoids interruption for all executions.

Self-healing Automation's definition and Its benefits

Self-healing is a mechanism made to solve the problem of identifying objects as the product evolves, and necessary changes are made. Offered by most automation frameworks in the market, Self-healing is automation for automation. Testing tools with self-healing can detect changes in the properties and automatically update with new attributes to ensure the designed test cases are functional.

In web testing, one common challenge is to deal with changes in the application under test as the product takes on another shape. Web UI testing can be considered an integral part of any web application product. It is what the users first experience, and it also reflects the quality of production for businesses. However, when the UI changes, test objects’ properties also change, causing the tests to fail later on.

When test scripts are easily affected by the rapid release and update of the product and its development team, test case maintenance becomes inevitable for testers. For both manual and automation testers, having to go back to old scripts and update new properties to each object is a daunting task, let alone the fact that Agile development is now the new norm.

The self-healing mechanism

When designing test scripts for Web UI testing, both low-code and code-driven practices, objects of the AUT are bound to a locator to run the test steps successfully. The AUT will not stay the same over time, and each tweak or change made to the UI will be mismatched with the predesigned test cases.

Testers usually expect their test cases to fail as updates are constantly added unless they have had plans for maintaining the test scripts despite future changes — by employing a self-healing mechanism.

Automation frameworks build the self-healing capability as a “smart” feature that makes testing easier for teams. To avoid failed test cases, the test automation tool will apply the new object’s locators and ensure the entire script stays up-to-date with each change made to the product’s design.

Why Self-healing is essential and its benefits to your team

Self-healing is the key to build an efficient maintenance process for automated testing. Without a doubt, with a well-planned out maintaining approach, testers free themselves from a heap of repeating tasks and never-ending edits in the test scripts.

Less failed test cases

Your tests might fail; that’s no news to any tester, however, sometimes the root cause of failures is simply that the UI changed and the test did not. With self-healing activated, tests are most likely to not fail due to the inability to detect the correct locator of the object — disrupting the flow of the whole project.

Prevent object locator flakiness

Flakiness in testing makes it hard for testers to identify whether or not their tests are stable. “NoSuchElementException” is one of the many errors that cause flakiness to the test design that Testers cannot have much control over. By ensuring the test’s design stays updated with the app, executions flow more easily, seamlessly, with less tendency to fail when it is unable to detect the object. (Find out more)

Save time and effort of maintaining test scripts

Failed test results prevent testers from gaining valuable insights on their tests, because the root causes may vary and do not reflect the real status and performance of the AUT.

Another important aspect of testing is to anticipate the users’ behavior and build a product that is ready for the unexpected. This does not sound simple; well, it truly is not. Self-healing takes away their concern for each time the UI gets updated: regression test cases have a high chance to fail, and eventually slow down the entire development flow.

Apply Self-healing using Katalon Studio

Prior to Katalon Studio version 7.6, users are familiar with the Auto-healing Smart XPath plugin provided by the Katalon team. The plugin is made to recover the broken default locators for testing in batch overnight.

As a successor, the new Self-healing capability of Katalon Studio is an upgrade of the same principle, only better.

See the full tutorial for self-healing

And other automation tools

Smartbear, another company also provides testing tools to have self-healing feature. So I think you could check it out from here.

Or with Testsigma, an AI-driven Test Automation tool they have UI Identifier to correct Object Locator Flakiness automatically. You can check out their blog at (https://testsigma.com/blog/self-healing-tests-maintenance-testsigma/)

Top 10 Codeless Testing Tools in 2021 | Every Tester Should Know

Brian Anderson — Tue, 21 Apr 2020 11:23:22 +0000

Coding — like any other craft — takes many years of practice to master. Having testers program a bunch of codes, fix errors in applications, and finish backlogs in weeks are challenging. According to the World Quality Report 2019–2020, lacking advanced programming expertise is chosen as one of the top challenges when applying test automation to agile projects. That’s where codeless automation testing tools came into place as a response to the challenge.
Simply put, codeless test automation means creating automated tests without the need to physically write codes. It provides testers lacking sophisticated programming skills with project templates for workflow, element libraries, and interface customization.
Furthermore, new innovations have been constantly applied to the testing process, for example, combining AI and Machine Learning into the self-healing mechanism, making the setup more user-friendly and maximizing test reliability. Thus, it helps simplify test automation and solve the coding skill conundrum.
The increasing adoption of codeless automation tools is anticipated to be one of the most influential automation testing trends in 2021.
The Most Popular Codeless Testing Tools in 2021

1. Katalon Studio

Katalon Studio is a complete codeless testing solution for Web, API, Mobile, and Desktop applications. Beginners can start with test automation immediately using its user-friendly interface, record, and playback, drag and drop commands. Instead of building a framework from scratch based on libraries, you can get out of the box and have Katalon Studio generate an automation framework for you.
For professional users, the rich set of built-in keywords and dual-scripting interfaces enable them to write advanced test scripts effortlessly. Thus, with Katalon Studio, you are able to extend testing capabilities, reuse in other projects, and enhance your team collaboration.

Highlight features:

*No programming skills & complex configuration required
*A mild learning curve, can be used for various types of testing
*Powerful Record and Playback to capture and convert all movements into backend codes (see Katalon Recorder)
*BDD Cucumber can convey test scenarios in natural languages, different teams can get involved in the testing process at ease
*Centralized analytics dashboard and reports
*Improve workflow, Orchestrate Tests & DevOps efficiently with Katalon TestOps
*Seamless integration with modern CI/CD tools and ALM systems such as Git, Jenkins, qTest, Jira, CircleCI,…
*Selenium lovers can leverage Smart Wait features to handle Selenium wait issues without any additional test scripts. Selenium can be easily migrated to Katalon. Users can maintain and continue to develop their existing Selenium test scripts in Katalon Studio.
*Built-in and custom keywords to transfer keywords in the test project without changing any external behaviors

Determined by more than 600 positive reviews as one of the 2020 Gartner Peer Insights Customers’ Choices for Software Test Automation, Katalon Studio positions themselves at the top of the codeless testing market. Moreover, now you can get their free resource about Test Automation on Katalon Academy.
Website: https://www.katalon.com
Pricing: Free — $869/year

2. Perfecto Scriptless

Perfecto Scriptless (formerly TestCraft) is an end-to-end automation testing platform for regression and continuous testing. Like any other codeless testing tool, the record-and-playback tool provides a test script with inputs coded beforehand that need to be customized later.
Tweaking some codes may break the test script. Thus, “Smart binding” in TestCraft refers to the smart element locators that can immediately capture various attributes such as IDs, position, text value, or size. They can even rebound during a run to match changes in the code. Since TestCraft merged with Perfecto, you can try out the highlight feature from both.

Highlight features:

*No extra plugins required
*Agility — Create test scenarios even before the application is ready
*Reports and analysis of every test step
*Support multiple data sets and parallel execution
*On-the-fly binding to adapt to most application changes
*Many built-in integrations including Test Management, Issue Management, Notifications & Communication
Website: https://www.perfecto.io/products/scriptless
Pricing: Commercial — Contact sales.

3. Perfecto

Perfecto is a cloud-based test automation tool for web-based and mobile applications, offering comprehensive AI analytics with error classifications to optimize the CI/CD pipeline. It also provides multiple supported devices, browsers, and OS combinations for mobile and web app testing. Within Perfecto, robust and AI-driven bindings account for 97.4% of app changes. One of them is the self-healing feature that enables test scripts to continuously run without any disruptions or maintenance required.

Highlight features:

*AI-driven test maintenance capabilities
*Smart reporting and analytics
*Debugging capabilities and test management
*Unlimited scalable execution on multiple Android devices and in parallel
*Increased test reusability
*Accelerate testing process and faster feedback between QA and developers
Website: https://www.perfecto.io/
Pricing: From $1,500/year.

4. CloudQA

CloudQA is a powerful cloud-based integrated codeless test automation platform designed to build, maintain, and execute tests against many browsers in parallel.
CloudQA gives you the equivalent power of Selenium without the sophisticated programming skills required. Recorded tests are put into maintainable modules and visualized into easily comprehensible forms. Even similar tests can be reused for regression testing, load testing, and synthetic monitoring.

Highlight features:

*High reusability for synthetic monitoring without modifying any codes
*Monitoring multiple data sets over various real-time browsers such as Chrome, Firefox, IE
*Huge capacity to maintain 1000’s of tests easily
*Support for data-driven testing
*Smart Record & Playback testing with screenshots after every step
*Schedule and run tests unattended
Website: https://cloudqa.io/
Pricing: From $3,588/year

5. Sikuli

Sikuli is a Python-based GUI test automation solution that can automate anything on the screen that has native GUI components such as toolbars, Windows dialogs, dialogue boxes, and Linux programs.
Moreover, the productive IDE in Sikuli plays as an editor in your visual scripting API to control screenshots and test the computer vision commands. However, compared to other codeless tools, Sikuli requires you to install the Java runtime as an extra plugin.

Highlight features:

*Image recognition algorithms to identify GUI elements based on images (see Extending Selenium with Image Recognition)
*Run automated tests for mobile on both iOS and Android devices
*Support various platforms including Windows, Mac, and Linux
*Use Python as a scripting language to extend the functionality of the tool (However, it can be scripted in JPython, an older version of Python)
*Not suitable for web automation tasks as it has no browser recorder
Website: http://sikuli.org/
Pricing: Open-source

6. Mabl

Mabl is a unified DevTestOps platform that makes it easy for both developers and non-developers to generate and execute automated functional UI tests faster and at scale.
Mabl can maximize the potential of your data, from user experience, visual design, to performance testing by running robust tests in minutes across all major browsers, in parallel and in the cloud without disrupting operations. Plus, reusable flows can modularize and re-add several if conditions to configure multiple data sets to run the same tests on and run simultaneously.

Highlight features:

*Mabl Trainer to track all interactions and allow adding assertions
*Reusable flows to edit any steps in the test
*Extensive data reports and insights
*Convert test output to multiple issues (such as Jira, BigQuery)
*Integrations with the CI/CD pipeline, Bamboo, Jenkins, Slack to keep pace with the deployment speed of DevOps
*Auto-heal keeps tests stable and test results secured
Website: https://www.mabl.com/
Pricing: Contact sales.

7. Applitools

Applitools automatically validates performance testing and user experience testing of your applications and websites in minutes. It covers all application platforms (Web, Mobile, and Desktop), test automation frameworks, and programming languages (Java, JavaScript, C#, Python, Ruby).
Since the new launch of Applitools Visual Grid, Applitools Servers receive DOM snapshots and analyze in parallel 30 times faster than the previous snapshot images (see Applitools testing framework).

Highlight features:

*Cross-device and browser tests
*Support multiple programming languages
*Comprehensive data reports with screenshots, baseline images and easy zoom-in on changes
*Functional feedback exchange between teams on Jira, Slack, or email
*Baseline branching to keep a separate test for each branch
Web API and merge CLI tools to allow you to merge the source and target baselines to resolve conflicts
*No inbound firewall changes or tunnel configuration required
Build functional and visual test results directly from TeamCity, Jenkins, Bamboo, HP Quality Center, Travis CI, Jira, CircleCI, and other CI systems
Website: https://applitools.com/
Pricing: Contact sales

8. Usetrace

Usetrace is a codeless automation testing tool that tirelessly automates cloud-based computers to test your applications and websites. These computers automatically repeat the traces of customers’ usage using record and playback, drag and drop interface — real-time on most used browsers.
Having Usetrace write all test scripts, your team can keep pace with the application from the dashboard, including how it is working in different browsers by diving into many traces.

Highlight features:

*Record and transform robust UI regression tests
Effective team collaboration on email, Jira, Slack, Atlassian HipChat, and FlowDock
*Monitor integral features — scheduled or on-demand
*End-to-end testing including backend, database, user experience, website performance, content, networks, and third-party testing
*Instant SMS notifications when your web tests fail
Website: https://usetrace.com/
Pricing: From $2,000/year

9. Leapwork

Leapwork is an easy-to-use codeless test automation tool with a user-friendly interface and advanced capabilities. Like any other codeless testing tool, within a few clicks, you can easily design automated test cases using the default library and predefined building blocks.
Leapwork even allows you to flexibly run tests on both local and distant environments by customizing run schedules. If any performance errors occur, Leapwork will automatically alert you in SMS. You can check live dashboards and insightful reports for better changes.

Highlight features:

*No need to modify automation flows whenever the system under test changes
*Combine automation flows into reusable components as sub-flows across test cases
*Support multiple types of applications (Web, Desktop and virtual applications)
*Support cross-browser testing and parallel executions
*Combining Center of Excellence set up to shorten the time for sharing automation flows, components, best practices and test cases across projects
*Documentation with videos and logs in audit trails
*Seamless integrations with REST API and CI/CD ecosystem to build your own integrations with third-party systems
Website: https://www.leapwork.com/
Pricing: Contact sales.

10. Ranorex

With the latest version of Ranorex 9.3 released, Ranorex is one of the best renowned GUI test automation frameworks for desktop, web-based, and mobile (native and WAP apps) testing. An impressive point of Ranorex is that its UI test execution reports are exported during test execution to reproduce bugs and maintain tests, thus reducing time to find out what and where things failed. They also comprise screenshots of those failed test cases.
Ranorex can be integrated with many technologies such as Silverlight, .NET, Winforms, Java, HTML5, Windows apps, iOS, and Android, to name but a few. However, it does not support macOS, Android web-based, and hybrid testing applications.

Highlight features:

*RanoreXPath to fully identify all types of GUI elements
*Ranorex Spy to clearly maps out GUI elements to the RanoreXPath expression
*Ranorex Recorder can capture and replay actions through an action table editor, then convert those actions into readable languages
*Comprehensive reports for quick error detection
*Ranorex Studio develops tests by completing codes, debugging, and managing test projects
*Flexible test automation interface that can easily run test suites by launching the .EXE files from the command line
*Enhancing test step reusability
*Only supports C# and VB.NET Shop languages
Website: https://www.ranorex.com/
Pricing: From $3,400/year

11. Test Architect

A codeless automation tool that targeted the extent of automation coverage & productivity by Keyword-Driven Testing. Test Architect really easy-to-use and scalable framework for Selenium. With features such as Intuitive IDE, Intelligent Out-of-the-Box Action Library, Interface Viewer, Multivariate Test Execution, Test Architect has become a friendly tool you should try.

Highlights features:

*Quickly set up and get your tests automated.
*Automate web browsers in Windows, Linux, and Mac.
*Extend Selenium WebDriver capabilities.
*Integration plugins with CI/CD/DevOps tools
*Automation analytics & test reports
*Built-in integrations to Visual Studio/TFS/Test Manager.
Website: https://www.testarchitect.com/
Pricing: Contact Sale

Conclusion

In summary, seeking the perfect codeless testing tool is still difficult among a handful of choices. Considering the advantages and disadvantages of each solution as well as your requirements, you can narrow your shortlist down based on your project status and team members’ capabilities.
All factors combined, the future of codeless automation testing tools is approaching at an exponential rate. They significantly relieve QA testers from the burden of mastering complex testing frameworks or technologies to automate tests fluently. The adoption of AI/ML takes these tools to a higher level with flexibility in changing codes, reducing flakiness, improving maintainability, and constant updates.
However, codeless doesn’t mean 100% codeless. Even though scripts are automatically created in the background, automation testers are required to have the basic coding knowledge to further customize their scripts.
What’s waiting for codeless test automation tools in 2021? Some QA professionals are viewing codeless as a preferred solution to embed into their existing DevOps cycle. Codeless testing tools would not replace automation testers completely, but shorten the pace of their jobs. The importance of codeless tools is expected to progress in 2021 and become the complementary piece of the automation testing puzzle.
References:
Top Software Test Automation Software of 2020 — Gartner Peer Insights
World Quality Report 2019–20 — Capgemini
Best Automation Testing Tools for 2021 (Top 10 reviews)
Top 15 Mobile Testing Tools for 2021 (Latest Update)

Top 10 Codeless Testing Tools in 2021 | Every Tester Should Know

Brian Anderson — Tue, 21 Apr 2020 11:23:22 +0000

Simply put, codeless test automation means to create automated tests without the need to physically write codes. It provides testers lacking sophisticated programming skills with project templates for workflow, element libraries, and interface customization.

Furthermore, new innovations have been constantly applied to the testing process, for example, combining AI and Machine Learning into the self-healing mechanism, making the setup more user-friendly and maximizing test reliability. Thus, it helps simplify test automation and solve the coding skill conundrum.

The increasing adoption of codeless automation tools is anticipated to be one of the most influential automation testing trends in 2020.

The Most Popular Codeless Testing Tools in 2021

1. Katalon Studio

For professional users, the rich set of built-in keywords and dual-scripting interfaces enable them to write advanced test scripts effortlessly. Thus, with Katalon Studio, you are able to extend testing capabilities, reuse in other projects, and enhance your team collaboration.

Highlight features:

No programming skills & complex configuration required
A mild learning curve can be used for various types of testing
Powerful Record and Playback to capture and convert all movements into backend codes (see Katalon Recorder)
BDD Cucumber can convey test scenarios in natural languages, different teams can get involved in the testing process at ease
Centralized analytics dashboard and reports
Seamless integration with modern CI/CD tools and ALM systems such as Git, Jenkins, qTest, Jira, CircleCI,…
Selenium lovers can leverage Smart Wait features to handle Selenium wait issues without any additional test scripts
Built-in and custom keywords to transfer keywords in the test project without changing any external behaviors

Website: https://www.katalon.com

Pricing: Free — $839/year

2. TestCraft

TestCraft is an end-to-end automation testing platform for regression and continuous testing. Like any other codeless testing tools, the record-and-playback tool provides a test script with inputs coded beforehand that needs to be customized later.

Tweaking some codes may break the test script. Thus, “Smart binding” in TestCraft refers to the smart element locators that can immediately capture various attributes such as IDs, position, text value, or size. They can even rebound during a run to match changes in the code.

Highlight features:

No extra plugins required
Agility — Create test scenarios even before the application is ready
Reports and analysis of every test step
Support multiple data sets and parallel execution
On-the-fly binding to adapt to most application changes
Many built-in integrations including Test Management, Issue Management, Notifications & Communication

Website: https://www.testcraft.io/

Pricing: Commercial — Contact sales.

3. Perfecto

Highlight features:

AI-driven test maintenance capabilities
Smart reporting and analytics
Debugging capabilities and test management
Unlimited scalable execution on multiple Android devices and in parallel
Increased test reusability
Accelerate testing process and faster feedback between QA and developers

Website: https://www.perfecto.io/

Pricing: From $3,600/year.

4. CloudQA

CloudQA is a powerful cloud-based integrated codeless test automation platform designed to build, maintain, and execute tests against many browsers in parallel.

CloudQA gives you the equivalent power of Selenium without sophisticated programming skills required. Recorded tests are put into maintainable modules and visualized into easily comprehensible forms. Even similar tests can be reused for regression testing, load testing, and synthetic monitoring.

Highlight features:

High reusability for synthetic monitoring without modifying any codes
Monitoring multiple data sets over various real-time browsers such as Chrome, Firefox, IE
Huge capacity to maintain 1000’s of tests easily
Support for data-driven testing
Smart Record & Playback testing with screenshots after every step
Schedule and run tests unattended

Website: https://cloudqa.io/

Pricing: From $1,188/year

5. Sikuli

Sikuli is a Python-based GUI test automation solution that can automate anything on the screen that has native GUI components such as toolbars, Windows dialogs, dialogue boxes, and Linux programs.

Moreover, the productive IDE in Sikuli plays as an editor in your visual scripting API to control screenshots and test the computer vision commands. However, compared to other codeless tools, Sikuli requires you to install the Java runtime as an extra plugin.

Highlight features:

Image recognition algorithms to identify GUI elements based on images (see Extending Selenium with Image Recognition)
Run automated tests for mobile on both iOS and Android devices
Support various platforms including Windows, Mac, and Linux
Use Python as a scripting language to extend the functionality of the tool (However, it can be scripted in JPython, an older version of Python)
Not suitable for web automation tasks as it has no browser recorder

Website: http://sikuli.org/

Pricing: Open-source

6. Mabl

Mabl is a unified DevTestOps platform that makes it easy for both developers and non-developers to generate and execute automated functional UI tests faster and at scale.

Mabl can maximize the potential of your data, from user experience, visual design, to performance testing by running robust tests in minutes across all major browsers, in parallel and in the cloud without disrupting operations. Plus, reusable flows can modularize and re-add several if conditions to configure multiple data sets to run the same tests on and run simultaneously.

Highlight features:

Mabl Trainer to track all interactions and allow adding assertions
Reusable flows to edit any steps in the test
Extensive data reports and insights
Convert test output to multiple issues (such as Jira, BigQuery)
Integrations with the CI/CD pipeline, Bamboo, Jenkins, Slack to keep pace with the deployment speed of DevOps
Auto-heal keeps tests stable and test results secured

Website: https://www.mabl.com/

Pricing: Contact sales.

7. Applitools

Since the new launch of Applitools Visual Grid, Applitools Servers receive DOM snapshots and analyze in parallel 30 times faster than the previous snapshot images (see Applitools testing framework).

Highlight features:

Cross-device and browser tests
Support multiple programming languages
Comprehensive data reports with screenshots, baseline images and easy zoom-in on changes
Functional feedback exchange between teams on Jira, Slack, or email
Baseline branching to keep a separate test for each branch
Web API and merge CLI tools to allow you to merge the source and target baselines to resolve conflicts
No inbound firewall changes or tunnel configuration required
Build functional and visual test results directly from TeamCity, Jenkins, Bamboo, HP Quality Center, Travis CI, Jira, CircleCI, and other CI systems

Website: https://applitools.com/

Pricing: Contact sales

8. Usetrace

Having Usetrace write all test scripts, your team can keep pace with the application from the dashboard, including how it is working in different browsers by diving into many traces.

Highlight features:

Record and transform robust UI regression tests
Effective team collaboration on email, Jira, Slack, Atlassian HipChat, and FlowDock
Monitor integral features — scheduled or on-demand
End-to-end testing including backend, database, user experience, website performance, content, networks and third-party testing
Instant SMS notifications when your web tests fail

Website: https://usetrace.com/

Pricing: From $2000/month

9. Leapwork

Leapwork even allows you to flexibly run tests on both local and distant environments by customizing run schedules. If any performance errors occur, Leapwork will automatically alert you in SMS. You can check live dashboards and insightful reports for better changes.

Highlight features:

No need to modify automation flows whenever the system under test changes
Combine automation flows into reusable components as sub-flows across test cases
Support multiple types of applications (Web, Desktop and virtual applications)
Support cross-browser testing and parallel executions
Combining Center of Excellence set up to shorten the time for sharing automation flows, components, best practices and test cases across projects
Documentation with videos and logs in audit trails
Seamless integrations with REST API and CI/CD ecosystem to build your own integrations with third-party systems

Website: https://www.leapwork.com/

Pricing: Contact sales.

10. Ranorex

Ranorex can be integrated with many technologies such as Silverlight, .NET, Winforms, Java, HTML5, Windows apps, iOS, and Android, to name but a few. However, it does not support macOS, Android web-based, and hybrid testing applications.

Highlight features:

RanoreXPath to fully identify all types of GUI elements
Ranorex Spy to clearly maps out GUI elements to the RanoreXPath expression
Ranorex Recorder can capture and replay actions through an action table editor, then convert those actions into readable languages
Comprehensive reports for quick error detection
Ranorex Studio develops tests by completing codes, debugging, and managing test projects
Flexible test automation interface that can easily run test suites by launching the.EXE files from the command line
Enhancing test step reusability
Only supports C# and VB.NET Shop languages

Website: https://www.ranorex.com/

Pricing: From $3,130/year

Conclusion

All factors combined, the future of codeless automation testing tools is approaching at an exponential rate. They significantly relieve QA testers from the burden of mastering complex testing frameworks or technologies to automate tests fluently. The adoption of AI/ML takes these tools to a higher level with flexibility in changing codes, reducing flakiness, improving maintainability, and constant updates.

However, codeless doesn’t mean 100% codeless. Even though scripts are automatically created in the background, automation testers are required to have the basic coding knowledge to further customize their scripts.

What’s waiting for codeless test automation tools in 2020? Some QA professionals are viewing codeless as a preferred solution to embed into their existing DevOps cycle. Codeless testing tools would not replace automation testers completely, but to shorten the pace of their jobs. The importance of codeless tools is expected to progress in 2020 and become the complementary piece of the automation testing puzzle.

References:

Top Software Test Automation Software of 2020 — Gartner Peer Insights
World Quality Report 2019–20 — Capgemini
Best Automation Testing Tools for 2021

Selenium Wait | The Good, The Bad and Alternative Solution

Brian Anderson — Mon, 02 Dec 2019 12:47:43 +0000

As a wanderer in automation worlds and a big fan of Selenium, I’m always eager to explore tools as much as possible. The most exciting thing about this hobby is that you can usually find out the beauty and the ugly of them and then challenge yourself to solve tools’ problems.

Selenium Wait Command you already know

One of the essential skills to master in Selenium is the use of the Wait commands. But ironically, this important skill is not for writing better scripts but for dealing with the timing issue in Selenium automation. And as you already know, that is due to the emergence of front end technologies such as Javascript or Ajax, which cause elements on web pages not available for interacting at predictable time intervals.

Anyway, let’s dive into the details of those Wait techniques, especially the disadvantages I have experienced with them, which I believe might be good for you to know.

Let’s take a look at the overview chart of Selenium Waits below:

I will use this website to make an example for this post.

Implicit Wait

The implicit wait will tell the WebDriver to wait for a certain amount of time when trying to find an element(s) if they are not immediately available before it throws a NoSuchElementException message. It is the global wait for all driver.findelement instances with the default setting are 0. This means that any find element on the page could take the time of implicit wait is set for.

Implicit wait comes in handy as it’s easy for applying and adjusting. However, this type of approach is a game of guess since the execution time depends heavily on many external factors such as browser performance, network performance, or application server performance. No one can tell how long to wait is optimized as waiting too long leads to waste while being too short leads to more likelihood of failure.

In the above example script, the value of the implicit wait is given as 30 seconds. If an element or elements are not immediately available on the web page, the wait command will trigger and wait until the WebDriver can find the web elements and continue the next steps. After 30 seconds, if the web element(s) could not be found, it raises an exception message, and the test will fail.

Implicit Wait methods take two arguments:

Time — Amount of time to wait before throwing an exception.
Unit — Time unit, which could be SECONDS, MILLISECONDS, MINUTES, etc.

I do not recommend using implicit waits in the automation scripts, especially in the new front-end technologies such as Javascript or Ajax, where each action does not reload the page or render it again. Example with the below the web page, after navigating to the homepage, the element //*[@id =”pageContent”] will show the text value as:

Hello, this is a demo for a Tutorialzine tutorial. To test it, click some of the buttons above. Have a nice stay!

After click on “Page 1” button, the value of //*[@id =”pageContent”] has changed to below content without reloading the element or web page.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam feugiat neque vel metus sodales auctor sed et arcu. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Phasellus cursus tellus ac urna sollicitudin viverra.

If we use implicit wait in this case, the wait command will not trigger as the element //*[@id =”pageContent”] is present , and it will get an element text and executing the assert command, it is not the value that we are expecting after a click on Page 1 button (actual this is the value before that), it will lead to an incorrect value between expectation and the actual one.

As the above result, the implicit wait does not work in my case when the web element is visible on the web page, but the content inside has changed with Ajax technology, so let me move to the next waiting method with Explicit.

Explicit Wait

The explicit wait is used to wait for specific conditions or the maximum time exceeded before throwing an ElementNotVisibleException.

This is kind of an intelligent solution as it will wait for dynamically loaded Ajax elements with the wait conditions. However, it can be only applied for a specified element, and you need to do this for case by case. The more intelligent and optimized for timing in comparison with implicit wait has its own price with “unnecessary” code everywhere in your automation scripts.

There are two main ways to implement Explicit Waits : WebDriverWait and FluentWait.

In the above example, instead of using implicit wait , I will use the WebDriverWait with the condition as ExpectedConditions.textToBePresentInElementLocated

After clicking on Page 1, if there is no expected message displayed, the WebDriver will wait for the textToBePresentInElementLocated before moving to the next steps. In this case, the assert keyword will execute “after” the expected content has loaded, and the asserted result will Pass.

The following Expected Conditions can be used:

alertIsPresent()
elementSelectionStateToBe()
elementToBeClickable()
elementToBeSelected()
frameToBeAvaliableAndSwitchToIt()
invisibilityOfTheElementLocated()
invisibilityOfElementWithText()
presenceOfAllElementsLocatedBy()
presenceOfElementLocated()
textToBePresentInElement()
textToBePresentInElementLocated()
textToBePresentInElementValue()
titleIs()
titleContains()
visibilityOf()
visibilityOfAllElements()
visibilityOfAllElementsLocatedBy()
visibilityOfElementLocated()

Beside WebDriverWait , you can implement Explicit with FluentWait , both are classes and implements the Wait interface.

The fluent wait can be considered an advanced explicit wait as you can specify the polling interval and exceptions to ignore along with the maximum wait time. Moreover, you can also return the wanted elements in one line of code rather than one line for waiting and another for element retrieving as in explicit wait. Being that flexible and convenient, this type of wait that I have used the most. However, fluent wait still has the same problems as mentioned in explicit technique: the waiting code is everywhere. Even if you wrap those codes into some common functions or places, it’s still there waiting as a slowly exploded boom.

Time is gold and why I try to find an alternative solution

No one can deny the effectiveness of Wait for tackling the timing issue, but everyone has to suffer the poor efficiency of this solution as well. This is because of the overhead scripting and maintenance effort, the unpredictability and instability of the automation execution, which eventually cost me a lot of time to make every just fine.

I bet that as automation engineers, we all suffer the hard times to adjust the Wait limit to make everything work in the daytime at the office but then all the test cases fail during overnight execution. Automated testing turns out not to make testing faster and more reliable but even slower and not trustworthy. Many cases that I have seen that teams decide to go back to manual testing in order to achieve their deadlines rather than keep moving with automated testing.

Furthermore, a huge amount of effort for automation that I’ve spent is for building a framework to mitigate the scripting and maintenance effort for implementing the Wait approach. Even though it can be reused in most of my projects but it can never come to a “perfect” state that I imagine it should be. My team members still make mistakes, overuse Wait command as the silver bullet and at the end of the day we still have the spend overhead effort to fix the problem of another problem and can never totally focus on the automation business.

Feeling tired of sticking to the all mighty Wait solution. I feel that I should stop depend on that approach only and look out to other solution that I make my life better

Smart Wait — The radical solution to Selenium timing issue

After a long time struggling with the timing issue, I figured out that the right solution for that problem must satisfy two essential conditions. Firstly, that solution has to deal with the dynamic nature of web page loading. And secondly, the solution should not introduce any overhead effort.

Katalon Smart Wait

Just recently, when I came across the Smart Wait function in Katalon Studio version 7 (www.katalon.com). This feature appears to match with the two conditions I mentioned above.

Deal with dynamic web page loading directly : I see that the Smart Wait works with the Ajax loading content, I have tested the same test cases with the Smart Wait option and it looks good.
Smart Wait internally captures all AJAX requests and DOM modifications. It would attempt to wait until all AJAX requests are completed so that all the submitted and requested information on the page are ready. Then every short period of time, Smart Wait would compare the previously captured DOM state against the current state to make sure that the displayed information on the page is not going through a mutation.
No code : The Smart Wait will be enabled as a default setting without more scripting required, you can turn it ON or OFF by going to the Project menu > Settings > Execution and Enable / Disable the Default Smart Wait option.

One of the trade-offs when using any kind of wait is the increase in the execution time. This certainly applies to Smart Wait as well, although given the increased instability of the test and that stability is more important, it is a reasonable exchange. Another limitation is that Smart Wait does not specifically check for conditions such as invisibilityOfElement or textToBePresentInElement such as the case in Explicit Wait. This is by design because these conditions are better made explicit. If your website already has a monitoring library that also captures AJAX requests, then it may come into conflict with Smart Wait.

Although having certain limitations, this is a very promising approach because it alleviates some of the frequently observed problems with Selenium such as elements being acted on when they are not ready.

Conclusion

The Implicit method is fast and easy to implement. However, the execution time depends heavily on many external factors and not working correctly in the new technology with Ajax or JavaScript.

Explicit methods are more flexible and allow you to wait for specific conditions or the maximum time exceeded but it can be only applied for a specified element, you need to do this for case by case and require more code.

Smart Wait in Katalon Studio effectively solves the wait issue without coding required, it saves a great amount of time in the setup stage and works perfectly with Ajax or JavaScript pages. Despite having some limitations, I believe that the Smart Wait feature would be improved in the very next releases of Katalon Studio.

You can download Katalon Studio and try it out yourself.

If you know any other solutions to handle Selenium Wait Commands, please comment below. Thanks for reading!

3 Latest Research on the Software Testing Industry | Gartner Referred

Brian Anderson — Thu, 09 May 2019 11:28:28 +0000

Gartner 3 Latest Research on the Software Testing Industry

Gartner is a well-known name among prestigious research companies and has been a reputable source of reference for customers to evaluate products from various industries. This article will give you a recap of the three latest findings of the software industry, providing you with more insight into this business field, as well as its product offerings.

Best Test Automation Tools as of Q1 2019

Having recently announced the results of the annual Gartner Peer Insights ‘Voice of the Customer’: Software Test Automation, Gartner seeks to give insights into the software test automation industry’s latest products. The goal of the report is to help product users gain better insights into different types of product offerings based on end-users’ reviews and Gartner’s validation criteria.

All Gartner reviews go through a strict validation process to ensure they are qualified, valuable, and authentic. To be selected as a Customer’s Choice, vendors are required to have a product aligned with the relevant market, receive an overall rating of 4.2 or higher out of 5 stars, and have at least 50 approved customer reviews.

Without further ado, the top choices of the Gartner Peer Insights ‘Voice of the Customer’ in the Software Test Automation industry are as follows:

1. Katalon Studio

Katalon Studio, developed by Katalon, is a powerful automation solution with a comprehensive and integrated set of features for Web, API, and mobile test automation that users can implement into their CI/CD pipeline. Katalon’s most popular product — Katalon Studio — is considered a tool that is easy to use, rich with features, and user-friendly for both novice and expert. Katalon Studio also supports a wide range of software methodologies such as data-driven testing, behavior-driven integration (BDD), CI — DevOps practices, and more.

“Easy To Use, Good UI And The BEST In The Industry Of Automation.” — Product Engineer — QA in the Manufacturing Industry

Website: https://www.katalon.com

2. Visual Studio

Developed by Microsoft, Visual Studio IDE is commonly recognized as a quality integrated development environment (IDE) that is perfectly compatible with .NET technology. It is even considered a great alternative to existing lightweight IDE, and a complete tool to develop applications that are based on Web, cloud, system, and data.

“Visual Studio IDE is the only IDE you need.” — software developer in the Services Industry

Website: https://visualstudio.microsoft.com/vs/

3. Selenium WebDriver

Selenium WebDriver is known for being the most popular free and open-source test automation framework. According to the peer reviews, Selenium WebDriver is commended for its flexible support of multiple system environments, browsers, and programming languages including Java, Python, C#, and PHP.

“One Stop Solution to Web Browser Automation” — Software Engineer in the Miscellaneous Industry

Website: https://www.seleniumhq.org/projects/webdriver/

4. NeoLoad

NeoLoad, developed by Neotys, is a testing platform for enterprises to automate tests from APIs, mobile, to Web applications. Peer reviewers consider NeoLoad an “excellent and great tool” with a strong feature set to analyze test results. The software is also commended as “a very efficient and up-to-date-interfaced platform.”

“The greatest testing app I have ever tried out” — Project Manager in the Media Industry

Website: https://www.neotys.com/

5. Parasoft

Parasoft provides innovative tools that automate time-consuming testing tasks and provide management with intelligent analytics necessary to focus on what matters. Parasoft products, including Parasoft Development Testing Platform, Parasoft Virtualize, Parasoft SOATest are recommended for their flexibility, comprehensiveness, and rich features such as integration with Selenium WebDriver, XML messages, functional API testing, and so on.

“Parasoft SOAtest helped us accelerate our CI/CD pipeline creation.” — API Test Lead in the Communications Industry

Website: https://www.parasoft.com/

6. Unified Functional Testing

Developed by Micro Focus, Unified Functional Testing (UFT) is praised as a top-notch tool for Web-based and API testing. The paid software is considered a value automation solution that is also easy to use and implement. UFT is also packed with versatile features, as well as the technical support services.

“Best Automation tool ever” — SAP Test Lead in the Manufacturing Industry

Website: https://www.microfocus.com/en-us/products/unified-functional-automated-testing/overview

7. Tricentis

The Austrian software company is known for the two most popular testing products: Tricentis Tosca and qTest, which provides solutions to optimize Agile and DevOps. Peer Insights reviewers recognize these products as a “scriptless automation solution” and a great tool for ERP testing as well as many other technologies.

“Brought us from Zero to hero in QA automation” — IT Leader SDLC and QA tools in the Finance Industry

Website: https://www.tricentis.com/

Apply Shift-Left Testing to Deliver Higher Quality Software Faster

“Application leaders responsible for developing digital business applications are under increasing pressure to maximize quality and velocity of delivery,” according to QASymphony, a software automation enterprise. Adopting a “shift left” a testing approach is no longer an option, but a requirement for development teams to release products with high quality faster.

There are three major challenges that software development teams are facing throughout the software development life cycle. First, conventional software development teams tend to focus on coding — which is the first phase of a development cycle — instead of testing these features. This likely causes code instability and decreasing developer efficiency.

Second, it is costly and time-consuming for teams to manually test and integrate with other tools, let alone perform these tasks early and often. And third, user-interface-driven testing is also slow, fragile, and does not provide full test coverage.

Adopting a “shift left” approach to testing is the key to tackle these challenges. The “shift left” model means teams should start testing features at the beginning stage and throughout the entire development cycle. In other words, “test early, test often.”

According to the report, application leaders in digital business should, first, combine coding and testing into one activity; and perform this activity at the beginning of the development cycle. Second, team leaders and testers should make it easier for developers to receive feedback about the quality of changes by implementing continuous integration (CI) and automated testing.

Finally, the report proposed a restructuring of the Test Automation Pyramid to allocate automation efforts and yield maximum benefits.

“By 2020, DevOps initiatives will cause 50% of enterprises to implement continuous testing using frameworks and open-source tools.”

Critical Capabilities for Automation Software Testing in 2019

While the software automation industry is waiting for Gartner to release the list of critical capabilities for 2019, application leaders can still refer to the 2018 report to evaluate which functional software test automation offerings are best for their testing purposes.

These critical capabilities are also evaluated based on five common testing use cases: Continuous Testing, Open-source-based Testing Acceleration, Progressive Web Apps/Responsive Web/Native Apps, API/Web Service Testing, and Enterprise End-to-end Testing.

1.Match to nontechnical role skills

This criterion measures how suitable and user-friendly the product is, not only to test automation experts but also to other roles with limited to no testing experience in the team.

2. Match to technical role skills

This criterion evaluates the product’s ability to help test automation experts — including testers and developers with proficient coding skills — enhance their productivity.

3. Templates and accelerators

Whether and how well the product supports industry verticals and/or vertical applications through templates and accelerators.

4. Change impact analysis

This measures how effectively the product helps identify sets of relevant test cases that need to be executed when there are changes in requirements and/or functionalities. This criterion also considers other factors such as risk, criticality, or business priority.

5. Cross-platform and cross-browser support

This criterion addresses the range of platforms, browsers, and environments on which the product can operate, including Windows, macOS, Linux, Android, and iOS.

6. Intelligent automation

Measures how well the implementation of artificial intelligence (AI), machine learning, self-healing heuristics, bots, and automatic (re)generation of tests the product offers.

7. Breadth of technology support

This criterion evaluates the product on its coverage of different technologies, including UI frameworks and widgets, API formats, and protocols to allow end-to-end testing.

8. Integration Capabilities

How well and how much the automation product integrates with other tools and environments, including IDE, CI/CD, defect management, and ADLM.

9. Open-source support

Whether and how well the product supports well-known open-source tools and frameworks such as Selenium, Appium, JUnit, and so on.

10. Dashboards and reporting

This final capability assesses the product’s ability and effectiveness in providing tracking tools after the tests have been executed.

References:

Gartner Peer Insights Customer’s Choice
Best Automation Testing Tools for 2019 (Top 10 reviews)
The State of Quality Report 2022 | Key Trends to Thrive in Software Testing
4 New Gartner Research Reports on Software Testing — Tricentis blog

Best Automation Testing Tools for 2021 (Top 15 reviews)

Brian Anderson — Thu, 26 Oct 2017 09:21:22 +0000

Best Automation Testing Tools for 2021

The demand for delivering quality software faster — or “Quality at Speed” — requires organizations to search for solutions in Agile, continuous integration (CI) and DevOps methodologies. Test automation is an essential part of these aspects. The latest World Quality Report 2020–2021 suggests that test automation is the biggest bottleneck to deliver “Quality at Speed,” as it is an enabler of successful Agile and DevOps adoption.

Test automation cannot be realized without good tools; as they determine how automation is performed and whether the benefits of automation can be delivered. Test automation tools is a crucial component in the DevOps toolchain. The current test automation trends have increased in applying artificial intelligence and machine learning (AI/ML) to offer advanced capabilities for test optimization, intelligent test generation, execution, and reporting. It will be worthwhile to understand which tools are best poised to take advantage of these trends.

Here are the top test automation tools that are believed to best address the challenges in automation over the next few years. The tools included in this list are selected from these criteria:

Supporting API and services testing
Offering some AI/ML and analytics capabilities
Popularity and maturity

Top 5 Automation Testing Tools for 2021

1. Selenium

Selenium is the household name when it comes to test automation. It is considered the industry standard for user interface automation testing of Web applications. Specifically, 54% percent of respondents used Selenium as their automation testing tool, according to the The State of Quality 2022 Report .

For developers and testers who have experience and skills in programming and scripting, Selenium offers flexibility that is unseen in many other test automation tools and frameworks. Users can write test scripts in many differents languages (such as Java, Groovy, Python, C#, PHP, Ruby, and Perl) that run on multiple system environments (Windows, Mac, Linux) and browsers (Chrome, Firefox, IE, and Headless browsers).

Since the latest version Selenium 4.0.0 Alpha 7 released in November of 2020, it offers a more seamless testing experience by allowing W3C standardization. Thanks to this new feature, developers now no longer need to modify codes to work with different browsers as Chrome, Safari, Firefox, Internet Explorer, and Edge will also follow W3C standard protocol.

To use Selenium effectively, users have to possess advanced programming skills and need to spend considerable time to build automation frameworks and libraries necessary for automation. This is the main disadvantage of Selenium, which is addressed in other tools built for codeless test automation like Katalon Studio.

Website : https://selenium.dev/

License : Open-source

2. Katalon Studio

Katalon Studio is a powerful and comprehensive automation solution for testing API, Web, mobile, and desktop application testing. It also has a rich feature set for these types of testing and supports multiple platforms including Windows, macOS, and Linux.

Leveraging Selenium and Appium engines, Katalon Studio provides a uniquely integrated environment for testers who find difficulties in integrating and deploying different frameworks and libraries to use Selenium and Appium, as well as those who are already familiar with these engines.

Katalon Studio was named a 2020 Gartner Peer Insights Customers’ Choicefor Software Test Automation with more than 600 positive reviews, which once again, proves the tool is now one of the biggest players in the market.

Highlights of the tool include:

A complete feature set for test automation of API/Web services, Web, and mobile applications
Supports both SOAP and RESTful for API and services testing
Hundreds of built-in keywords for creating test cases
Support BDD Cucumber to express test scenario in natural languages
Can be used for both automated and exploratory testing
The testing capability can be extended via plugins on Katalon Store
View quality reports on Katalon TestOps, a test orchestration platform for Agile teams.

In January 2021, Katalon released its newest update — Katalon Studio 7.9.0 — with major enhancements: upgraded Eclipse IDE framework with modern look and feel, integration with application lifecycle management (ALM) tools, defaulted Class File Decompiler for debugging support, and resolved several bug issues.

Website : https://www.katalon.com/

License : Free - $759 per year

3. UFT One

UFT One (formerly known as UFT) is a popular commercial tool to test Web, desktop, mobile, and RPA application. It has been extended to include a good set of capabilities for API testing. By supporting multiple platforms for the target application under test (AUT), UFT One provides a convenient choice to test the AUT that operates on desktop, web, and mobile.

UFT One provides several advanced capabilities for smart object detection, image-based object detection, and correction. In November 2020, Microfocus released the latest version of UFT (v15.0.2). The update offers new features and enhancements that streamline the testing processes, improve testing efficiencies, and sustain quality while reducing testing time.

Several highlights of the tool:

Create a more platform-agnostic test with the suggested object steps in AI Transformation Assistant
Use ParallelRunner to run multiple API and GUI tests in parallel
Track and report WiFi state, thermal state, and available disk space metrics in the Record & Run settings
Support automatic page checkpoints for Firefox or Chrome browsers

Website : https://software.microfocus.com/fr-ca/software/uft

License : From $3,200 per year.

4. TestComplete

TestComplete continues to be on the list this year for its powerful and comprehensive set of features for Web, mobile, and desktop application testing. Testers can use JavaScript, VBScript, Python, or C++Script to write test scripts.

Like UFT One, TestComplete has an object recognition engine that can accurately detect dynamic user interface elements. This engine is especially useful in applications that have dynamic and frequently changing user interfaces.

The TestComplete version 14.72 was released in January 2021 including improvements in test run and web testing. Self-healing functionality leverages the improved AI-based algorithms to find a replacement object by its image. Testers can easily use TestComplete’s record and playback feature, like Katalon Studio. They can insert checkpoints into test steps to verify results. As a product of SmartBear, TestComplete can be integrated easily with other products offered by SmartBear.

Website : https://smartbear.com/product/testcomplete/overview/

License : From $6,095 per year

5. SoapUI

SoapUI is not a test automation tool for Web or mobile app testing, but it can be a tool of choice to test API and services. It is a headless functional testing tool specifically designed for API testing.

SoapUI supports both REST and SOAP services. API automation testers can use either the open-source or pro version. The pro edition has a user-friendly interface and several advanced features such as assertion wizard, form editor, and SQL query builder. SoapUI is a tool of the ReadyAPI suite, offered by SmartBear.

The tool provides a quite comprehensive feature set for API testing with many advanced capabilities, including:

Generating tests easily using drag and drop, point-and-click
Powerful data-driven testing with data from files and databases
Asynchronous testing
Scripts can be reused easily
Creating mock services with RESTful mocking

Moreover, SoapUI has recently brought the API Explorer feature from SoapUI Pro into the open-source version, allowing developers and testers to instantly debug their API’s responses.

The latest version 5.6 of SoapUI (Open-source) was released on July 2020 with updated third-party libraries (org.apache.httpcomponents, commons-logging, commons-codec, and Junit). Whereas, SoapUI Pro 3.5.1 is the latest successor in the commercialized version powered by Smartbear, including a number of improvements to make API testing seamless and more powerful.

Website : https://www.soapui.org/

License : Free - $749 per year.

Please note that SoapUI and Postman are two representatives of several top API testing tools in this list only. To learn more about the comparison between these tools, please refer to this post.

6. IBM Rational Functional Tester (RFT)

IBM Rational Functional Tester is a test automation tool designed for testing applications that are developed using different languages and technologies such as Web, .Net, Java, Visual Basic, Siebel, SAP, PowerBuilder, Adobe Flex, and Dojo Toolkit. It is also a data-driven testing platform for functional and regression testing.

Some highlights of the tool:

Visual editing through screenshots provides a visual, storyboard format for representing test actions that allow users to accommodate frequent user interface changes and avoid increases in maintenance overhead.
Advanced ScriptAssure technology
Earlier data detection
Test scripting allows users to choose between either Java or Visual Basic .NET
Integration with other software to support collaborative application lifecycle management

RFT V10.1.0 (July 2020) has been made generally available with new features as well as bug fixes to client-reported and internally found defects. The unified report enhanced in this version is available to Functional, Web UI and Compound tests, and Accelerated Functional Test (AFT) suites, as well as an extensive user interface, to analyze and filter the test result details.

Website : https://en.wikipedia.org/wiki/Rational_Functional_Tester

License : Commercial — Contact Sales

7. Tricentis Tosca

There are a few continuous testing platforms that provide comprehensive toolsets to support most, if not all, testing activities ranging from test design and test automation to test reports and analytics. Tricentis Tosca is one of them.

This tool has many features such as dashboards, analytics, integrations, and distributed executions to support continuous integration and DevOps practices. Moreover, it offers a friendly user interface and a rich feature set for designing, implementing, executing, managing, optimizing API tests.

Some other highlights of the tool:

Can be easily integrated to be a crucial part of DevOps processes
API tests can be performed across browsers, mobile devices, and platforms
Multiple protocols and standards are enabled, including HTTP(s) JMS, AMQP, Rabbit MQ, TIBCO EMS, SOAP, REST, and IBM MQ
A good set of test reporting and analytics capabilities
Automate UI test cases based on a mockup instead of codes

Tricentis Tosca supports API security configuration in the API Connection Manager. It also allows users to use the signature security option to sign multiple parts of a message.

Tricentis claims to significantly reduce regression testing (within the magnitude of minutes from weeks). This claim, however, needs to be carefully verified by testing teams.

In the latest update 13.4 in September 2020, the first version of Project NEO, Tricentis’ AI-powered test automation engine was introduced. With Project NEO, automated testing can be quickly and virtually executed.

Website : https://www.tricentis.com/

License : Commercial — Contact Sales

8. Ranorex

Having been around for many years, Ranorex provides a comprehensive and professional set of features for Web, mobile, desktop, and API testing. Leveraging its experience in desktop-based test automation, Ranorex has advanced capabilities for UI element identification, editing, and management.

Like Katalon Studio, Ranorex makes automation testing easy for testers with its friendly and intuitive GUI, record/playback, and script generation.

Testers can integrate Ranorex with Selenium Grid to enable distributed testing along with parallel test executions.

Ranorex 9.3 introduces enhanced Jira and TestRail reporting. The newest release, Ranorex Studio 9.3.4 (November 2020) increased support for Delphi TAdvStringGrid’s hidden columns, also for Apple iOS 14 and Android 11 based devices, in addition to some bug fixes.

Website : https://www.ranorex.com/

License : From €5,460 for Premium Bundle.

9. Postman

Postman is another automation tool designed for API testing. Users can install this tool as a browser extension or a desktop application on Mac, Linux, Windows. It is popular not only among testers for API test automation but also developers who use the tool to develop and test APIs. It is, in fact, a development environment to develop and test APIs.

Some highlights of the tool:

Comprehensive feature set for designing, debugging, testing, documenting, and publishing APIs
Friendly and easy-to-use user interface
Supporting both automated and exploratory testing
Accepting Swagger and RAML API formats
Requests and respondents can be packaged and shared with team members

Starting from version 7.3 in 2020, Postman helps users better organize the collections and API elements (mock server, monitors, tests, and documentation) generated from API schemas by using the new advanced preferences. The latest version 8.0 for better API development experiences has been released in January 2021. The new version enforces the universal navigation function (workspaces, reports, explore, etc.) as well as empowers a unified web and native interface.

Website : https://www.getpostman.com/

License : Free — $216 per user per year

10. Apache JMeter

Highlights of the tool include:

Lightweight with a simple and easy-to-use user interface
Test results can be replayed
Support CSV files to set values for API parameters
Support integration with CI tools such as Jenkins. JMeter is often used as a part of CI and DevOps toolchains

As the most recent release in December 2020, JMeter 5.4 has been stacked with multiple changes:application-modal splash screen, LogKit logger partial remove, bug fixes, and JMeter implements for functional testing.

Website : https://jmeter.apache.org/

License : Open-source

11. Telerik Test Studio

Developed as a Windows-based software testing tool, Telerik Test Studio is widely known for web and desktop functional testing, software performance testing, and mobile application testing. This solution allows both code-based capabilities and a codeless approach, ensuring optimal app quality with the most outstanding results.

Highlights of the tool include:

Intuitive recorder with cross-browser support
Test results displayed in the executive dashboard
Intelligent hybrid element detection
Scheduling and concurrent runs
Test element management

The latest release, Test Studio R3 2020 SP1 (v.2020.3.1209)in December 2020, brings brand new features, such as Dark Theme and new notification version in the status bar. Additionally, there are also updates in the communication mechanism for created projects to work efficiently.

Website : https://www.telerik.com/teststudio

License : Contact Sales

12. Watir

Watir, pronounced as “water”, is a light-weight open-source testing tool designed for automating web application testing. Watir brings on a rich set of APIs to help testers interact with the browser, locate page elements, work with alerts, headless testing, and more.

Some highlights of the tool:

Location web elements
Taking screenshots of the testing done as required
Page performance
Test alerts popup in your UI or website
Headless testing to execute UI test cases at the command line

After a long time without an update, Watir came back with release 6.17 on August 28, 2020. This version has addressed some critical bugs relevant to locator code, browser instance, and more. Moreover, with Watir 6.17, browsers now accept a service parameter with a Selenium Service instance value. Ruby 2.5 or greater is required for this version.

Website : http://watir.com/

License : Open-source

13. Worksoft

Known as a platform for continuous test automation software, Worksoft offers a suite of features used for ERP applications, web apps, mobile apps, and more. This platform is certified by SAP for integration capability with SAP applications.

Highlights of the tool include:

Using Groovy as the scripting language & support external Java library
Ability to support Agile and DevOps testing practices
Seamless integrations with other testing tools, ALM systems, and DevOps toolchains
Object recognition capabilities for SAP Fiori plus the rapid release of version updates

The latest version Worksoft Certify 12, offers expanded DevOps integrations, plus out-of-the-box optimizations for modern web and cloud applications. Certify also adds support for Postman, as well as improvements to 12 Jenkins with an official plug-in in the Jenkins store.

Website : https://www.worksoft.com/

License : Commercial (Contact Sales)

14. Appium

The other open-source tool in this list is Appium, an automation testing tool for mobile web, hybrid applications on iOS/ Android mobile and Windows desktop platforms.

Highlights of the tool include:

Support multiple programming languages (Java, Ruby, Python, etc.)
Cross-platform (Android & iOS) testing on real devices, emulators, and simulators

In January 2021, Appium had a patch release with a bug fix in setting the session capabilities.

Website : https://github.com/appium

License : Open-source

15. Cucumber

Last but not least, Cucumber is an automation testing tool that supports Behavior Driven Development (BDD), providing an easy way for everyone to write and execute a test case, regardless of technical knowledge.

Highlights of the tool include:

Working across multiple platforms: Java, JavaScript, Ruby, .NET, and many other platforms
Seamless automation with Selenium, API calls, or direct function calls in the same process
Integration with Jira to empower team’s collaboration
Reports in HTML, JSON, and other formats to monitor the team’s performance

The latest version Cucumber Open 6.9.1 was released on December 14, 2020, with some bug fixes in Core and JUnit Platform.

Website : https://cucumber.io/tools/cucumber-open/

License : Open-source

— — — — —

This list of top tools here is not exhaustive; but it represents the best tools that are mature, popular, and providing capabilities using AI/ML to address the challenges that organizations are now facing when it comes to deliver “Quality at Speed.” This list also includes the tools that solely support API and services testing essential for successful Agile and DevOps transformation.

It can be said that the software testing tools landscape in 2021 continues to evolve. The emerging trends — AI and machine learning, codeless, big data and IoT testing, the intensive adoption of test automation (especially in Agile teams), etc. — will create both challenges and opportunities for the existing tools and new players to assert value to the testing communities.

Your choice of testing tools should not only meet your current needs but should also focus on potential trends and improvements. A decent tool should support basic optimization, automation of test case and data generation, smarter solutions, and analytics.

More tools will be introduced as there are opportunities for growth; even though the level of test automation in organizations is low at between 14% and 18% (WQR report). Moreover, it’s going to be exciting to see how the existing and upcoming tools apply AI/ML to address the challenges ahead. API and services testing is also a trend that should see further development in the future.

References:

Top Software Test Automation Software of 2020 — Gartner Peer Insights
The State of Quality 2022 Report
Top 10 API Testing Tools (Details & Updates Done for You!)
Top 15 Mobile Testing Tools for 2021 (Latest Update)

Best Automation Testing Tools for 2020 (Top 10 reviews)

Brian Anderson — Thu, 26 Oct 2017 09:21:22 +0000

Best Automation Testing Tools for 2020

The demand for delivering quality software faster — or “Quality at Speed” — requires organizations to search for solutions in Agile, continuous integration (CI), and DevOps methodologies. Test automation is an essential part of these aspects. The latest World Quality Report 2018–2019 suggests that test automation is the biggest bottleneck to deliver “Quality at Speed,” as it is an enabler of successful Agile and DevOps adoption.

Here are the top test automation tools that are believed to best address the challenges in automation over the next few years. The tools included in this list are selected from these criteria:

Supporting API and services testing
Offering some AI/ML and analytics capabilities
Popularity and maturity

Top 5 Automation Testing Tools for 2020

1. Selenium

Selenium is the household name when it comes to test automation. It is considered the industry standard for user interface automation testing of Web applications. Almost nine out of ten testers are using or have ever used Selenium in their projects, according to Survey on Test Automation Challenges.

For developers and testers who have experience and skills in programming and scripting, Selenium offers flexibility that is unseen in many other test automation tools and frameworks. Users can write test scripts in many different languages (such as Java, Groovy, Python, C#, PHP, Ruby, and Perl) that run on multiple system environments (Windows, Mac, Linux) and browsers (Chrome, Firefox, IE, and Headless browsers).

Selenium has recently released its first alpha version of Selenium in April 2019. The announcement of the Selenium 4 official version is yet to be determined, but you can expect the release would come with many improved and enriched features.

Website: https://selenium.dev/

License: Open-source

2. Katalon Studio

Highlights of the tool include:

A complete feature set for test automation of API/Web services, Web, and mobile applications
Supports both SOAP and RESTful for API and services testing
Hundreds of built-in keywords for creating test cases
Support BDD Cucumber to express test scenario in natural languages
Can be used for both automated and exploratory testing
The testing capability can be extended via plugins on Katalon Store In-depth views of reports on Katalon TestOps

In April 2020, Katalon has released its newest update — Katalon Studio 7.3 — with major enhancements that support scalable projects, solving the Wait issue in Selenium, sharing test artifacts across multiple projects, and more.

Website: https://www.katalon.com/

License: Free - $759 per year

3. UFT One

UFT One provides several advanced capabilities for smart object detection, image-based object detection, and correction. In January 2020, Microfocus released the latest version of UFT (v15.0). The update offers new features and enhancements that streamline the testing processes, improve testing efficiencies, and sustain quality while reducing testing time.

Several highlights of the tool that support API testing:

Intuitive user interface for creating, executing, and reporting API tests
Support generating API tests from WADL documents
Tests’ actions, activities, and parameters can be visualized in diagrams

Website: https://software.microfocus.com/fr-ca/software/uft

License: From $3,200 per year.

4. TestComplete

TestComplete continues to be in the list this year for its powerful and comprehensive set of features for Web, mobile, and desktop application testing. Testers can use JavaScript, VBScript, Python, or C++Script to write test scripts.

The TestComplete version 14.4 was released in April 2020 includes improvement in its integration with Jira — creating issues of any type on a user’s Jira project. Cross-platform web testing allows users to record/create manually web tests in the supported browsers and run these tests in a wide range of other browsers. Self-healing functionality that leverages the improved AI-based algorithms to find a replacement object by its image. Testers can easily use TestComplete’s record and playback feature, like Katalon Studio. They can insert checkpoints into test steps to verify results. As a product of SmartBear, TestComplete can be integrated easily with other products offered by SmartBear.

Website: https://smartbear.com/product/testcomplete/overview/

License: From $5,803 per year

5. SoapUI

The tool provides a quite comprehensive feature set for API testing with many advanced capabilities, including:

Generating tests easily using drag and drop, point-and-click
Powerful data-driven testing with data from files and databases
Asynchronous testing
Scripts can be reused easily
Creating mock services with RESTful mocking

Moreover, SoapUI has recently brought the API Explorer feature from SoapUI Pro into the open-source version, allowing developers and testers to instantly debug their API’s responses. The latest version 5.5 of SoapUI (Open-source) was released in February 2019 which added the Endpoint Explorer dialog to help users send exploratory requests and analyze responses without creating a project. Whereas, version 3.1.0 (Pro) allows users to use external data sources to quickly react if users want to simulate different user behavior without recreating their test case.

Website: https://www.soapui.org/

License: Free - $709 per year.

Please note that SoapUI and Postman are two representatives of several top API testing tools in this list only. To learn more about the comparison between these tools, please refer to this post.

6. IBM Rational Functional Tester (RFT)

IBM Rational Functional Tester is a test automation tool designed for testing of applications that are developed using different languages and technologies such as Web, .Net, Java, Visual Basic, Siebel, SAP, PowerBuilder, Adobe Flex, and Dojo Toolkit. It is also a data-driven testing platform for functional and regression testing.

Some highlights of the tool:

Visual editing through screenshots provides a visual, storyboard format for representing test actions that allow users to accommodate frequent user interface changes and avoid increases in maintenance overhead.
Advanced ScriptAssure technology
Earlier data detection
Test scripting allows users to choose between either Java or Visual Basic .NET
Integration with other software to support collaborative application lifecycle management

RFT provides a feature called ‘storyboard testing’ which helps visualize and edit tests using natural language and application screenshots. RFT’s ScriptAssure capability allows testers to generate test scripts resilient to changes in the user interface of the AUT. RFT can also integrate with other IBM’s application lifecycle management tools, such as IBM Rational Team Concert and Rational Quality Manager.

Website: https://en.wikipedia.org/wiki/Rational_Functional_Tester

License: Commercial — Contact Sales

7. Tricentis Tosca

Some other highlights of the tool:

Can be easily integrated to be a crucial part of DevOps processes
API tests can be performed across browsers, mobile devices, and platforms
Multiple protocols and standards are enabled, including HTTP(s) JMS, AMQP, Rabbit MQ, TIBCO EMS, SOAP, REST, and IBM MQ
A good set of test reporting and analytics capabilities

Tricentis Tosca supports API security configuration in the API Connection Manager. It also allows users to use the signature security option to sign multiple parts of a message.

Tricentis claims to significantly reduce regression testing (within the magnitude of minutes from weeks). This claim, however, needs to be carefully verified by testing teams.

In the latest version 13.1, released in March 2020, some features and integration are no longer supported, such as Mobile Exploratory Testing, Flood integration, Android SDK, etc.

Website: https://www.tricentis.com/

License: Commercial — Contact Sales

8. Ranorex

Like Katalon Studio, Ranorex makes automation testing easy for testers with its friendly and intuitive GUI, record/playback, and script generation.

Testers can integrate Ranorex with Selenium Grid to enable distributed testing along with parallel test executions.

Ranorex 9.3 introduces enhanced Jira and TestRail reporting. The newest release, Ranorex Studio 9.3.1 (March 2020) offers the Ranorex Instrumentation Wizard to resign iOS IPA packages that were signed with code signing v2.0.

Website: https://www.ranorex.com/

License: From €2,290 for Premium License.

9. Postman

Some highlights of the tool:

Comprehensive feature set for designing, debugging, testing, documenting, and publishing APIs
Friendly and easy-to-use user interface
Supporting both automated and exploratory testing
Accepting Swagger and RAML API formats
Requests and respondents can be packaged and shared with team members

In the released version 7.2 in June 2019, Postman extended their support for GraphQL request and schemas, GraphQL query auto-completion function, and GraphQL variables. The latest version 7.21.2 for Mac, Windows, and Linux App has been released in April 2020 with minor bug fixes.

Website: https://www.getpostman.com/

License: Free — $216 per user per year

10. Apache JMeter

JMeter is an open-source tool designed for test loading and performance measurement — two features of which JMeter is known. However, the tool is now also used for API and services testing, especially for API performance. JMeter is the third most popular tool for test automation, cited by 25% of respondents in the Test Automation Challenges survey .

Highlights of the tool include:

Lightweight with a simple and easy-to-use user interface
Test results can be replayed
Support CSV files to set values for API parameters
Support integration with CI tools such as Jenkins. JMeter is often used as a part of CI and DevOps toolchains

The most release JMeter 5.2 in November 2019 has been stacked with multiple new features, enhancements, and many bug fixes, such as new protocol, JMESPath extractor, JDBC improvements, StringtoFile, HTTP Samplers. The latest version 5.2.1 has minor bugs fixed only.

Website: https://jmeter.apache.org/

License: Open-source

— — — — —

This list of top tools here is not exhaustive; but it represents the best tools that are mature, popular, and providing capabilities using AI/ML to address the challenges that organizations are now facing when it comes to delivering “Quality at Speed.” This list also includes the tools that solely support API and services testing essential for successful Agile and DevOps transformation.

It can be said that the software testing tools landscape in 2020 continues to evolve. The emerging trends — AI and machine learning, codeless, big data and IoT testing, the intensive adoption of test automation (especially in Agile teams), etc. — will create both challenges and opportunities for the existing tools and new players to assert value to the testing communities.

References:

Top Software Test Automation Software of 2020 — Gartner Peer Insights
100+ Best Software Testing Tools — QASymphony’s research
Top 10 API Testing Tools (Details & Updates Done for You!)