DEV Community: Kornelia O'Doherty

Career Overview of Software Automation Testers in 2019

Kornelia O'Doherty — Mon, 11 Feb 2019 10:33:53 +0000

In this article, I will show you why gaining test automation skills are needed for your quality assurance career if you want to remain marketable as a tester. I will tell you what technical skills and automation tools you need to learn about if you’re just getting started with test automation.

Why 2019 is a great time to be an automation tester?

For the past several years, QA and test analyst job ads have stressed the need for test automation skills. Four years ago, recruiters clearly advised testers to gain technical and test automation skills. Since then, the demand for test automation has continued to grow, and now some teams are completely eliminating manual testers and hiring only test professionals with test automation skills.

Today, more and more components of the lifecycle are being automated and there is a focus on automating the entire lifecycle. Increased number of applications and new releases have also led to a rise in the importance of automation. According to the World Quality Report 2018-2019, test automation continued to be the most popular approach for generating and executing functional test cases. By the end of 2018, 18% of functional test cases were generated using test automation tools, and 16% were executed using test automation tools. Similarly, 16% of all security tests were executed using automation tools, and automation was also applied to the execution of the 16% of all performance test cases. Quite encouragingly, 15% of all end-to-end business scenarios were also being executed using test automation tools.

Since test automation is one of the career fields experiencing the most growth in the world right now, it’s rare to see a job listing for a test role where there isn’t at least a demand for an understanding of test automation and at least one scripting language. At the time I’m writing this article, there are roughly 16,955 available jobs in Test Automation Engineering on Indeed, 30,705 jobs on Glassdoor, and 5,580 jobs posted on LinkedIn. These numbers indicate the high demand for this skill nowadays, and for that reason you shouldn’t worry much about the risk of career change. Keep in mind the fact that as an automation tester, you’ll always be needed and will find no lack of leading companies trying their hardest to recruit you, and there are constantly opportunities to grow in your career to reach a managerial level.

Moreover, if you want a high growth and a high paying career, test automation engineering/ automation testing is also the way to go. According to the report, the global test automation market was valued at approximately USD 15.87 billion in 2016 and is expected to reach approximately USD 54.98 billion by 2022, growing at a CAGR of 23.01% between 2017 and 2022. The national average test automation engineer salary is $105,630 in the United States.

What do you need to prepare for your automation testing career?

To excel in this field, testers must be aware of automation tools and modern techniques that are being used in the testing industry. This doesn’t mean you’re being asked to be programmers, but it does require automation testers to have a programming background. Programming skills can help testers communicate better with developers and participate in unit testing. As automation tests are closely connected with code, the knowledge of the programming language that the team works with is one of the critical requirements for a tester. If the product is written in Ruby, then the tester needs to have Ruby experience on their resume. Besides definite programming languages such as Java, C/C++, Ruby, Python, Perl, some other required skills include knowing SQL and relational databases, experience with HTTP/HTTPS, HTML, CSS, JS, XML and testing APIs, knowing one of the frameworks (i.e., Cucumber for Ruby) is definitely an advantage.

You can initiate your test automation path by referring some popular software testing blogs and forums like Utest, Software Testing Help, Stickyminds, Ministry of Testing, etc. These sites help educate and inspire teams of all types and sizes to test automation.

As I mentioned, the more technical and test automation skills you have, the more marketable you’ll be as a tester. However, anyone who is serious about a career in testing would do well to pick up a powerful automation tool. Most available test automation tools in the market are developed with a programming mindset. Some of the most popular tools:

Selenium
Katalon Studio
UFT
TestComplete
SoapUI

Learn more about the above-mentioned tools here.

Among these tools, I have found Katalon Studio to be the most user friendly automated software testing tool. You can easily automate web services and mobile testing and execute the scripts on your desired platform. The most interesting features is the spy utility, which captures the web and mobile objects with just one click. Katalon Studio is intelligent enough to capture your element with all possible locators. If you just started learning programming for automation testing, please find the simplest test scenario that needs to be repeated a thousand times in your project. Then create a simple automation script to run it and check the result of the test. These easy tests will motivate you to create more and more scripts and avoid manual procedures whenever possible.

Other ways to gain solid knowledge in test automation are joining local meetups and conferences such as STAREAST, STARWEST, Agile Testing Days, and QualityJam. Last month, I had a chance to join a webinar co-hosted by Katalon Studio and ToolsQA about the software testing trends of 2019. I found this webinar to be amazing and practica, and I would suggest you try to register for similar events.

More details are available at https://testautomationresources.com/software-testing-basics/test-automation-engineer-salary/

How to start with REST API & RESTful Web Service Testing

Kornelia O'Doherty — Mon, 10 Dec 2018 09:13:57 +0000

REST API & RESTful web services explained

REST API, which stands for representational state transfer application programming interface. Some of the simplest questions that might come up when you’re first starting to understand REST API is, what is being represented exactly, what is a state, and what is being transferred. So here in the article, let’s look at all these words individually and learn what they all mean.

The word ‘representational’ means there is a transfer of representations of resources and the resources can be pretty much anything that can be named on the Internet like a list of users or a list of photos, comments, posts, articles, pages, videos, books, profiles, etc. To understand how exactly we get a representation of resources, you need to look at how everyone interacts with web pages, the client-server model, and the HTTP. Here’s the HTTP protocol:

Everytime you type a URL in a browser, you’re sending a request to the web server and the web server responds with a resource. What’s important to understand here is, everytime you type a URL in a browser or click on a webpage link, you’re sending a request for a specific resource or resources from a web server. And the web server responses and delivers the request your resource to your browser as in a formal web page or whatever format that the resource is in. The second important point to understand is that web server doesn’t actually deliver the resources. It doesn’t send you the database that it has, but rather a representation of that resource in a format that is readable, for example, HTML or image. Think of actual resources, physical things located on a web server database or stored a web server hard drive, and representations of those resources as copies in either readable format for human beings like HTML or image or easy-to-work-with formats for programmers like JSON and XML. It’s also helpful to think of the whole web as a bunch of resources. We always request the resource when clicking on a link or type in a URL, no wonder that URL stands for Uniform Resource Locator.

So what’s about the word ‘state’? Take an example, whenever I click a link, the application state changes and I am presented with another resource, which is a page with all the other content of that resource. And as I click through all these different pages, the application state keep changing from one state to the next. When the resource gets transferred from the web server, we send a request for the resource and we get a representation back and whatever formats the resources presented. This is what is meant by the word ‘transfer’. However, it can also refer to the transfer of the application state when we click on the next link and get transferred to another page.

Implementing REST API & RESTful web services testing using tools

So far, I have been used many tools to handle REST API Testing, some of them are really amazing, including Postman, Katalon Studio, JMeter, Rest-Assured. If you want to automate API testing but do not yet have a testing tool and do not have any financial support, I highly recommend Katalon Studio since this tool is a completely free full package of powerful features.

In case you want to learn more, here is a step-by-step tutorial about how to test REST API & RESTful web services with Katalon Studio https://testautomationresources.com/api-testing/rest-api-restful-web-services-testing/

How to Start Writing Test Cases As A Beginner?

Kornelia O'Doherty — Thu, 29 Nov 2018 04:19:32 +0000

What is the concept of test cases?

Test cases are a set of conditional cases in which a tester can rely on to determine whether an application, a software system, or one of its features performs as desired or not.

A test case is a single executable test which a tester carries out. It guides them through the steps of the test. You can think of a test case as a set of step-by-step instructions to verify something behaves as it is required to behave.

Steps to design a test case

Step 1: Define the purposes of your testing process.

First, you need to understand the software requirements specification. When starting to write test cases for the features of a software, the first thing to do is to understand and determine the requirements of the system.

Step 2: Determine the performance indicators of your software system. (based on your understanding of the system)

In order to write a good test script, you need to be familiar with the functional requirements. You also need to know how the software used including various functionalities and organizational functions.

Step 3: Identify non-functional requirements.

The third step is to understand other aspects of the software related to non-functional requirements, such as hardware requirements, operating systems, security aspects. They need to be considered as other prerequisites aside with test data preparation.

Testing non-functional requirements is very important. For example, if the software requires a user to fill in forms, you must determine a reasonable timeframe so that your user will not run out of time while waiting for submission. At the same time, it is also important to check the log-in time to ensure that the user session is not expired, which is called security test.

Step 4: Determine the test case template

Test case samples, test case templates that should be chosen must include test types such as UI, functionality, fault tolerance, compatibility, and performance. Each type should be defined in accordance with the logic of the application.

Step 5: Indicate the interaction between the modules of the software

At this point, you have to understand the functionality of each module and its interactions with other modules to identify the system flow. Test cases should be designed to cover the highest level of module interactions. For example: When testing the shopping cart feature of an e-commerce website you need to test the inventory management system to validate if the same number of purchased products is deducted from the store. Similarly, when handling re-testing, you need to test its impact on the financial part of the application along with the inventory management system.

Structure of a test case

The format of a typical test case includes:

Test Case ID: The value needed to determine the number of instances required for testing.

Function: Based on the functionality of the system you can split the functions to create clearer test cases.

Test Data: The data that need to be prepared for testing.

Test Steps: Describe the test steps.

Expected results: Expected results from the steps above.

A result: Usually is PASS, FAIL, or PENDING. This is the actual result of the test case in the system environment.

Comments: This column is used to note screenshots and related information when executing test cases.

Besides, you can add more columns such as: Testers, Execute Date, etc.

Defining the test case

With each testing value there are always 3 scenarios that can happen:

Normal case: common test cases
Abnormal case: abnormal test cases
Boundary case: the boundary test cases. In larger cases, it will be divided into smaller cases correspondingly.

Read more for a practical example of writing test cases for the Facebook login function at https://testautomationresources.com/software-testing-basics/writing-test-cases/

How to automate your regression testing projects?

Kornelia O'Doherty — Thu, 22 Nov 2018 07:52:32 +0000

What is regression testing?

Regression testing, which is also known as repeated testing, is the process to ensure all the old functionalities still correctly works with the new changes. In other words, handling regression testing is to test an already tested application to find defects as a result of the changes. This is a usual step of any software development process and done by testing specialists. Testers do regression testing by re-executing the tests against the modified application to evaluate whether the revised code breaks anything which was working earlier. The only reason regression testing might not work is that changing or adding new code to a program can easily introduce errors into code that is not intended to be changed.
Regression testing is required in the following scenarios:

When the code is modified according to the change of need in the requirement
When new functionality is added
While fixing defects
While fixing performance issues

Why should regression testing be automated?

Let's consider an example and how to perform regression testing in that situation:
At the beginning of a project, assuming we have six developers and two testers. The Agile model is set up every two weeks for a Sprint. Everything started.
In the first Sprint, we start with the basic features (e.g., about ten functions), the testers begin designing testing scenarios for testing (e.g. about 100 scenarios). The very first Sprint receives a good rating from customers.
In the second sprint, the developers continue to create new features - about 10, and testers also do things like in the first Sprint - with 100 new scripts plus 100 old scenarios that need to be retested. Well, only 200 scripts, everything is still in control.
In the next Sprint, the developers need to make eight new features and update the two old features due to new customer requirements. At this point, the two testers not only have to design test scripts for the 8 new features but also have to test and update 200 old scenarios. The whole implementation is about 300 scenarios. Did you feel there is something wrong?
Over the next few Sprints, the three developers still meet the number of features and the changing requirements, but with two testers, the number of scripts to create and update are much more. Tiredness begins to spread. The lack of time and the risk of miss bug is higher and higher. Too many problems arise.
Therefore, when regression testing is automated, it enables testers to check into a variety of changes and unusual cases in the production environment. Not all regressions are caused by new features or the consequences of routine bug fixes; database updates or new browser versions can also cause them. Regression can also be an issue with efficiency and speed. Automating those cases which are stable and repeatable allows manual testers to spend more time testing various environments and for merging more complicated cases at a higher level.
And what's more, regression analysis is the key to success. It needs to deal with intelligence rather than hard work.

Highest Return: Execute tests that contribute to high coverage of the requirements, then any others…
Quickly Lower Risk: Execute tests for the most critical requirements, then any others…
Practically Safe: Execute tests for all the critical requirements, then any others… Especially since often ~20% of the test cases are covering ~80% of the business value

How to improve automated regression testing?

Two automated regression testing approaches are most used: data-driven testing and keyword-driven testing.
Data-driven testing is a framework where test input and output values are read from data files (data pools, ODBC sources, CSV files, Excel files, DAO objects, ADO objects, and such) and are loaded into variables in captured or manually coded scripts. In this approach, variables are used for both input values and output verification values. Navigation through the program, reading of the data files, and logging of test status and information are all coded in the test script. Let's consider a simple example of a data-driven script for setting up Admin accounts in a system:

private void Auto_ID_400_test()
{
     Common.InitializeWebDocument();
     DataInit.Init();
     BasePage.NavigateToHomePage();
     HomePage.NavigateToSettings();
     SettingBasePage.NavigateToBase();
     Base.AdminSection.AddNewAdmin(Data.Read("NewAdmin"));
}

public static void AddNewAdmin(DataTable dataTable)
{
     Report.Log(ReportLevel.Info, "Add new Admin:");
     OpenAddnewAdminPopUp();
     FillNewAdminData(dataTable);
     ConfirmAdmin(Read(dataTable.GetValue"NewAdminName"));
}

Keyword-driven testing is a technique that separates much of the programming work from the actual test steps so that the test steps can be developed earlier and can often be maintained with only minor updates, even when the application or testing needs change significantly. The keyword-driven testing methodology divides test creation into two stages: Planning Stage, Implementation Stage.
Example: Test case for login: Navigate to the login page, type a user ID, type the password, Verify login.
Then, automated testing engineers develop scripts for the keywords:

This strategy does not require programming skills, so it much easier for non-technical people to contribute to the regression testing process. By way of explanation, it is more suitable for the projects catering to a large audience and requiring a broader focus than the data-driven testing.

More details about this approach are available at https://testautomationresources.com/software-testing-basics/automated-regression-testing/

What you need to know about the levels of Software Testing

Kornelia O'Doherty — Wed, 14 Nov 2018 09:42:06 +0000

Before release, any software product goes through various levels of testing to make sure that it is working properly. Generally, software testing is divided into four levels.

1. Unit Testing

Unit testing (or component testing) is the first stage in the testing process. A unit is the smallest testable and independent component of the software. Normally, Unit testing is performed by software developers before handing software over to testers for executing other formal tests. The main purpose of unit testing is to make sure that every unit is working as designed.

In addition, as the cost of fixing bugs in the late stages is much higher than in the early stages, unit testing helps reduce cost by finding problems early. White box testing method is normally used at this stage

2. Integration Testing

After unit testing is integration testing. In this stage, all units are integrated together and tested. It is a form of testing in the testing process performed to detect defects in the interactions and the interfaces between the integrated units. Black box testing technique is usually used at this stage, but in many cases, both Black and White testing method are combined together.

There are four main integration testing approaches, namely:

Big bang: In the big bang approach, All components or modules are integrated together to complete a software system. After that, integration testing is performed.
Top-down: In this method, the process is carried out from the highest level modules to the lower level ones. This method requires Test stubs, Stub is a module which will temporarily substitute for submodules if they are not available for integration during the early phases.
Bottom-up: Bottom-up testing is a method where the lowest level components are tested first, and then progressively higher level components are tested. Drivers are required in this method to simulate the main modules which are not available in the early phases.
Sandwich: Sandwich testing (or Hybrid approach) is a type of integration testing which is the combination of Top-down and Bottom-up testing.

3. System Testing

After integration testing, the fully integrated application is tested to check that whether the system meets its software requirements specifications (SRS). System Testing is a type of black box testing method thus the knowledge of internal code is not required. This testing is performed in an environment that closely resembles the production environment in order to provide more reliable and efficient outcoming. It is an important phase as it helps to evaluate the functional, business and technical requirements.

There are various types of system testing:

Usability Testing: Focuses on the user’s ease to operate and use and checks that the user interface is user-friendly.
Load Testing: Load Testing is necessary to know that a software solution will perform under real-life loads.
Regression Testing: Type of software testing that checks the performance of existing functionalities when a change is made.
Functional Testing: Type of black box testing that bases its test cases on the specifications of the software component under test. Recovery Testing: to test that how well a system recovers from crashes, hardware failures, or other catastrophic problems.
Security Testing: To test that the system is secured enough to protect it from unintended users.
Performance Testing: Performance testing is a type of testing that is performed to determine how fast and stable the system performs under a particular workload.
Load Testing: Load testing is a kind of testing which determines the performance of the system under real-life load conditions. it measures the maximum load capacity of the system when multiple users access the system at the same time.

4. User Acceptance Testing

User acceptance testing (End-User Testing) is the last phase of the testing process. The user acceptance testing is performed by end users to make sure that it can handle required tasks in real-world scenarios as per specifications. If the software product passes this stage, it means that the end user has accepted it and it is ready to go live.

Below are the different types of user acceptance testing:

Alpha & Beta Testing: aim to validate the fitness-for-use of the system by the business user. The user acceptance test is performed by the users and application managers.
Contract Acceptance Testing: a developed software is tested against certain criteria and specifications which are predefined and agreed upon in a contract. The project team defines the relevant criteria and specifications for acceptance at the same time when the team agrees on the contract itself.
Regulation Acceptance Testing: also known as Compliance Acceptance Testing, examines whether the software complies with the regulations. This includes governmental and legal regulations.
Operational Acceptance Testing: also known as Operational Readiness Testing or Production Acceptance Testing, these test cases ensure there are workflows in place to allow the software or system to be used. This should include workflows for backup plans, user training, and various maintenance processes and security checks.

Black Box Testing: This is a software testing method that analyses certain functionalities of the application without letting testers see the internal code structure. Black Box Testing is part of User Acceptance Testing, because Black Box Tests share the same principles as UAT.

Factors affecting test scope: There are some factors that we need to consider before choosing the proper levels for your tests.

Project size.
The complexity of the project.
Budget
Time scope for the project.
Human resources.

5. The advantages of testing at different levels

Software development naturally split into phases.
Easily track bugs.
Ensures a working subsystem/ component/ library.
Software reuse more practical.

This write-up is originally posted at Test Automation Resources

Easier handling multiple browser testing with Katalon Studio

Kornelia O'Doherty — Tue, 13 Nov 2018 08:30:42 +0000

Introduction

This tutorial gives a brief information on how to run a test case in multiple browsers. Katalon makes easier to do grid option in automation testing without downloading any external resources. With Katalon multiple browser testing can be done with simple steps.

Requirements

Tool: Katalon Studio
Free download: https://www.katalon.com/download
Tutorials: https://www.katalon.com/resources-center/tutorials/

How to implement multiple browser testing using Katalon Studio?

Step 1:

To start Katalon Studio, double-click on the katalon.exe.

Step 2:

After launching Katalon Studio, provide your registered username and password to activate your Katalon Studio.

Step 3:

Once Katalon Studio is activated enter the Project name and its description.

Step 4:

Create a new test case and enter the name and description of the test case.

Step 5:

Click on the Record Web icon and continue with the created test case.

Step 6:

Enter the URL in the dialog box opened after clicking the Record Web icon in case need to pause or stop the recording option in between is available in the same dialog box which travels throughout the recording.

Step 7:

Select the browser you need to record the UI elements from the drop down.

Step 8:

Once the browser is selected browser will be initiated and the selected browser with URL will be opened once the page is completely loaded user can interact with the page and record the necessary action of the web page and the UI elements will be recorded whenever the action is performed by the user.

Step 9:

Once after the stop button is clicked the record will be finished the action, action data and the Element will be saved and displayed.

Step 10:

After clicking OK the Object Repository dialog will be opened and the user had to make a new folder and save the Object Repository.

Step 11:

Once the above step is completed the created object repository folder will be displayed under Object Repository. (Note: My Object Repository is named as GUI)

Step 12:

While clicking the Object Repository folder the captured element will be displayed.

Step 13:

To run the test case in multiple browsers right-click the test suite option.

Step 14:

Select the new option and create a new Test Suite and give an appropriate test suite name and description.

Step 15:

Now the user had to integrate the test case with the test suite. Click the add button once the test suite is created successfully will open the test suite page and click the add option.

Step 16:

Once the add button is pressed a dialog box will be opened to add the test case to the test suite. Now select the check box and click OK. The test case will be integrated into the test Suite.

Note: If there are multiple test case select the test cases according to the order.
My test case is named as GUI_Automation.

Step 17:

Select the new option and create a new Test Suite collection and give an appropriate test suite collection name and description.

Step 18:

Now the user must integrate the test suite with the test suite collection. Click the add button once the test suite collection is created successfully will open the test suite collection page and click the add option.

Step 19:

Now select the added test suite to the test suite collection to run the test case in multiple browsers.

Note: Add the same test suite for more than two times to run the test suite in different browsers.
I have name my test suite as GUI_Multiplebrowser.

Step 20:

Once the test suite is selected use the “Run with” option to select the browser to run the test suite in different browsers.

Step 21:

Once the test suite is selected with the appropriate browser the test suite collection is ready to execute.

Step 22:

The test suite collection can be executed in two different ways

1. Sequential: the test suites will be executed one after another.
2. Parallel: the test suites will be executed at the same time.

Step 23:

I will run my test suite collection in Parallel mode and will execute the test suite collection.

Step 24:

The test suite execution will run the test cases in three different browsers and log can be viewed in the log.

Full Source Code:

import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.checkpoint.CheckpointFactory as CheckpointFactory
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as MobileBuiltInKeywords
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testcase.TestCaseFactory as TestCaseFactory
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testdata.TestDataFactory as TestDataFactory
import com.kms.katalon.core.testobject.ObjectRepository as ObjectRepository
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WSBuiltInKeywords
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUiBuiltInKeywords
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import internal.GlobalVariable as GlobalVariable
WebUI.openBrowser('')
WebUI.navigateToUrl('http://www.practiceselenium.com/')
WebUI.click(findTestObject('Object Repository/GUI/Page_Welcome/span_See Collection'))
WebUI.click(findTestObject('Object Repository/GUI/Page_Menu/span_Check Out'))
WebUI.setText(findTestObject('Object Repository/GUI/Page_Check Out/input_email'), 'test@yopmail.com')
WebUI.setText(findTestObject('Object Repository/GUI/Page_Check Out/input_name'), 'Testing')
WebUI.click(findTestObject('Object Repository/GUI/Page_Check Out/div_Address'))
WebUI.setText(findTestObject('Object Repository/GUI/Page_Check Out/textarea_address'), 'New Testing , Katalon')
WebUI.selectOptionByValue(findTestObject('Object Repository/GUI/Page_Check Out/select_Visa              Maste'), 'American Express', true)
WebUI.setText(findTestObject('Object Repository/GUI/Page_Check Out/input_card_number'), '65656565656')
WebUI.setText(findTestObject('Object Repository/GUI/Page_Check Out/input_cardholder_name'), 'TESTING')
WebUI.setText(findTestObject('Object Repository/GUI/Page_Check Out/input_verification_code'), '6565656')
WebUI.click(findTestObject('Object Repository/GUI/Page_Check Out/button_Place Order'))
WebUI.closeBrowser()

The tutorial is orginally posted at Test Automation Resources

Introduction to Exhaustive Testing with Examples

Kornelia O'Doherty — Fri, 09 Nov 2018 04:52:42 +0000

What is Exhaustive Testing?

Exhaustive testing, which is also known as complete testing, occurs when all the testers in your team are exhausted and when all the planned tests have been executed. It is a quality assurance testing technique in which all scenarios or data is tested for testing. In a more understandable way, exhaustive testing means ensuring there are no undiscovered faults at the end of the test phase. Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. As testers, we often say “well I just never have enough time for testing”. Even if you had all the time in this world, you still wouldn’t have enough time to test all the possible input combinations and output combinations.

Strategies for exhaustive testing

We know that one of the 7 testing principles of software testing says that exhaustive testing is not possible. Now the question which lies is that: as exhaustive testing is not possible how we are supposed to get enough coverage to tell that the application under test is tending to not having any defects?

Instead of trying to test everything with what we tend to do and that we want to apply some helpful approaches, we use risk and priorities to focus testing efforts. We may do techniques such as pairwise testing to reduce the number of combinations, or things like cause-effect graphing and do more of a logic-based approach to optimize our testing. Here are basic and fundamental strategies that one can definitely include in their test plan:

Impact on Business: This not the only one but is one of the most important parameters to understand the risk. Testers should question that if a certain functionality of a module being tested fails how much risk it will bring to the entire module. Testers should constantly work with the Business Analyst of the module to understand the threat brought by the failure of each functionality so that it can be prioritized high up in the testing.

Probability of failure: This parameter also plays a major role in understanding the health of a functionality. Some functionality can have a high probability of failure compared to other functionalities, for example, let’s consider an e-commerce site which will have one functionality of logging in as a user and other to determine the users browsing history and correctly give the user more options to look at(cross sell/up sell). As a tester, we can identify that the second functionality has a higher probability of breaking because it contains a lot of business algorithms to be implemented compared to a login page which might have less probability of failure.

Regression: Testers should look at this parameter by identifying functionalities which have to be regressed more often when there is a change or addition to certain areas of the AUT. Correct impact analysis should be done to determine the areas correctly. Impact analysis should always involve developers, QA, and the Business Analyst team to get a correct understanding of the impacted areas.

Recovery: If a functionality breaks how fast can it be recovered or the recovery time taken. QA personnel can get the inputs from developers on the recovery time for each functionality of the application under test.

Why Exhaustive Testing is impractical and impossible?

It is not possible to perform complete testing or exhaustive testing. For most of the systems, it is near impossible because of the following reasons:

The domain of possible inputs of a program is too large to be completely used in testing a system. There are both valid inputs and invalid inputs.
The program may have a large number of states. There may be timing constraints on the inputs, that is, an input may be valid at a certain time and invalid at other times. An input value which is valid but is not properly timed is called an inopportune input. The input domain of a system can be very large to be completely used in testing a program.
The design issues may be too complex to completely test. The design may have included implicit design decisions and assumptions. For example, a programmer may use a global variable or a static variable to control program execution.
It may not be possible to create all possible execution environments of the system. This becomes more significant when the behavior of the software system depends on the real, outside world, such as weather, temperature, altitude, pressure, and so on.

Exhaustive Testing Examples

The IE Tools > Advanced Options window

53 binary conditions

1 condition with 3 options

1 condition with 4 options

2⁵³ = 9,007,199,254,740,992

x 12

= 108,086,391,056,891,904 possible combinations of conditions

At one second per test execution:

108,086,391,056,891,904 / 360 = 300,239,975,158,033.067 hours (12,509,998,964,918.04 days or 34,273,969,766.9 years) to test all possible combinations.

More details are available here for the deep learning about this technique.

7 Problems with Software Test Automation

Kornelia O'Doherty — Tue, 30 Oct 2018 07:10:43 +0000

There are lots of problems occurring in every test automation processes. It is much more challenging when you encounter them without preparations, so having some idea of the type of problem would help you deal with them. This blog post brings out some of the commonest problems described in details.

1. Unrealistic expectations

Most people in our industry believe new technical solutions will save the day. Testing tools are no exception. There is a tendency to be optimistic about what can be achieved with a new tool. Indeed, we cannot deny that tools nowadays can solve almost every problem we are currently facing in test automation. However, the effect of optimism also leads to unrealistic expectations. If management expectations are unrealistic, then no matter how well the tool is implemented from a technical point of view, it will not meet expectations.

2. Poor testing practices

Automation is not always a good idea if testing practices are poor, with poorly organized tests, inconsistent documentation, and tests that are not very good at finding defects. The first and the most important to mention is, improving the effectiveness of testing.

3. Expectations that automated tests will find new defects

A test is most likely find defects the first time it is run. If a test already ran and passed, re-running the same test case a thousand times still cannot yield any differences, unless the test is exercising code that has been changed or could be affected by a change made in a different part of the software, or is being run in a different environment.

Testing tools are useful things, but they are not those things that can find a large number of new defects. Or more correctly, they are regression testing tools. Their use is in repeating tests that have already run. Therefore, tests that do not find any defects are not worthless, even though good test design should be directed at trying to find defects.

4. False sense of security

Just because the test suites do not find any defects, doesn’t mean there is no defect in the software. This is very crucial because if the tests contain defect themselves, they will bring incorrect outcomes. The automation test will simply preserve those defective results indefinitely.

5. Maintenance of automated tests

Every time when the software is changed or updated, your tests must be updated either so that they could re-run successfully. Test maintenance has been the death of many test automation initiatives. When it takes more effort to update the tests than it would take to re-run them manually, test automation will be abandoned. Your test automation initiative should not fall victim to high maintenance costs.

6. Technical problems

Most popular testing tools are large and complex that require detailed technical knowledge. In addition to technical problems with the tool themselves, you may experience technical problems with the software you are trying to test. If the software is not designed and built with testability in mind, it can be very difficult to test, either manually or automatically. Trying to use tools to test such software is an added complication which can only make test automation even more difficult.

Nonetheless, as for tests automation, Katalon Studio is a great and free tool. The test automation framework provided within Katalon Studio was developed with the keyword-driven approach as the primary test authoring method with data-driven functionality for test execution.

7. Organizational problems

Test automation needs to be supported by management and implemented into the culture of your organization. Time must be allocated for choosing tools, for training, for experimenting and learning what works best. Test automation is an infrastructure issue, not just a project issue. In larger organizations, test automation can rarely be justified on the basis of a single project. Standards are needed to ensure consistent ways of using the tools throughout the organization. Otherwise, every group may develop different approaches to test automation, making it difficult to transfer or share automated tests and testers between groups.

Original source https://testautomationresources.com/software-testing-basics/problems-test-automation/

All You Need to Know About API Testing

Kornelia O'Doherty — Mon, 22 Oct 2018 10:02:55 +0000

What is an API?

If you have already known, or if you have googled about, or if you are a little confused about what it says by looking all the technical terms over the Internet, just keep everything inside and the focus on what we are talking. So, API is nothing but it's an application program interface. As we already said the name itself says it is an interface.

How does API Testing work?

It is all about requests and responses between the client and server. You want something, you request for something so there is a way it gets carried to the server and you get the response back from the server. In other words, it's all about your requesting and you know getting some responses from the provider.

Real world examples

Example 1: Let's compare it with a real-world example: You go to a restaurant you sit at your table and you choose that you need ABC. You will have your waiter coming up and taking a note of what you want. You tell him that you want ABC. So, you are requesting ABC, the waiter responds back with ABC he gets in the kitchen and serves you the food. In this case, who is your interface in between you and the kitchen is your waiter. It's his responsibility to carry the request from you to the kitchen, make sure it's getting done, and you know once it is ready he gets back to you as a response.

Example 2: Another important example that we can relate is travel booking systems. For instance, take Kayak the biggest online site for booking tickets. You enter your destination, once you select dates and click on search, what you get back are the results from different airlines. How is Kayak communicating with all these airlines? There must be some ways that these airlines are actually exposing some level of information to Kayak. That's all the talking, it's through API's

Example 3: Now open UBER and see. Once the site is loaded, it gives you an ability to log in or continue with Facebook and Google. In this case, Google and Facebook are also exposing some level of users' information. There is an agreement between UBER and Google/Facebook that has already happened. That's the reason it is letting you sign up with Google/ Facebook.

What is the use of API's?

If I expose a free API, it's not that I am exposing all the information that there is. I have my own criteria like what level of information that I should expose to my user or to the outside world. So I tie that information and I just expose it, now people use it. Tomorrow, they want some extra level of information, they comfort me and ask me for that. Why should I give it for free? That's when I charge based on the demand, I do make money out of my extra services.

Think about the development perspective, UBER has an ability to share location where it uses Google Map API there. UBER has to focus on what it is actually doing rather than already existing functionality. It can just plug and play. There is no reinvestment of time to develop the same functionality. That's how people expose some level of API's to the outside world and some of them will be charged based on how you are using it.

SOAP vs REST

SOAP has been used since many years ago but these days it's all about REST (Representational State Transfer). If you just go to any Job Portal and just click on 'REST API Automation Tester' or ''REST API Testing Jobs, there are tons out there.

REST is kind of API which is actually efficiently being used by many companies like eBay, Amazon, Google. When such big companies are using definitely they have already transitioned from SOAP to REST. It means that you know REST is in high demand now. There are few advantages from the development perspective, REST supports a number of data formats. The most important ones are the way the request is sent and how we read the response. There is a specific format SOAP always uses XML format whereas REST uses JASON. Now, because JASON is being used JavaScript object notation, it's a form of representation. If you click on a JASON format on Google, objects will be inverted commas followed by the parameters. So whatever, JASON is a way of our communication, a format which is being used for the information exchange such as request versus response.

More details about REST API Testing: 16 Common REST API Testing Interview Questions

What to test in API's?

There is a huge need for us to make sure that what we are requesting and what we are getting are matched. If we are requesting for some information, we have to make sure that the response is correct. That's where QA/API testers come into the picture. It is our responsibility to make sure that every request is getting proper responses.

Learn more about available API Testing Tools: SoapUI vs Postman, Katalon Studio: A Review of Top 3 API Tools

Common Types of Test Design Techniques

Kornelia O'Doherty — Fri, 12 Oct 2018 06:23:45 +0000

Test design techniques help every software development project improve its overall quality and effectiveness. There’s a multitude of software testing techniques in the automation landscape. Each of them has its own strengths and weaknesses. This blog post will give an overview of some of the most popular techniques, divided into categories. But first and foremost, I suggest you learn the overview of Test Design Techniques in Software Development before seeing each technique in detail.

Specification-based or Black-box technique
Use these techniques to determine the external factors—such as technical specifications, design, and customer’s requirements, etc.—of a software program. Testers view the software as a black box with inputs and outputs. The purpose is to validate the quality and correctness without disrupting the internal details.

- Equivalence Partitioning:
The idea of this approach is grouping the inputs with the same attributes to partitions. Code is not visible to testers. Your task is to pick one condition out of each partition, which covers all possible scenarios, to execute test cases. If a condition of a partition is valid, other conditions are valid too. Likewise, if a condition in a partition is invalid, other conditions are also invalid. This helps reduce the number of test cases.
- Boundary Value Analysis:
This is one of the software testing techniques in which test cases are designed to include values at the boundaries. If the input is within the boundary value, it is considered ‘Positive testing.’ If the input is outside of the boundary value, it is considered ‘Negative testing.’ The goal is to select test cases to execute boundary values. In other words, the behavior of Negative testing is more likely to be incorrect than the behavior of Positive testing; and boundaries are an area in which testing is more likely to yield defects.
- Decision Table Testing:
This technique can be used in test design because it helps testers explore the effects of combining different input values when adhering business rules. A Decision Table is a tabular representation of conditions versus test actions. Conditions are considered as inputs, while actions are considered as outputs.
- State Transition Diagrams:
Using this approach, the tester analyzes the behavior of an application under test (AUT) for different input conditions in a sequence. You can provide both positive and negative input test values and record the system behavior. Any system in which you get a different output for the same input is a finite state system.
*- Use Case Testing:
Use case testing is a functional testing technique, meaning programming skill is not required. It helps the tester determine which test scripts are executed on the entire system from the beginning to the end of each transaction.

Structure-based or White-Box techniques
This is a testing method in which the internal structure of applications is transparently seen and tested. The tester chooses inputs to exercise paths through the code and determines the appropriate outputs. Programming knowledge is required.

- Statement Coverage or Line Coverage:
In this technique, every statement in the source code is executed at least once. Thereby, we can check what the source code is and is not expected to do. However, we cannot test the false condition in the source code.
Statement coverage = (No. of statements Executed/Total no. of statements in the source code)100
*- Condition Coverage or Predicate Coverage:
Condition coverage is seen for Boolean expression. Condition coverage ensures whether all the Boolean expressions have been covered and evaluated to both TRUE and FALSE.
- Decision Coverage or Branch Coverage:
Test coverage criteria require enough test cases so that each condition in a decision takes on all possible outcomes at least once, and each point of entry to a program or subroutine is invoked at least once. That is, every branch (decision) is either true and false. It is helpful to invalidate all branches in the code to make sure that no branch leads to any abnormal behavior.
- Multiple Condition Coverage:
Every combination of ‘true’ or ‘false’ for the conditions related to a decision has to be tested in this technique.

Experience-based technique
These techniques are logically drawn from the experience in designing test cases and test conditions:

- Exploratory Testing:
Usually, this process is carried out by domain experts. They perform testing just by exploring the functionalities of the application without having the knowledge of the requirements. Testers can explore and learn the system while using these techniques. High severity bugs are found very quickly in this type of testing.
- Error Guessing:
Error guessing is one of the testing techniques used to find bugs in a software application based on the tester’s prior experience. In Error guessing, no specific rules are applied.

Conclusion
These are only three test design techniques besides hundreds to choose from. Each one serves best for specific types of problems in software. If we use one more than the others, our test coverage will suffer. Remember to choose the most suitable technique for your projects, not the most popularly used one.

Original source: https://testautomationresources.com/software-testing-basics/test-design-techniques-types/

Moving From Manual To Automation Testing at a Small Company

Kornelia O'Doherty — Mon, 01 Oct 2018 07:25:26 +0000

Obstacles of moving from manual to automation testing

Too busy running manual tests
Put automation off until the "next release."
Company culture is not open to change
Company is not sure about the investment
Not enough knowledge or experience to implement
Not sure how or where to start

Steps to go from manual to automation testing

Step 1: Build a good test case foundation

Why create a good test case foundation? Good manual test cases are blueprints for what you will and will not be able to - automate. Explore automation framework options
Aim for goodness, not perfection
Test cases will evolve and change as you dig in deeper and learn more
Pick a tool to store and organize your manual test case. This can be as simple as you need
Common test case storage options:

Option 1: Documentation only examples

Excel sheet (Microsoft, google docs)
Wiki page

Option 2: Software tool examples

Testlink - open source - https://testinglink.org/
Testrail - server and cloud offerings - https://www.gurock.com/testrail/

Manual test case organization considerations:

The option should include tracking which tests can/should be automated and what has been automated.
Are you planning on storing test results and/or test plans with your manual test cases?
Create a wish list (need to have versus nice to have)

Tips: you can start simple and then import your list into a more complex tool if needed or wanted

Step 2: Explore automation framework options

By framework, I mean what your tests are coded in that will run the actual tests. For example, the framework is what will actually send the test to the Selenium Server.
What if you don't know what framework are available? If you don't know, ask resources around you including: Developing team members, QA folks in professional organizations, QA/ Tech mailing lists or Forums, Slack groups, Twitter experts
Narrow down your list to about three options and create a matrix that compares each option. This could include building your own.

Framework options and examples:

If you are using Selenium, you have some great options: Java, Robot, Python, Ruby, Cucumber (and many more). This can be hosted or self-managed.
Many Selenium hosting providers give you "wizards" that will help you set up the framework environments that will work with the solution.

Step 3: Explore infrastructure options

What hardware is needed?

Servers: physical, self-managed virtual machines, cloud hosted
How will the hardware be installed? maintained?
Backup strategy What software is needed?
Buy versus Build versus Hosted
Versioning strategy for the manual and automated test cases Will the automated test cases exist in the same repository as the project code?

Step 4: Make the first choice:

After you have proved to yourself that your approach will work, time to get the team to buy in on your solution What is the quickest way to get to Proof of Concept (POC)?
Develop a good plan and get moving: The plan should include a REAL DATE for a POC. Many times you can take advantage of trial membership for some testing products and tools. If you need the trial extended to reach out to the company and just ask.
Have at least one backup plan in case the first choice does not work out.

Step 5: Start simple

What is the simplest way to implement your framework approach using your actual product? Selenium example: Can you use Selenium IDE scripts to confirm assumptions about product test-ability?

Step 6: Confirm your choice works

Go beyond the simple checks you have performed
Confirm that your choice will work in your specific environment
Many software providers will allow you to sign up for trial accounts so you can take a closer look.

Tips: there is no harm in asking for an extension on a trial account if you needed more time for analysis.

Step 7: POC for team buy-in

Proof of Concept (POC) is a chance to share your results, recommendations, and excitement with the large team.
Best to show a live demo if possible
Have dates and requirements for implementation
Helpful to have metrics or other supporting detail to exemplify why automation

Step 8: Implement

My automation implementation:

Git: QA repository containing test cases, test queues, and vagrant/AWS configuration
Jenkins: takes QA git repository and deploys on a Linux system based o config settings
Linux system: launches vagrant/AWS system and executes Robot scripts. Testing bot: robot scripts are executed against the Selenium server(s) and the internal vagrant/AWS system.
Slack: posts test results from the Linux system to the team. This is broken up between pass and failed test results with emojis.

Note: This video is originated from SeleniumConf UK
This article was originally posted at Test Automation Resources.

More resource: Software Test Automation Engineer Salary and Career Overview in 2019

Test Automation Framework Interview Questions and Answers

Kornelia O'Doherty — Tue, 25 Sep 2018 09:00:46 +0000

Introduction to automation testing framework

What is a framework?
A framework defines a set of rules or best practices which can be followed in a systematic way to achieve the desired results.

What are the advantages of using automation testing framework?

It saves time and money
Reusability of code
Easy reporting
Easy for compatibility testing
Low-cost maintenance
Automated testing is more reliable
Automated testing is more powerful and versatile
It is mostly used for regression testing
Minimal manual intervention
Maximum coverage

How to create any framework?

Beginner: No, I didn't get any chance to create a framework. I have used the framework which is already using the project.
Experienced: Yes, I have created a framework. (or) No, but I have involved in the creation of a framework.

Some popular Test Automation Frameworks?
In a test automation project, we do perform different tasks by using different types of files. To organize and manage all the files and to finish all the tasks in a systematic approach we use a framework.

Mention the name of the framework which you are currently using or which you have hands-on experience?
Already the organization which I am working for is using a hybrid framework.
(or)
I have experience with the hybrid framework, and we are using that.
(or)
It's easy to handle all my scripts to execute and generate logs and reports and also to capture screenshots by using the hybrid framework. So I have selected the hybrid framework and using it in my project.

Can you explain the framework which you used in your project?
Please check the link: How To Explain Selenium Test Automation Framework In The Interview

The uses of automation testing tools

What is Automation Testing?
Automation testing is the process of testing the software using an automation tool to find the defects.
In the process, executing the test scripts and generating the results are performed automatically by automation tools.
Some most popular tools to do automation testing are:

HP QTP
Selenium Webdriver

What are the most popular testing tools for functional testing?

Selenium
HP QTP/UFT

Why do you prefer Selenium Automation Tool?
It is a free and open-source tool
It has a large user base and helping communities
It supports cross-browser compatibility
It supports platform compatibility
It supports multiple programming languages

Handling automation test cases

What type of test cases do you pick up to automate?
I focus on the test cases which should be executed in a repetitive manner such as regression test cases, smoke test cases, and sanity test cases
Read more: Do We Really Write Test Cases For All Testing Types

What type of test cases you won't pick up to automate?* Before picking up the test case to automate, I do check whether the application is stable or not. So based on this,
I don't pick up test cases when the AUT changes frequently.
I don't pick up the test cases which I run rarely.
I don't pick up the test cases which I run only one time.
I don't automate when I do usability and exploratory testing.

How many test cases you have automated per day?
It depends on test case scenario complexity and length. I did automate 2 to 5 test scenarios per day when the complexity is limited. Sometimes just 1 or fewer test scenarios in a day when the complexity is high

How you build Object Repository in your project?
There is no default Object Repository concept in Selenium.
In Selenium, we call objects as locators such as ID, Name, Class Name. Tag Name, Link Text, Partial Link Text, XPath, and CSS.
Object Repository is a collection of objects. One of the ways to create Object Repository is to place all the locators in a separate file that is a properties file.
The best way is to use the Page Object Model. In the Page Object Model Design Pattern, each web page is represented as a class. All the objects related to a particular page of a web application are stored in a class.

Note: This video is contributed by Software Testing Materials

More resource: Software Test Automation Engineer Salary and Career Overview in 2019