The primary responsibility of the test lead or the test analyst is to come up with ‘Test case design & Planning’ in order to attain maximum test coverage. As it is nearly impossible to achieve 100% test coverage (unless and until your product has very simple requirements & implementation), the test team should come up with test cases/test suites that can help in testing the core functionalities of the product, thereby achieving the best coverage results. Along with test coverage, another Key Performance Indicator (KPI) that is critical as far as software testing is concerned is ‘Defect Yield Rate’. Higher the test coverage and defect yield rate, better the quality of the product.
Some things are easier said than done and the same can be said about ‘Website Testing’. Due to factors like reduced Time To Market, growing business competition, tight deadlines, and other varying factors; the development and test teams are always under intense pressure to maximize their productivity (thereby improving the Return on Investment [ROI]). For the development team, the criteria could be LOC (Lines of Code) pushed to the project/product repository; for the test team, the criteria could be number of tests executed/bugs raised/number of tests developed.
Browser compatibility testing as we know is practice where the permutations and combinations could turn out to be overwhelming. It is important to invest your efforts in planning & executing the right tasks. In order to achieve the same, your test team should make use of a smart testing approach i.e design, plan & execute test cases that can help in achieving the best results. ‘Pairwise Testing’ is one such testing methodology that can be used to get the best test coverage by writing effective test cases.
Here, I will be portraying a pairwise testing tutorial, I will be doing so by demonstrating a pairwise testing example. We will also look on how to leverage pairwise testing algorithm for performing cross browser testing.
Test your native, hybrid, and web apps across all legacy and latest mobile operating systems on the most powerful Android emulator online.
Introduction To Pairwise Testing
Pairwise Testing, also called as all-pairs testing is a black-box test design technique that can deliver nearly 100% test coverage. As per the ISTQB (International Software Testing Qualifications Board), the official definition of Pairwise Testing is as mentioned below
“Pairwise Testing is a black box test design technique in which test cases are designed to execute all possible discrete combinations of each pair of input parameters.”
In a complex project, the output for majority of the test cases may not depend on one single parameter. There could be multiple factors like state transitions/state machines, input parameters, shared parameters, variable factors like accessible memory/user preferences, etc. depending on the type & domain of the project. Though there are methodologies like Boundary Value Analysis & Equivalence Partitioning to derive values of these variable factors; their usage can only be limited to arrive at the ‘probable values’ that these factors can have.
Though you may come up with the possible values for these variable factors, come up with test cases/test suites that can cover all the combinations can be a herculean task. Hence, it becomes critical to come up with a ‘subset of combinations’ which when inputted to the test cases result in achieving the best results i.e. maximum test coverage.
The lingering question could be ‘How do you give precedence to certain test combinations over others?’ and what if you do not test some input combinations that may have affected the overall test coverage. This is where ‘Pairwise Testing’ or ‘All-pairs Testing’ can be used for coming up with effective tests that have multiple input parameters. The mechanism used by pairwise testing to come up with discrete test combinations will be discussed in subsequent sections.
Mobile emulators online from LambdaTest allows you to seamlessly test your mobile applications, websites,and web apps on mobile browsers and mobile devices.
Pairwise Testing – An Example
Pairwise testing should be used in scenarios where the application/web-app being developed has huge number of entries and large test configurations. For instance, if you are working on a website/web-application that can be downloaded & used by consumers from across the globe, you need to make sure that it works seamlessly on different browsers/devices/operating systems. Your job is to devise a foolproof cross browser testing strategy so that the application is tested on different combinations that yield best possible test results.
In order to come up with different test combinations for browser compatibility testing using the pairwise testing algorithm, techniques like ‘Orthogonal Array’ and tools that make use of ‘Classification Tree’ concept can be used.
Before we had a look at a ‘cross browser testing’ use case; to start with, let us have a look at a rather simpler pairwise testing example.
Pairwise Testing Example – Problem Statement (Mobile E-Commerce O2O Application)
- For a pairwise testing example we will consider an e-commerce store (or a store which is an integration of O2O i.e. Online to Offline ) which caters to ‘buying & selling’ of mobile phones. Consumers can ‘Buy’ new/old phones from the online medium or from the physical outlet/store.
- Apart from buying phones of the brand (Samsung, One Plus, Oppo), customers can also sell their old phones online or by visiting their store.
- An online store is open 365/7 whereas the store is open only during Business working Hours (9 AM ~ 8 PM).
- Customers should have multiple touch points in order to have a look & feel of the product (primarily phones from Samsung, One Plus, Oppo) and there should be facility to ‘buy/sell’ old/new phones. Due to business reasons, the store (online, as well as offline) is only operational in cities of Hyderabad & Mumbai only.
Once we have the requirements, it is time to have a look at the different parameters (fixed & variable) involved in development & testing of the application
Variables & Operations Involved In The Process Of Pairwise Testing Example
- Transaction category – Buy and Sell (2 combinations)
- Model category – Samsung, One Plus, Oppo (3 combinations)
- Location category – Mumbai, Hyderabad (2 combinations)
- Booking type/category – In-store, online (2 combinations)
- Timings – Working (9 AM ~ 8 PM), Non-Working (365/7, only in case of online category) (2 combinations)
- IMEI Validity – Valid (phones with Indian IMEI for only Samsung, Oppo, One Plus phones), Invalid (international unlocked phones) (‘N’ combinations for phones with valid IMEI, let’s assume ‘N’ = 1000)
Now, that we have the details about the requirements. Can you imagine the number of possible combinations.
(2 * 3 * 2 * 2 * 2 * 1000) = (48,000) test combinations
In order to cater to these many combinations, the test team might need to invest significant amount of time required for test case development & testing. If we look closely, IMEI can be categorized in a simplistic manner and classified in only ‘two main categories’ (valid and invalid) i.e. 2 combinations.
We again have a look at the number of valid combinations.
(2 * 3 * 2 * 2 * 2 * 2) = 96 combinations
To arrive at the best possible test combinations, we make use of the ‘Orthogonal Array’ technique, using which we get pre-defined arrays with respect to combination for every generation of test case. To start with, the number of columns in the array is normally number of entries + one (One extra column is added for serial number).
In the application which we are planning to test, our columns would be Model, Transaction, IMEI, Location, booking type, timings.
| ORDER CATEGORY | MODEL | LOCATION | IMEI VALADITY | BOOKING TYPE | TIMINGS |
|---|---|---|---|---|---|
| Buy | Samsung | Hyderabad | Valid | In Store | Working |
| Sell | One Plus | Mumbai | Invalid | Online | Non-Working |
| – | Oppo | – | – | – | – |
The number of entries that each column can have is below
| ORDER CATEGORY | MODEL | LOCATION | IMEI VALADITY | BOOKING TYPE | TIMINGS |
|---|---|---|---|---|---|
| 2 | 3 | 2 | 2 | 2 | 2 |
Before we generate the combination of test cases, we have to make sure that the column which has the maximum number of values (in our case, it is Model) should be the first column. The orthogonal array for pairwise testing example of ‘Mobile e-store O2O Application’ would look like below
| MODEL | ORDER CATEGORY | LOCATION | IMEI VALADITY | BOOKING TYPE | TIMINGS |
|---|---|---|---|---|---|
| 3 | 2 | 2 | 2 | 2 | 2 |
| MODEL | ORDER CATEGORY | LOCATION | IMEI VALADITY | BOOKING TYPE | TIMINGS |
|---|---|---|---|---|---|
| Samsung | Buy | Hyderabad | Valid | In Store | Working |
| One Plus | Sell | Mumbai | Invalid | Online | Non-Working |
| Oppo | – | – | – | – | – |
Which are the most wanted automation testing tools in 2023 that have climbed the top of the ladder so far? Let's take a look.
As you can see from the above array, Phones can be purchased/sold via online/offline medium. Irrespective of the medium being used, only customers from Hyderabad, and Mumbai can use the application. Customers can access the online store at anytime and offline store is open only during ‘Business hours’. We now work on the different test combinations that are possible with the available data set.
To start with, fill all the columns starting with the one which can take the maximum number of entries i.e. in our case, it is ‘Model’ (3 entries). The order columns are updated in decreasing chronological order. The array would look something as shown below
| MODEL | ORDER CATEGORY | LOCATION | IMEI VALADITY | BOOKING TYPE | TIMINGS |
|---|---|---|---|---|---|
| Samsung | – | – | – | – | – |
| Samsung | – | – | – | – | – |
| – | – | – | – | – | – |
| – | – | – | – | – | – |
| One Plus | – | – | – | – | – |
| One Plus | – | – | – | – | – |
| – | – | – | – | – | – |
| – | – | – | – | – | – |
| Oppo | – | – | – | – | – |
| Oppo | – | – | – | – | – |
In order to come up with the different combinations for our pairwise testing example, we make a list of patterns that are possible for one particular model (e.g. Samsung). Phone of model Samsung can have ‘Buy/Sell’ option and that operation is only possible for cities like Mumbai & Hyderabad. Booking can be done ‘in store/online’. If the IMEI is valid, the necessary operation (Buy/Sell) can go through, else the operation is rejected.
Based on the analysis, below is the table derived from our requirements. As you can see, there are two entries for ‘Buy + Hyderabad’ and ‘Sell + Mumbai’. In order to have different combination of testcases, we have created a new combination under One Plus for ‘Buy + Mumbai’ & ‘Sell + Hyderabad’
| MODEL | ORDER CATEGORY | LOCATION | IMEI VALADITY | BOOKING TYPE | TIMINGS |
|---|---|---|---|---|---|
| Samsung | Buy | Hyderabad | Valid | In Store | Working |
| Samsung | Sell | Mumbai | Invalid | Online | Non-Working |
| – | – | – | – | – | – |
| – | – | – | – | – | – |
| One Plus | Buy | Mumbai | Valid | Online | Working |
| One Plus | Sell | Hyderabad | Invalid | In Store | Non-Working |
| – | – | – | – | – | – |
| – | – | – | – | – | – |
| Oppo | Buy | Hyderabad | Valid | Online | Working |
| Oppo | Sell | Mumbai | Invalid | In Store | Non-Working |
By looking at the entries, we observe that is not a fair distribution for ‘IMEI Validity’ columns & ‘Order Category’ columns i.e. there is no entry for ‘Buy + Invalid’ and ‘Sell + Valid’. Hence, we perform a swap operation for the last 2 entries in the ‘IMEI Validity’.
| MODEL | ORDER CATEGORY | LOCATION | IMEI VALADITY | BOOKING TYPE | TIMINGS |
|---|---|---|---|---|---|
| Samsung | Buy | Hyderabad | Valid | In Store | Working |
| Samsung | Sell | Mumbai | Invalid | Online | Non-Working |
| – | – | – | – | – | – |
| – | – | – | – | – | – |
| One Plus | Buy | Mumbai | Valid | Online | Working |
| One Plus | Sell | Hyderabad | Invalid | In Store | Non-Working |
| – | – | – | – | – | – |
| – | – | – | – | – | – |
| Oppo | Buy | Hyderabad | Invalid | Online | Working |
| Oppo | Sell | Mumbai | Valid | In Store | Non-Working |
A potential combination could be when a particular product (irrespective of the brand) is brought during the ‘non-working hours’. We do not bother about the manner in which the product was purchased i.e. online/in store. Hence, we add two more test cases where all the column entries are ‘NA’ except for ‘Timings’. With only two values for Timings, we add another two test cases. The final test array has 8 entries and is shown below
| MODEL | ORDER CATEGORY | LOCATION | IMEI VALADITY | BOOKING TYPE | TIMINGS |
|---|---|---|---|---|---|
| Samsung | Buy | Hyderabad | Valid | In Store | Working |
| Samsung | Sell | Mumbai | Invalid | Online | Non-Working |
| – | Buy | NA | NA | NA | Non-Working |
| – | – | – | – | – | – |
| One Plus | Buy | Mumbai | Valid | Online | Working |
| One Plus | Sell | Hyderabad | Invalid | In Store | Non-Working |
| – | Sell | NA | NA | NA | Working |
| – | – | – | – | – | – |
| Oppo | Buy | Hyderabad | Invalid | Online | Working |
| Oppo | Sell | Mumbai | Valid | In Store | Non-Working |
Automate Cypress testing and perform browser automation testing with LambdaTest. Our cloud infrastructure has 3000+ desktop & mobile environments.
As shown in the final computation table, we have narrowed down from 96 test combinations to only 8 combinations. This would reduce the overall complexity involved in execution of these test combinations. The combinations that are generated for pairwise testing might differ from one scenario to another.
You could even make use of pairwise testing online tools to create different test combinations. For the purpose of verification, we use this pairwise testing tool. The input values that were provided to the tool are shown below
Once you have keyed in the input values for your pairwise testing example, you have the option to generate ‘All combinations’ (by clicking ‘Generate All Combinations’, as well as ‘Pairwise combinations’ (by clicking ‘Generate Pairwise’). As expected, when we clicked on ‘Generate All combinations’, it generated 96 different
combinations.
When we clicked ‘Generate Pairwise’, there were 9 different combinations generated, most of them are inline with the combinations that were generated earlier using the ‘Orthogonal Array’ technique. Here is the permalink to the test combination generation. Below is the snapshot of the combinations created by the pairwise testing tool.
| S No. | MODEL | ORDER CATEGORY | LOCATION | IMEI VALADITY | BOOKING TYPE | TIMINGS |
|---|---|---|---|---|---|---|
| 1 | Samsung | Buy | Hyderabad | Valid | Instore | Working |
| 2 | Samsung | Sell | Mumbai | Invalid | Online | Non-Working |
| 3 | Samsung | – | – | – | – | – |
| 4 | One Plus | Sell | – | Valid | Online | – |
| 5 | One Plus | – | Hyderabad | Invalid | – | Working |
| 6 | One Plus | Buy | Mumbai | – | Instore | Non-Working |
| 7 | Oppo | – | Valid | – | – | Non-Working |
| 8 | Oppo | Buy | – | Invalid | Instore | – |
| 9 | Oppo | Sell | Hyderabad | – | Online | Working |
Now that you have a fair idea about the usage of pairwise testing with example mentioned above, let us have a brief look at the scenarios when using it would be highly beneficial.
Note- Temporal new date-time API provides a common interface to the existing time and date representations of all modern browsers.
Pairwise Testing – When To Use It?
Though Pairwise testing can be highly effective in locating issues/bugs in the product, the test analyst or test lead needs to decide whether it should be used for testing their product. Pairwise testing is based on the ‘coupling effect’, a research which was conducted by Jefferson Offutt in the year 1992. As per this finding, major complicated faults are implicitly tested if the product is explicitly tested for simple faults.
This means that if there is a fault that occurs due to settings applied to some configuration variables, there is a high possibility that the fault could be the setting of a ‘smaller subset of configuration variables’.
Combinational data generation can be used to extend the capabilities of pairwise testing to triples, quartets, etc. In a nutshell, if the configuration variables/test combinations (data set) included in the pairwise test plan are ‘ideal for testing’ (i.e. testing that combination indirectly means that many other combinations have been tested), you can achieve the best results from this test approach. Let’s have a look at some of the major advantages of Pairwise testing.
- It can be used in any type & scale of project, irrespective of the domain being used i.e. medical, agriculture, technology, etc.
- As pairwise testing is based on the principle that majority of the bugs are triggered through an input parameter or due to an interaction between a pair of parameters, there is a significant reduction in the number of test cases being executed. The test cases for pairwise testing can be instrumental in attaining higher test coverage with better quality assurance.
- There is a significant reduction in the effort involved in designing test cases. As the focus is on the test cases that have the maximum impact, teams can achieve better test coverage with less efforts.
Though pairwise testing or all-pairs testing has lot of advantages, there are certain limitations as well. Some of the limitations are below:
- Pairwise testing might not be effective if the interaction between the configuration variables is not understood while coming up with the test cases.
- As the test combinations are derived using techniques like Orthogonal Array, there are possibilities where the most important inputs are not given proper consideration during test case design & development. In case of cross browser testing; ‘browser version’ is an important parameter and testing may falter in case certain browser versions are not tested to the full potential.
- The ROI (Return on Investment) of Pairwise testing may be less in case the input values that are used for testing are incorrect. Though boundary value testing is important to achieve maximum test coverage, it should not happen that testing is performed with incorrect input values. Take the case of testing your web application/website on different devices. Having an input combination of Linux & Internet Explorer (IE) may not fetch any results and should be ignored. If there are more test cases that focus on such category of inputs, lesser would be the advantage of using pairwise testing.
Hence, it is recommended to perform a feasibility test before opting for pairwise testing. If there are other projects in your organization where pairwise testing has been used, test leads should have a look at the learning (wrt pairwise testing) and other aspects related to testing.
This Cypress automation testing tutorial will help you learn the benefits of Cypress automation, and how to install Cypress and execute Cypress automation testing over scores of browsers and operating systems online.
Using Pairwise Testing To Speed Up Cross Browser Testing
As mentioned earlier, software products have to be tested for ‘cross-browser + cross-platform + cross-device’ compatibility before they are released to the end customers. There could be a possibility that your target customers might be using ‘old version of Chrome browser on Windows 7’ or ‘latest version of Firefox on Ubuntu’, etc.
If the product is not verified for ‘overall compatibility’, it could result in average customer experience as there could be issues in UI or in functionality. For example, the product explainer video is located on YouTube, but the target browser does not have ‘Adobe Flash support’. Hence, it is important that the product goes under cross browser testing against browsers/operating systems/devices that are used by your ‘target customer/audience’. Maintaining an in-house infrastructure to accommodate different test devices having multiple versions of browsers is impractical & non-scalable. This is where LambdaTest as a cross browser testing tool on the cloud can be beneficial for testing. You can perform Automation testing using their online Selenium grid, UI regression testing, Responsive testing, live interaction testing, and so much more on the LambdaTest platform.
The section where we discussed about the ‘Pros & Cons of Pairwise testing’ clearly mentioned about the importance of focusing on the test combinations that can give the maximum ROI as far as pairwise testing is concerned. There are scenarios where you have to perform cross-browser testing of your product and you focus your effort on ‘test combinations that are invalid’ or ‘ignore combinations that matter the most’. Create a browser compatibility matrix to keep your priorities in check. Also, use a web analytics tool to track browsers which bring your the most traffic.
Let’s consider the previous pairwise testing example of ‘Mobile e-store O2O Application’ where your test team has to verify the basic functionalities (customer signup, sign-in, product search, and product checkout) of the online store on different browsers (Firefox, Chrome, Internet Explorer, Safari, etc.), Operating systems (iOS, macOS, Windows, Ubuntu, Android, etc.). To keep the scenario less complex, we have not introduced the ‘browser versions’/‘OS versions’/device type, etc.
Below is the input table that consists the requirements. As we would use the orthogonal array technique for generation of test combinations, ‘Operating System’ would be the first column in the array.
| Operating System | Browser | Functionality being tested |
|---|---|---|
| Android | Internet Explorer (IE) | Sign-up |
| macOS | Mozilla Firefox | Sign-in |
| iOS | Google Chrome | Product Search |
| Windows | Safari | Product Checkout |
| Ubuntu | – | – |
You need to apply the same logic which was explained in the section to generate test combinations for pairwise testing. Below is the list of the test combinations to be used for cross browser testing based on our pairwise testing example.
| S No. | Operating System | Browser | Functionality being tested |
|---|---|---|---|
| 1 | Android | Internet Explorer (IE) | Sign up |
| 2 | Android | Mozilla Firefox | Sign-in |
| 3 | Android | Google Chrome | Product Search |
| 4 | Android | Safari | Product Checkout |
| 5 | Android | – | – |
| 6 | Mac OS | Mozilla Firefox | Product Search |
| 7 | Mac OS | Google Chrome | Product Checkout |
| 8 | Mac OS | Safari | – |
| 9 | Mac OS | – | Signup |
| 10 | Mac OS | Internet Explorer (IE) | Sign-in |
| 11 | iOS | Google Chrome | – |
| 12 | iOS | Safari | Signup |
| 13 | iOS | – | Sign-in |
| 14 | iOS | Internet Explorer (IE) | Product Search |
| 15 | iOS | Mozilla Firefox | Product Checkout |
| 16 | Windows | Safari | Sign-in |
| 17 | Windows | – | Product Search |
| 18 | Windows | Internet Explorer (IE) | Product Checkout |
| 19 | Windows | Mozilla Firefox | – |
| 20 | Windows | Google Chrome | Signup |
| 21 | UBuntu | – | Product Checkout |
| 22 | UBuntu | Internet Explorer (IE) | – |
| 23 | UBuntu | Mozilla Firefox | Signup |
| 24 | Ubuntu | Google Chrome | Sign-in |
| 25 | Ubuntu | Safari | Product Search |
As seen in the combinations above, there are certain cases which can be considered as ‘false-positives’ e.g. Internet Explorer (IE) is not present on iOS & macOS operating systems. Hence, these combinations have to be ignored even though it is a part of the test array.
Complete list of test combinations for the above use case can be found at here. As a test team or test lead, your job does not end at generation of the test combinations, but to carefully allocate ‘adequate resources’ (manpower & time) to verify combinations that are important for your project. If you focus on the wrong combinations, you may not be able to achieve adequate test coverage and your product may be released with bugs.
Once you have identified the necessary combinations, you can make use of cloud based cross browser testing provided by LambdaTest. Using LambdaTest Capabilities Generator, you can use automation test framework like Selenium with languages like C#, Python, Java, Ruby, PHP, Javascript.
The capability generator provides you with code for different combinations using parameters like Operating System/Browser type/Browser version/Resolution, etc, based upon your selections. As you already have a list of test combinations generated using the Orthogonal Array tech.
In case your team is not well equipped with automation techniques, they can make use of manual cross browser testing on LambdaTest. While planning these tests, it is important to keep a track of the test combinations generated for pairwise testing since the basis of any type of testing is the ‘test combinations’.
Note- Unhandledrejection event is fired when a Promise has been rejected, but because some other RejectionHandling code has since caught and dealt with the rejection. The rejectionhandled event will not be fired if the Promise was never handled at all.
Pairwise Testing tools
There are many tools available for generation of test combinations for pairwise testing. You can find the complete list of tools over here. It is necessary that you have a look at the product license and necessary terms & conditions before short listing any tool.
Run your Playwright test scripts instantly on 50+ browser and OS combinations using the LambdaTest cloud. Read more.
Conclusion
There are many testing methodologies that are used for testing different categories of products/projects. Pairwise testing is one approach that should be used in scenarios where you want to achieve the best test coverage with optimum efforts. Similar to any test approach, it is recommended that you have a look at case-studies/projects in your organization where pairwise testing was successfully deployed and fetched good results.
Combinations generated for pairwise testing example can be used to verify your product for cross browser testing by making use of LambdaTest. The core USP of pairwise testing is ‘achieving good test results in minimum time duration’ and using LambdaTest, you can double the benefits of pairwise testing.
Related Articles:
1. Automation Testing With Selenium, Cucumber & TestNG
2. Automated Cross Browser Testing With Protractor & Selenium
3. Cross Browser Automation Testing Using Watir
Top comments (0)