DEV Community: Sourojit Das

How to set goals for Software Quality

Sourojit Das — Tue, 17 May 2022 13:47:09 +0000

Setting Goals Together

What is Software Quality and Why is it important?

The International Software Testing Qualifications Board (ISTQB) , one of the world's leading software certification bodies defines software quality as "The totality of functionality and features of a software product that bear on its ability to satisfy stated or implied needs", whereas the defines software quality as "The degree to which a system, component, or process meets customer or user needs or expectations."

To get a better idea of what these definitions mean in the context of setting SMART goals for software quality, it is necessary to understand how the lack of Quality has impacted some major organizations in the past, through the following Case Studies.

The Mariner 1 Spacecraft, 1962. NASA --- In 1962, NASA launched an unmanned space mission for data-gathering to fly past Venus. It barely made its way out of Cape Canaveral, before veering off course. With a dangerous crash-landing on their hands, NASA activated the self-destruct button. The subsequent investigation revealed that a very simple software error, a missing hyphen in a line of code, led to an estimated loss of 18 Million USD (as of 1962), let alone the loss of face.

EDS Child Support System, 2004, Government of the United Kingdom --- In 2004, the UK government introduced a new system to manage the operations of the Child Support Agency (CSA). A Texas-based company called Electronic Data Systems (EDS) was awarded the contract and called it CS2. A plethora of problems plagued it from the initial go-live with over 1,000 reported problems, of which 400 had no known workaround, resulting in around 3,000 IT incidents a week. An internal memo leaked later described the system as badly designed-tested-implemented. Not only did it lead to the UK government spending 768 million GBP instead of the estimated 450 million GBP, but the contractor also announced a fiscal loss of USD 153 million in the subsequent year.

Bitcoin Hack, Mt. Gox, 2011 --- Bacon in the 2010's Mt Gox was the largest bitcoin exchange in the world until a software glitch led to the creation of unredeemable transactions costing them USD 1.5 Million in lost bitcoins. This severe loss, in an already fluctuating market, led to the company declaring bankruptcy a few years later.

Considering the above three cases, a clear inference is that the software failed in its ability to satisfy stated or implied needs (as defined by ISTQB) or meet customer or user needs or expectations (as defined by IEEE). Thus comprehensively failing the spirit of both the accepted definitions of Software Quality.

Even though companies spend amounts of money, human resources, and time on QA processes still not being able to achieve the goals is clearly due to a lack of Software Quality.

Even the 12th edition of the World Quality Report 2020--21 lists " QA is a key business priority for organizations to achieve digital transformation", the Consortium for Information & Software Quality (CISQ) records in its 2020 report that poor quality software costs US companies over USD 2 trillion annually.

Diligent QA Leaders would always seek to ensure optimal software quality for their organizations and set quality goals in the software testing process to avoid such fatal consequences, and this article aims to aid them in their endeavor of Setting SMART goals for Software Quality.

Overview of Software Testing Objectives

Source

The major objectives of any software testing process are outlined in the diagram above. Any attempt to set quality goals for software development will need to take these objectives into account.

Prevent Defects: The most important objective of any Software Quality process is to prevent bugs in the software development process. Early defect detection reduces the costs and labor associated with fixing these defects. By conducting a root cause analysis of previous defects, we can also future-proof our solution against any future flaws.
Evaluate Work Product: Any goal of a software quality assurance process demands the creation of artifacts that help set in the context of the product being developed. The principles of agile development demand a shift left in terms of QA , i.e. identifying any ambiguity or contradictory requirements in the requirement document, design, and user stories before development begins to save significant time and cost in the future.
Verify Requirements: Meeting the needs of the client is the primary aim of the software development process. Any and all software testing goals gravitate around ensuring that the expected standards are met, and QA techniques are developed to ensure full compliance with both industry standards and user expectations.
Validate test objects: All test objects need to be validated in order to ensure that they meet the expectations set by the business requirements of the project. Both manual, as well as automation test procedures, can be implemented to ensure the validity of the test objects.
The bedrock of client confidence in a project is the Build Confidence:software QA process. Having a low number of or ideally no defects during user acceptance testing and presenting the clients with comprehensive test reports is a surefire way to build confidence in your product.
Reduce Risk: A major goal of any software quality process is to try out as many possible scenarios while testing the product, in the shortest possible time, to identify and root out any risks during deployment.
Share Information to stakeholders: An essential goal for any software quality process is to provide the relevant stakeholders with objective and accurate information to make an educated judgment of the quality of the tested product. Test Coverage reports and other software testing metrics can cover these bases.
Find Failure and Defects: The primary goal of any software QA process is to test the software as thoroughly as possible to uncover any bugs and ensure superlative software quality.

These software quality objectives provide a useful yardstick for the QA team to evaluate and modify any software testing goals based on their progress. These also enable us to understand the activities which lead to the fulfillment of these objectives to meet our software quality goals.

How to Set Goals for Software Quality

1. Understand what's most important, get buy-in from your stakeholders

Before we set any goals for software quality, it is important to understand what are the customer's primary objectives from the software test process. Depending on what the user wants, it is important to ask a variety of questions -

How long will the testing process take?
How much money does the test process require?
How much of the software can be tested through automation?
Has the test effort been adequate? Could more tests have been executed in the same time frame?

The primary goal for software quality management in software engineering should be to measure software performance across the entire user journey. That means considering the entire user journey from login to checkout. The users should be guaranteed the desired experience at every stage of the user journey.

This stage of the process helps to check the boxes for Building Confidence and Sharing Information with stakeholders from the objectives mentioned above.

2. Define a SMART Goal

Once the QA Leader has a roadmap for setting the software quality goals, they should set SMART goals to obtain clarity and motivation to complete actionable tasks. By having defined objectives and fixed deadlines, they are more likely to achieve success in what matters rather than just pluck low-hanging fruit.

The SMART acronym stands for -

Specific: Having a very specific goal. It is best to stick to one very objective aim for every action item
Measurable: It is important to make sure that the goal set is measurable in some objective terms to evaluate whether it has been achieved. This can be set in terms of code coverage or test coverage . There are also a plethora of software testing metrics that help in quantifying the software testing process.
Attainable: The goals created should be realistic. There shouldn't be any limitations on how big or small it can be, but it should be completed within the time frame determined.
Relevant: The goal should be important to both the QA team as well as the rest of the stakeholders. This is why the previous step is of utmost importance.
Time-Bound: It is best to assign an end date to the goal so that it doesn't get overtaken by the rush of day-to-day activities. Following an agile testing process helps define the workload into manageable sprints which help in fulfilling these objectives.

Pro Tip: Try Testing on Real Device Cloud for Free Continuous Testing helps the companies develop software that is customer-centric and simultaneously in line with healthy development practices. Bear in mind that the resultant code's accuracy can only be guaranteed when it is tested on a

" reduce the number of defects by 20% during the next quarter" --- assuming that the number of bugs reported during a previous quarter is known;
" keep re-work time below 25% of the total time spent on development in every sprint";
" increase internal testing effectiveness" --- targeting a defect detection rate (before vs. after delivery) higher than, let's say, 80% per release or major milestone.

real device cloud . BrowserStack's *3. Elucidate the activities required to reach the goal cloud Selenium grid *equips QAs with 3000+ real browsers and devices for automated testing so that every feature can be monitored in real-world circumstances.

Plans for training your staff on required automation suites or other processes that will improve the overall QA practice.
R&D activities to ensure the election of the best automation tool , and the adoption of best practices for your QA process.
Setting up a Continuous Testing process if possible. This ensures that all code is run through a series of tests from the smallest unit to entire components before they are merged to the main repository.

Some examples of such quality objectives are:

Reviews

An example of a bad quality objective could be something vague like "Reduce the number of bugs in every sprint", as it is not really achievable or time-bound!

Tests

Defining SMART goals helps in Evaluating Work Products, Building Confidence, and Sharing Information with stakeholders from the objectives mentioned above.

The activities required to achieve the set goals for software quality can be defined in 3 ways:

These include:

An appraisal is an objective evaluation of an artifact against a defined set of standards. It consists of -

These are periodic checks of the test process, the progress of test execution, and even the documents created during the software development lifecycle to ensure that the activities being performed on a daily basis are in sync with the user goals and customer requirements.

4. Determine the resources available to see if they are sufficient to match your goals

This is the backbone of any software quality management process. Tests must be performed often and as early as possible. Test automation and parallel testing are two ways to speed up the test process and ensure faster test coverage in minimal time. The entire QA process hinges on the use of a real device cloud . Without real device testing, it is not possible to identify every possible bug a user may encounter. Naturally, undetected bugs cannot be tracked, monitored, or resolved. Moreover, without procuring accurate information on bugs, QA metrics cannot be used to set baselines and measure success. This is true for manual testing and automation testing .

CoP is the cost of prevention resources and activities;
CoN is the cost of non-conformities: efforts needed to fix defects, re-testing and reworking / refactoring your product;
CoA is the cost of appraisals: review time, testing of new or changed functionalities, independent verification, validation, or audit sessions.

Audits are usually done by an independent auditor outside of the QA team. They evaluate the work against existing industry standards like ISO standards and so on. Audits need to be planned carefully, defined using a standard model like CMMI, and formalized into a report to share with the stakeholders. Any areas of discrepancy must be converted into a set of SMART goals to ensure full compliance with industry standards.

5. Time your activities to suit your goals and objectives

NOTE: BrowserStack has achieved SOC2 Type 2 compliance, uses HTTPS by default, stores credentials securely, and does much more to ensure the security of our users. Please read our security documentation for detailed descriptions of our policies and controls.

6. Monitor Quality Objectives continuously

These include the efforts to fix the bugs found during testing. A plethora of associated tasks like changing the team structure, testing tool, or test methodology can also be part of this activity.

Conclusion

This step helps to Prevent Defects, Validate test objects, Reduce Risk, Share Information with stakeholders, and Find Failures and Defects from the objectives defined initially.

There are 3 "actors" in the team who coexist and influence the team dynamic. These are the people (including their skills, availability, and knowledge), the technology (that supports the QA process), and the work processes (waterfall, agile, DevOps, and so on). There must be a balance between these three to ensure the optimal creation and fulfillment of software quality goals. However, there is a "cost" to this balance, and that can be summed up using the following formula -

A successful quality assurance plan will aim to keep the CoQ within an acceptable range in the overall project's budget.

Some of the activities described above are once-off activities like Audits, or defining SMART goals. However, others are more continuous. Any processes related to review, verification, and validation will be repetitive across sprints for an agile process for instance.

Once the goals have been set, it is important to monitor them continuously in order to ensure that they are relevant to the project and that they are optimal in satisfying the overarching need to adhere to agreed quality standards and user expectations.

In the course of this article, we have defined what software quality means, why is it important for software engineering teams, and how to set about defining goals to meet it. Creating SMART goals for software quality helps keep the process within an agreed time frame and budget, and keeps the team focused on what is most critical for the success of a software project.

When done correctly, the returns from setting Software Quality Goals in terms of team and customer satisfaction, defect reduction, and product quality are undeniable. Add to that the advantage of being seen as acting towards a goal, rather than reacting to what is happening in the SDLC process, which builds customer confidence in the product and leads to a real quality product and not one that barely does what it is set out to do.

How To Perform Salesforce Testing

Sourojit Das — Sat, 23 Apr 2022 10:08:16 +0000

No more long-winded discussions about Salesforce Testing

The key goal of any business is to ensure customer satisfaction. Businesses invest a lot of time and effort into satisfying the customer base with an emphasis on rapid and efficient customer interactions, thus underlining the need for efficient CRM products in the market.

What is CRM?

According to market leader Salesforce, a CRM can be defined as "a strategy that companies use to manage interactions with customers and potential customers. CRM helps organizations streamline processes, build customer relationships, increase sales, improve customer service, and increase profitability." It is also known as Customer Relationship Management (CRM).

CRM tools have been on the rise for the past couple of years and it is very important for growing a business. Some benefits of a CRM software are as follows:

Improving efficiency: Having a central data repository ensures better team collaboration, workflow, and project management.
Customer retention: By relying on a single source of truth, there is more confidence in working with customers which enhances customer relations.
Transparency: Everyone has an idea about the processes, and can work with more clarity and better teamwork
Increased sales: By streamlining the sales process, pipeline building a task automation and sales data analysis will significantly improve your sales

What is Salesforce?

With a plethora of CRMs available in the market today, each provides its own set of benefits. However, Salesforce, the world's first cloud-based CRM remains the market leader in helping its clients sell, service, market, analyze, and connect with their clients in an affordable and hassle-free manner.

As per a Gartner report, the global CRM market grew from $61.6 billion in 2019 to $69.3 billion in 2020, reflecting an increase of 12.6%, and is currently the world's third-fastest growing segment in the IT space.

The Worldwide Semiannual Software Tracker by the International Data Corporation (IDC) has ranked Salesforce #1 in Customer Relationship Management Solutions for the eighth consecutive year running and its data clearly shows Salesforce rising head and shoulders above its competitors over the last few years.

This clearly highlights the need for Salesforce Testing in particular, and CRM testing in general to any QA Leaders intent on streamlining their process for a seamless customer experience.

What is Salesforce testing and why test salesforce applications?

The main goal of Salesforce testing is to test the customized application features, rather than those built into SalesForceDotCom (SDFC).

Since one of the most important USPs of SFDC is the fact that everything in it can be customized based on user requirements, Salesforce testing of these features serves to guarantee that any enhancements or custom features will not fail in production, or throw a spanner in the works of other elements in the SF application.

Testing the custom features in SDFC is important because:

It allows us to check if the code is functional.
It helps verify that the application can cater successfully to client needs based on business requirements.
It helps catch issues early in testing, making them easier to fix.
With the help of functional flows, we can obtain a report on the basis of test case statuses.

Initially, Salesforce services were offered using its Classic Platform, popularly known as Aloha. This was in vogue during the 2000s and was supplemented by Force.com which was SF's first PaaS (Platform as a Service) offering. This could allow clients to build custom applications on top of Salesforce without any additional software.

In more recent years, SF has offered the Lightning platform to its clients which comes with a number of additional functionality and security features, as well as an improved UI experience.

For the purposes of this article, we shall be dealing with the Salesforce Lightning platform.

The Scope of Salesforce Testing

Like any other platform, Salesforce applications also undergo the same phases of the software testing life cycle as shown below.

A short description of these types with examples is discussed below:

This is done mainly by developers themselves to test the code they wrote. These tests are not restricted to any specific user scenarios and tend to be highly specific.

Example: A developer has scripted a password input text field with its validation ar (8 characters long, must contain special characters.); makes a unit test to test out this one specific text field (has a test that only inputs 7 characters, no special characters, empty field)

Testers / QA professionals mainly perform this. It revolves around use cases and functional system requirements. Testers generally have no understanding of the code (also known as Black Box Testing).

Example: A tester writes a test case to ensure that the password is accurately saved in the database after generating one. This ensures that the code written by the developer is integrated with the system's database.

User Acceptance Testing

Acceptance testing determines if a software system meets all predetermined specifications. It evaluates if the system complies with business, technical, and aesthetic requirements so that business stakeholders and end-users can be satisfied alike.

Acceptance tests are generally divided into User Acceptance Testing and Business Acceptance Testing. The former checks to ensure that the product is meeting all performance standards from the users' perspective while the latter establishes that the software aligns with business goals and requirements.

Example: The client tests a particular functionality, like adding a customer to the CRM on a custom module on-site

Both the QA team and end-users perform testing in the production environment.

Example: Checking for customer creation and deletion processes in the CRM after go-live.

This can be done in both a manual as well as an automated manner. The main purpose of this type of testing is to check whether any changes made to a custom functionality have not impacted other functionalities and everything works as before

Example: Checking for the login functionality after a new UI has been designed for the login page.

Of these many types of testing, the Selenium Unit tests need to be written in the APEX programming language used by Salesforce, whereas the others can be done either manually or using an automation test tool like Selenium.

How to perform Salesforce Unit Testing in APEX

The @isTest is used to define classes and methods that only contain code used for testing the application.

A sample format would be

@isTest private class MyTestClass { @isTest static void test1() { // Implement test code } @isTest static void test2() { // Implement test code } }

Below are a few examples of implementing Unit tests for some common Salesforce constructs:

A trigger is an APEX script that executes prior to or posts a Data Manipulation Event (DML). These allow the performance of custom actions before or after a record has undergone insertion, update, or deletion.

@isTest public class TriggerTestClass { static testMethod void testMethod1() { // Perform DML here only } }

For Standard Controllers

A standard controller takes a record and places it in a variable that it makes available to a page.

@isTest public class ExtensionTestClass { static testMethod void testMethod1() { Account testAccount = new Account(); testAccount.Name='Test Account record' ; insert testAccount; Test.StartTest(); ApexPages.StandardController sc = new ApexPages.StandardController(testAccount); myControllerExtension testAccPlan = new myControllerExtension(sc); PageReference pageRef = Page.AccountPlan; // Add your VF page Name here pageRef.getParameters().put('id', String.valueOf(testAccount.Id)); Test.setCurrentPage(pageRef); //testAccPlan.save(); call all your function here Test.StopTest(); } }

Some best practices for Unit testing in Salesforce include:

Test class should be annotated with @isTest.
Classes with @isTest annotation can't be an interface or enum.
Test method should be static and no void return type.
Starting with the Salesforce API 28.0 test method can not reside inside non-test classes.
To deploy to production at least 75% code coverage is required

Manual and Automation Tests for Salesforce

Once the developers are happy with the test coverage through Unit Tests in Salesforce, it is time to perform a variety of manual or automation tests. However, before we start testing the SF applications, we must ensure to set up a Sandbox that allows us to perform these tests in isolation.

Salesforce Sandbox testing can be done on a variety of Sandbox types like:

Developer Sandbox: This is intended for development and testing in an isolated environment, This contains a copy of the production environment's configuration.
Developer Pro Sandbox: This can host larger sets of data than the vanilla Developer Sandbox, and has the capacity for increased development and quality assurance tasks and for integration testing or user training.
Full Sandbox: This is meant as a dedicated testing environment, and supports performance testing, load testing, and staging. They are intended as a replica of the production org, including all data, such as object records and attachments, and metadata. However, the significant lag due to the refresh interval makes them unsuitable for development purposes.

A Variety of licenses can be purchased for these Sandbox types as per your requirement.

Pro Tip: No emulator or simulator can replicate real user conditions. Software needs to be tested on real devices to work in real-world circumstances such as a low battery, incoming calls, weak network strength, and so on. If an in-house lab is not accessible, opt for a real device cloud.

Manual Tests in Salesforce

Manual testing, as the term suggests, refers to a test process in which a QA manually tests the software application in order to identify bugs. To do so, QAs follow a written test plan that describes a set of unique test scenarios. The QA is required to analyze the performance of the web or mobile application from an end user's perspective.

For the purposes of this article, take a simple positive test scenario of a successful login to the Salesforce site and write a brief test case accordingly.

Test Steps:

The user navigates to the Salesforce login page.

In the 'email' field, the user enters their registered email address.
The user enters the registered password.
The user clicks 'Login.'
The user is redirected to the Home page

Test Scenario: To authenticate a successful user login on Salesforce.com

Automated Tests in Salesforce

Automation Tests in Salesforce can be done using a variety of tools. However, we will consider test automation using Selenium With Java for the purposes of this article.

Selenium is compatible with multiple operating systems like Windows, Linux, Solaris, and Macintosh. It also supports multiple browsers like Chrome, Safari, IE, Edge, and Firefox. It is used to automate functional tests. Also, Selenium is easy to integrate with tools like Jenkins, Maven, and Docker to achieve a continuous testing approach. Tools like TestNG and JUnit further help in structuring the selenium tests for easy maintainability and generating reports.

For the purposes of this test, we shall build upon the example for manual testing shown above, and add a few features. This test will demonstrate how to use Selenium with Java to create a custom field for a custom object.

Step 1: Login to the Salesforce

To launch the website in a browser of your choosing, set the system properties to the path of the required driver for the browser. Since this example uses Google Chrome, it should be set to the ChromeDriver. The code for the same is as follows

Webdriver driver = new ChromeDriver(); System.setProperty("webdriver.chrome.driver", "Path of the chrome driver");

Maximize the browser for a clear picture of test cases being executed using the following command.

driver.manage.window.maximize();

Step 2: Navigate to salesforce.com

driver.get("https://login.salesforce.com");

It will appear as seen below

Step 3: Enter Username and Password using the following command

driver.findElement(By.xpath("//input[@id='username']")).sendKeys(""); driver.findElement(By.xpath("//input[@id=password]")).sendKeys("");

Step 4: Click the Login Button using the following command

driver.findElement(By.xpath("//input[@class='button r4 wide primary']")).click();

Since you may encounter some issues while loading the page, add a wait to let the page load completely

driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);

This will lead to the Home Page as seen below

Step 5: Find the Quick Find Search Field in the UI on the home screen and enter object in the Quick Find Search field.

driver.findElement(By.xpath("//input[@class='quickfindInput version2 ']")).sendKeys("object");

Step 6: Click appearing after entering object in the Quick Find Box

driver.findElement(By.xpath("//a[@id='CustomObjects_font']")).click();

Step 7: Click on the " Create New Objects Link "

driver.findElement(By.xpath("//a[@value='New Custom Object']")).click();

Step 8: Choose the Label text box to input the object name and do the same for Plural Label

driver.findElement(By.xpath("//input[@id='MasterLabel']")).sendKeys("abc"); driver.findElement(By.xpath("//input[@id=PluralLabel']")).sendKeys("abc's");

Step 9: Click on Save button at the end of the page

driver.findElement(By.xpath("//a[@value=' Save ']")).click();

The Custom Object created can be seen below

Best Practices of Salesforce Testing

Now that it is identified what to test, where to test, and how to test in terms of testing salesforce applications, let's discuss some best practices for testing salesforce effectively.

A proper test strategy and test plan are absolutely critical for Salesforce testing as it encompasses many components and business processes. A clear view of the high-level objectives, the stakeholders present, the different phases of testing involved, and the tools to be used must all be determined in this phase.

Salesforce offers a variety of Sandbox options for isolated Salesforce Sandbox testing. To satisfactorily test business-critical functions, all systems should be properly integrated in the sandbox so that proper integration testing can be performed in real-time scenarios like on production.

Automating critical and often repeated business processes, like a checkout or order fulfillment flow, in Salesforce saves lots of time when testing. Salesforce supports several types of API endpoints that can be used for automation testing. API endpoints can be automated by using tools like Postman, REST-Assured & Soap UI.

Navigating through frames
Executing against dynamic content
Handling tables
Object dependency
Heavy DOM structure and shadow DOMs
Driving data
Reusability
Long steps for end-to-end testing

Common Challenges in Salesforce Testing

Some of the core challenges that QA professionals face during Salesforce automation are:

SF Classic supports the latest IE 11 version, but it is not supported by Lightning.
SF Lightning does not support private browsing in the latest version of firefox but classic does
Certain features in Salesforce, like the Standard mail merge or extended mail merge (Salesforce Classic only Feature), and some client applications like Connect Offline work only on the Internet Explorer.

It is best to choose the correct test automation framework that provides solutions to these challenges. However, an even bigger challenge is the issue of successful cross browser testing in Salesforce regardless of whether it is done manually or through an automation framework.

Since Salesforce customers have an option of using the Classic or the Lightning platform features, there is an amount of variation involved in performing some tasks based on the platform of choice as well as the browser being used. For example:

Since all organizations cannot allocate funds for infrastructure to test have a massive test environment for Salesforce Testing, cloud-based platforms like Browserstack to implement cross-browser testing for all your Salesforce testing needs

Salesforce is a leading CRM, equipped with a large feature set, that can be customized and reconfigured to suit an organization's business needs. However, only focusing on developing SF modules is not enough. Salesforce testing is critical in ensuring customer satisfaction and delivering processes that not only work seamlessly but also enable the maximum leverage of Salesforce's unique capabilities to ensure customer success.

Parallel Testing: The Essential Guide

Sourojit Das — Fri, 15 Apr 2022 17:14:33 +0000

Parallel Testing: The Essential Guide

With more and more software engineering teams gravitating to the CI/CD model, the onus to reduce the time to market by delivering a foolproof product is increasing. And in QA, the only way to ensure minimal impact from bugs is to test the code as soon as possible and fail fast.

For QA teams, it means increasing test coverage through automation, but often the sheer number of tests required to be performed along with the paucity of time to test the features means that classical sequential automated tests are not enough. Parallel testing allows teams to perform automated tests against several configurations at the same time to scale downtime and budget constraints while allowing for increased test coverage and ensuring quality.

Parallel Testing: What it is and How it works

Parallel Testing is a process to leverage automation testing capabilities by allowing the execution of the same tests simultaneously in multiple environments, real device combinations, and browser configurations. The overarching goal of parallel testing is to reduce time and resource constraints. Unlike distributed testing, where the different test components running in a distributed manner interact with each other, the parallel test approach is free from any such interaction.

To better understand how parallel testing helps boost standard automation testing capabilities, let us take a simple example --- an automated functional test of a signup form. To perform this test for 45 different Browser/OS configurations, with each test taking an average time of 2 minutes, then the total run time of tests would be 90 mins or 1.5 hrs when run successfully in sequence.

Now imagine, when running 3 parallel tests simultaneously, the total execution time would have come down to 30 mins.

And for 6 parallels, it would be even further reduced to 15 mins- a far cry from what was expected before.

Considering this scenario to be one of the most basic functional tests that are automated, it boggles the mind how parallelization of tests can shave valuable minutes and hours from the most cumbersome of test suites.

The number of parallels required to optimize test times depends on the following key factors:

The current number of browsers (and/or) Devices on which the scenario needs to be tested
Number of parallel tests currently being run ( we will keep this as 1 if only sequential tests are being run)
Current Build Time
The desired number of browsers (and/or) Devices on which the scenario needs to be tested, and
Desired Build Time

For more complex scenarios involving a shift in any of these parameters, it is best to use a Parallel Test Calculator to understand the number of parallel sessions required to achieve test coverage and build execution time goals.

When to perform Parallel Testing

Now that we have some inkling of how parallel testing works and how it can optimize test execution time frames let's have a look at certain scenarios where parallel test execution is ideal:

In case the software engineering team is uncertain about the functionality of an application across devices and configurations, especially in regression-like scenarios with an updated version of an app.
In a case where legacy data is imported from an older system to a newer one, Parallel tests can be immensely helpful to check if everything has been moved across seamlessly using an automated regression test suite.
For cases of automated browser testing, when an app needs to be tested across different combinations of browsers, OS, and devices.

How to approach parallel test creation

A common way to perform parallel testing is to create a suite of projects to test browser compatibility and then a master project to test the aforementioned set of projects. Upon set up, it is time to create a set of Entry and Exit Criteria.

For Entry Criteria, we must define those checkboxes that must be ticked off before test execution starts, for e.g.

Set up a suitable test environment, for instance, Selenium with TestNG
Define the pre-conditions and test scenarios for the automated browser test process
Migrate any new and legacy data required

The exit level criteria can focus more on:

Understanding the difference in terms of test suite execution impact
Measuring the output of a new system as compared to a legacy system (if applicable)

Parallel Testing using TestNG and Selenium

TestNG is a Java-based testing framework that helps to organize tests in a structured manner and enhances the maintainability and readability of test scripts. The large feature set of TestNG allows for the easy configuration of parallel tests. The auto-defined XML file in the TestNG framework allows testers to set the parallel attributes to methods/tests/classes by leveraging the allowance for multithreading in Java. This file can help determine the number of threads desired for parallel test execution.

The parallel attribute can be extended to:

Methods: To run methods in separate threads
Tests: To run all methods sharing the same tag in the same thread
Classes: to run all methods belonging to a class in a single thread
Instances: to run all methods in the same instance in the same thread

Whereas the thread count attribute can help define the number of threads the tester wishes to create when executing the tests in parallel.

The below code snippet helps to demonstrate parallel test execution for opening the BrowserStack homepage using two methods. One of the methods performs this action on Google Chrome, whereas the other does so on the Firefox browser.

import org.testng.annotations.AfterClass; import org.testng.annotations.Test; import org.testng.annotations.Test; import java.util.concurrent.TimeUnit; import org.openqa.selenium.By; import org.openqa.selenium.WebDriver; import org.openqa.selenium.chrome.ChromeDriver; import org.openqa.selenium.edge.EdgeDriver; import org.openqa.selenium.firefox.FirefoxDriver; import org.openqa.selenium.ie.InternetExplorerDriver; import org.testng.annotations.Test; public class ParallelTestWithMultiThread { WebDriver driver; @test() public void testOnChromeWithBrowserStackUrl() { System.setProperty("webdriver.chrome.driver", ".\Driver\chromedriver.exe"); driver=new ChromeDriver(); driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS); driver.get("https://www.browserstack.com/"); driver.manage().window().maximize(); System.out.println("this is the test related to chrome browserstack homepage"+ " " +Thread.currentThread().getId()); } @test() public void testOnChromeWithBrowserStackSignUp() { System.setProperty("webdriver.gecko.driver", ".\Driver\geckodriver.exe"); driver=new FirefoxDriver(); driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS); driver.get("https://www.browserstack.com/users/sign_up"); driver.manage().window().maximize(); driver.findElement(By.id("user_full_name")).sendKeys(""); driver.findElement(By.id("user_email_login")).sendKeys(""); driver.findElement(By.id("user_password")).sendKeys(""); System.out.println("this is the test related to chrome browserstack login"+ " " +Thread.currentThread().getId()); } @AfterClass public void close() { driver.quit(); } }

The TestNG XML file below helps to set up the parallel attribute as well as the thread count for the task.

<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">

Once we are done with the execution, the TestNG report will look something like this

A clear sign of the time savings obtained from parallelization can be seen below, in the real-time result, where we can see that parallel test execution is quicker by almost 1.5 times than serialized test execution

Running MBUnit tests in Parallel on Browserstack Automate

BrowserStack Automate can enable users to leverage the power of MBUnit to run tests across various browser, device, and OS configurations in parallel.

For test execution, the steps are as below -

Clone the mbunit-browserstack repo on GitHub git clone https://github.com/browserstack/mbunit-browserstack.git
Open MBUnit-BrowserStack.sln in Visual Studio\ Configure the credentials required and the details of the browsers/devices for parallel testing in the App.config file within the mbunit-browserstack/MBUnit-BrowserStack/ directory
Understand the ROI from automation and get buy-in from the stakeholders

<?xml version="1.0" encoding="utf-8" ?>

Now the tests are ready to run in parallel on BrowserStack using the following simple steps:
Building the solution in Visual Studio
Executing the tests with the fixture parallel from Test Explorer.
Other capabilities can be added under the environments option as described below:

Advantages and Limitations of Parallel Testing

Now that it is discussed when and how to run parallel tests, let us look at some of its advantages and limitations:

Advantages of Parallel Testing

Speed: As we have demonstrated both theoretically and in practice, sequential automated test execution is time-consuming. Parallel tests can speed up the process significantly as the same test is being run against different configurations at the same time.
Cost Efficiency: Creating, maintaining, and keeping test environments active need time and effort. However, these overheads are hardly a headache in parallel testing, and the required test environment can be leased when required and updated as per demands. Also, cloud-based test grids allow test execution at high concurrency, reducing the overall cost per test.
Optimization of CI/CD Processes: The effectiveness of a CI/CD pipeline is determined by its total run time. With feedback loops being of the utmost importance in CI/CD, the sooner the team has a result, the sooner they can apply the fix, refactor, and reiterate the process. Large test suites can be broken up into smaller independent jobs and run in parallel to quickly obtain results, thus improving the performance of the pipeline

Limitations of Parallel Testing

For the parallel testing of different modules, we need to create independent modules. Also, modules with dependencies cannot be included in the parallel approach.
For parallelization, one needs to have a detailed understanding of the product and its flow for better results. Even though parallelization can help in cross browser compatibility testing, its coverage of multiple browsers is restricted until and unless it is accompanied by distributed testing where the setup of multiple machines and browsers is provided.

Best Practices of Parallel Testing

None of the advantages of parallel testing will be of any use; however if we fail to follow best practices-

Make autonomous tests: One test should not depend on the outcome of another, as parallel tests may not occur in any defined order.
Test one feature at a time: The major advantage of parallel testing is modularization. A test that encompasses many features defeats the purpose of failing fast which is absolutely key to CI/CD pipeline performance.
Avoid the use of static objects: Using static objects may prevent another test, running in parallel, from manipulating it.
Resetting the test data: Cross-contamination of data needs to be prevented by modular creation, usage, and tearing down of test data to ensure the test leaves the system the same way it entered it.

In a nutshell

Owing to the need for having access to multiple platforms and browsers to run tests in parallel, the cost of compatibility testing with parallel testing increases. Also, we may come to a point where access to all browsers and versions may not be possible. BrowserStack provides access to numerous platforms and browsers with their corresponding versions on the cloud. One can run automated parallel tests and use multiple browsers and versions.

How to move from Manual to Automation Testing

Sourojit Das — Thu, 24 Mar 2022 16:49:03 +0000

With software requirements changing at lightning speed in an agile environment, more and more organizations look to inculcate more atomic development cycles for accelerating their Time To Market (TTM). A 2021 study by digital.ai has discovered that 86% of software development teams have adopted agile, up from 37%, a year ago. This shift to more agile software development methods has led to a simultaneous demand for more efficient means of software testing during the software is developed. Companies run the risk of suboptimal software delivery due to costly manual test case maintenance cycles as well as the lack of visibility these legacy processes bring to the table.

A Statista study highlights that 32% of all Software projects fail due to the simple lack of time to test the product thoroughly.

Testing timelines are cut shorter, for quicker software delivery. It is mostly because the consumers have different options to explore, as a result of which, the user expectations have increased. Hence, it is important to deliver a high-quality software release with new features every few weeks for customer engagement and retention.

The heavy workload upon software testers working against the clock to perform high volumes of repetitive manual tests is a major concern. Additionally, customer dissatisfaction arising from uncaught defects during production and costly downtime resulting from sub optimally tested features are driving factors too. These reasons have cumulatively led to the increasing adoption of automation testing in the industry.

In such conditions, traditional manual testing is often seen as being unnecessarily time-consuming and prone to human errors. Thereby, it is essential to transform from manual to automation test practices for keeping up with global trends and business demands.

Before discussing the details of transitioning from manual to automation testing, let’s have a brief overview of both these methods along with some scenarios where they are optimal.

What is Manual Testing

Traditional manual testing requires testers to manually investigate defects in software by adhering to a predefined test plan comprising a number of test cases. The testers take it upon themselves to analyze the behavior of the app and check if the functionality is synchronous with the expected behavior.

Testers identify bugs and report them to developers who fix them after replicating the errant behavior. This can become challenging for a test lead/manager especially when the timelines are shorter.

When to use Manual Testing

Manual testing is useful when the QA process requires cognitive and behavioral abilities beyond that of software. It is recommended to use Manual testing in situations like:

Also, for scenarios where:

The testers need a quick turnaround on a particular scenario with a clear intent. Since automation tests take time to set up, it might be faster to test one-off scenarios manually. However, scenarios that require a repetition with different inputs would be better off being tested through automation testing.
The projects are short-term with a small number of feature tests. Since automation is time-consuming to set up, small projects can be tested manually.
End-user usability is of importance, as no machine is accurate enough to decide the satisfactoriness of a user journey without an actual user testing it.

All of these test scenarios can now be expanded into a set of positive and negative test cases.

What is Automation Testing

Automation testing primarily uses frameworks to execute test cases. Each framework is unique to an extent and is powered by scripts adhering to the rules of the framework in question. These frameworks have components like:

Function libraries
Test Data Sources
Object Details
Other reusable modules

These frameworks can be classified as

Linear
Data-Driven
Keyword-Driven
Agile

When to use Automation Testing

Automation tests work best when we need to run a large volume of repetitive tests with differing inputs. For example, Data-Driven testing of a large number of regression tests. As in this case, the functionality has essentially remained the same, it will not require much manual oversight.

Also, when human resources are better utilized in other tasks which require greater cognitive effort, it is better to run automated tests. Parallel testing offers a great avenue to run a large volume of tests without compromising test accuracy.

Thus, to recap, we have identified a couple of important caveats before focussing on the transition from manual to automation testing:

Not everything can be automated
Manual testing cannot be done away with completely (at least in some areas).

How to switch from Manual to Automation Testing

As a quality assurance leader, once you have identified the need to automate a portion of your existing test portfolio, it is important to consider the steps required to achieve your goal. Here are a few simple guidelines to help make that decision:

Understand the ROI from automation and get buy-in from the stakeholders

Change is inevitable, however, necessary. But for change to be accepted, the stakeholders need to be convinced of the ROI arising from that change. The Return on Investment from automation can be best described in terms of:

Increased speed of test coverage: Automated tests can work wonders for repetitive test cases and help minimize the cognitive load for data-driven regression testing allowing your team to work in a more agile way across sprints.
Reduced cost: Though some may claim that the initial cost of setting up an automated test framework is higher than standard manual testing, one will only reap increasing dividends by setting up the infrastructure. Studies have shown that the costs of test automation reaches a positive balance after the third test cycle, and this can be speeded up even further in an agile scenario.

Providing a baseline for Continuous Integration and DevOps: An automation test suite is a foundation for transitioning into a scalable CI/CD and DevOps model. Continuous Testing is the execution of automated tests at regular intervals made to correspond to code changes. These tests are conducted as part of the software delivery pipeline and provide faster feedback on changes to the repository.
Increased Accuracy and Productivity: Test automation provides faster test coverage with data-driven accuracy and frees up resources to divert their attention to more critical tasks thereby increasing overall productivity as well.

Decide what to Automate

The key decision in any test-automation strategy is to select which test suites to automate. Some of the tests which are most suitable for automation are as follows:

Regression Test Suites: These are usually voluminous, require the same set of variables to be used as input and are usually run several times during the development
Data-Driven Test Suites: Even a lot of functional tests are data-driven and need to be tested with numerous divergent data sets to test a variety of positive and negative scenarios.
Performance Testing Suites: Tests to monitor system performance under different circumstances can be automated to reduce test times.

Select the Right Framework(s) for the project

Selecting the right framework is essential for streamlined testing. Test frameworks are often licensed and need time to set up. Moreover, it is important the team is well-versed with the tool for delivering accurate results. Selecting the right framework depends upon:

The Nature of the Software: Understand the primary audience and thus the main runtime system for your software. If it is desktop-based, a tool like Selenium is preferable. If it is mobile-first then Appium can be considered, and so on.
Programmer Skillset: It is vital to choose frameworks that are compatible with your team’s forte and experience. Some popular languages for test automation frameworks are Java, JavaScript, Ruby, C#, etc. It is best to select a framework that supports the programming language preferred for practice by the team.
Cost: Automation is sometimes expensive. Depending on your need you may choose a licensed framework like QTP or an open-source framework like Selenium Webdriver.

Set bite-sized targets and a rapid learning curve

Test automation is often touted as a silver bullet in the world of QA. However, the journey from a manual to an automated test suite might not be smooth. It’s best to start with small goals. Automate the test suite that is most used in regression testing. Isolate any common errors, and iteratively expand the scope and complexity of the automation to increase test coverage.

Aim for thorough visibility and continuous testing

Continuous Integration (CI), Continuous Deployment (CD), and Continuous Testing (CT) are all key components in Agile. While CI enables developers to merge code changes to the repository on a daily basis, CT is required to ensure that those changes work as expected.

Have a well-defined lifecycle for your automation test suite

Once the automation suite has been set up (sometimes at the cost of considerable time, effort and finances) it is vital to ensure there is a clear roadmap to implement and maintain the same. Some common pitfalls of test automation include:

lack of specific goals,
inadequately testable applications,
low visibility of the automation ROI across the organization.

It is important to make automation a core part of the STLC (Software Testing LifeCycle) to ensure its success. Also, following automation best practices ensures that the transition is seamless and the rewards are commensurate to the efforts.

Test automation has become indispensable for organizations seeking to make the most out of Agile Software Development practices and incorporate CI/CD and DevOps into their development plans. Though manual testing still retains a niche, a transition to automation can result in speedier test coverage with enhanced accuracy. Performing tests on Real Devices would help generate more accurate results.

How to create Selenium test cases

Sourojit Das — Thu, 03 Feb 2022 11:40:30 +0000

Selenium has established itself as one of the most popular automation testing frameworks within the software engineering community. Comprising a suite of tools – IDE, RC, WebDriver, and Grid – Selenium has quickly established itself as a market leader in the test automation sector.

Selenium capitalized on the inadequacies of previously used frameworks like HP QTP, and Mercury’s Astra QuickTest which compelled users to write scripts in VBScript and were primarily intended for desktop-based tests. Its support of a variety of languages like Java, Python, C#, Perl, Ruby, .net and PHP, has made it popular with individual testers and companies like Netflix, Google, HubSpot, and Fitbit among others.

The core idea behind Jason Higgins launching Selenium back in 2004 was to remove the need for users to perform repetitive manual testing. This article will focus on how to write test cases in Selenium in a way that allows us to leverage its ability to provide maximum, accurate test coverage in a limited time.

Let’s start by giving a brief outline of how test cases are written, and then proceed to create a Selenium WebDriver test script from a manual test case using the JAVA programming language.

How to extract test cases from Business Requirements

All software must undergo End to End testing before being released. This method tests the application workflow from beginning to end by replicating different user scenarios. The first step in performing an End to End test is to analyze business requirements. A tester considers the different user personas involved, aims for maximum test coverage, and considers using automation to achieve it.

The usual process starts with the tester studying the business requirements to isolate a set of user stories. These user stories help represent the business requirements – what a person using the product would like to be able to do. This, in turn, helps cater to a series of user personas, and create a number of test scenarios.

Once the test scenarios are framed, a set of industry-wide best practices are followed to frame these scenarios into a sequence of actions that can help verify a particular functionality. This is what is known as a Test Case.

Writing a test case using Selenium

To write the first test case, consider a sample business requirement – Ensure secure user entry to a BrowserStack Account.

Given that a user trying to gain entry to a BrowserStack Account could either be a new or an existing user, test scenarios can be framed around both the Register and Login pages.

For the purposes of this article, consider the user persona to be that of a registered user.

On viewing encounter the BrowserStack Login page, ask the following questions:

Can a user login with the correct user ID and password?
What happens when an invalid email id and invalid password are entered into the form?
What happens when a valid email id and invalid password are entered into the form?
What happens when an invalid email id and valid password are entered into the form?

All of these test scenarios can now be expanded into a set of positive and negative test cases.

Positive test cases ensure that the ‘happy-path’ or expected user journey when the correct data is entered, works as intended. Negative test cases focus on invalid actions, often by replicating the input of invalid data or seeking access to an invalid component.

In this case, the first test scenario can lead to the development of a positive test case, whereas the rest lead to negative test cases.

Example of a Test Case in Selenium

Let’s build a Selenium test case example based on the first test scenario.

Test Scenario: To authenticate a successful user login on Browserstack.com

Test Steps:

The user navigates to the BrowserStack sign-in page.
In the ’email’ field, the user enters their registered email address.
The user enters the registered password.
The user clicks ‘Sign Me In.’

Prerequisites : A registered email ID with a unique username and password.
Browser: Chrome v 86.
Test Data: ** Legitimate username and password.
**Expected/Intended Results: Once username and password are entered, the web page redirects to the user’s dashboard.
Actual Results: ** As Expected
**Test Status: Pass/Fail: Pass

Converting a Selenium Test Case to a Test Script

After configuring the system to execute test scripts, let’s convert the manual test case into an executable test script. For that, carry out the following steps:

Create a Selenium WebDriver instance.
Perform browser optimization if necessary.
Write a sequence of commands to execute the test steps.
Validate the actions performed.

Let’s explore each step in detail:

Step 1 – To launch the website in a browser of your choosing, set the system properties to the path of the required driver for the browser. Since this example uses Google Chrome, it should be set to the ChromeDriver. The code for the same is as follows –

`Webdriver driver = new ChromeDriver();

System.setProperty("webdriver.chrome.driver", "Path of the chrome driver");`

Step 2 – Proceed to maximize the browser for a clear picture of test cases being executed using the following command.

driver.manage.window.maximize();

Step 3 – Once the basic tasks are complete, execute each step of the manual test case articulated in the previous section.

Implement test steps

Step 1 – The user navigates to Gmail.com

driver.get("https://www.browserstack.com/users/sign_in");

Step 2 – In the ’email’ field, the user enters their registered email address.

To perform this step, locate the email field on the Google Sign In form, and enter the requisite data.

Use the ID locator in Chrome DevTools to identify the necessary elements for the Email field, and store them in a webelement variable.

`driver.findElement(By.id("user_email_login"));

WebElement username=driver.findElement(By.id("user_email_login"));`

Once the element has been located, perform the requisite action:

username.sendKeys("abc@gmail.com");

Step 3 – The user enters the registered password

Repeat the same process for the password field as well

`driver.findElement(By.id("user_password"));

WebElement password=driver.findElement(By.id("user_password"));

password.sendKeys("your_password");`

Step 4 – The user clicks ‘Sign In’

Once the email and password web elements have been located and the actions have been performed, the tester can perform the final desired action. In this case, the user clicks ‘Sign In.’

login.click();

Now, validate the actions performed using assertions. Assertions help testers compare the expected and actual results of the actions performed. If these are in sync, the test case is said to have passed. Otherwise, the test case is considered to have failed.

The assertion, in this case, is of the general form:

Assert.assertEquals(String actual, String expected);
The String actual variable holds the post-login value, which is:

String actualUrl="https://live.browserstack.com/dashboard";

The method below can help get the expected URL:

String expectedUrl= driver.getCurrentUrl();
Thus, the final code will look something like this:

Assert.assertEquals(actualUrl, expectedUrl);
If the test case passes, it will retrieve the same else will return as having failed.

The final code will be as follows:

import org.openqa.selenium.By; import org.openqa.selenium.WebDriver; import org.openqa.selenium.WebElement; import org.openqa.selenium.chrome.ChromeDriver; import org.testng.Assert; import org.testng.annotations.Test; public class LoginAutomation { @Test public void login() { System.setProperty("webdriver.chrome.driver", "path of driver"); WebDriver driver=new ChromeDriver(); driver.manage().window().maximize(); driver.get("https://www.browserstack.com/users/sign_in"); WebElement username=driver.findElement(By.id("user_email_Login")); WebElement password=driver.findElement(By.id("user_password")); WebElement login=driver.findElement(By.name("commit")); username.sendKeys("abc@gmail.com"); password.sendKeys("your_password"); login.click(); String actualUrl="https://live.browserstack.com/dashboard"; String expectedUrl= driver.getCurrentUrl(); Assert.assertEquals(expectedUrl,actualUrl); } }

How to Select Test Cases for Selenium Automation

Selenium and other test automation frameworks help reduce development cycle timeframes, avoid repetitive tasks and achieve 100% test coverage in a quick and efficient manner. But, to get the most out of Selenium tests, it is necessary to focus on some best practices when selecting test cases for automation.

Select test suites which offer the highest Return on Investment in terms of cost-saving, increasing efficiency, and improving test quality.
Identify test cases on what they are meant to do. needing to be automated. Based on business goals and requirements, automate the following:

a.Test cases to be executed with different data sets
b.Test cases to be executed on different environments
c.Test cases to be executed for different user types
d.Test cases that have multiple dependencies, etc

Consider the execution time and testing frequency of these test cases. If one or both of these parameters are high, then it is likely a test automation framework is required to write these test cases.