Complete Guide of Regression Testing: Definition, Tools & How to do

What is regression testing?

Regression testing refers to testing an application to ensure that the previously code remains operational as expected after changes or enhancements made to it. It is a quality control measure that helps developers and testers identify potential defects or problems that may occur with new code additions, design modifications, or system upgrades before a new feature markets.

It should be noted that regression testing is more than just simply re-testing. Re-testing concentrates solely on the test cases that have failed, whereas regression testing is used to examine the test cases that have passed to identify any unanticipated new bugs.

When to perform regression testing?

Typically, regression testing is performed after any changes have been made to the software to verify the changes have not introduced any new issues or affected any previously working functionality, like the following scenarios:

1. After making changes to the software.
2. After fixing a defect.
3. After updating to the operating system, browser, or other external applications occur that may affect product functionality.
4. After a major release.

How to perform Regression Testing?

Regression testing can be executed both manually and in an automated manner. Test Engineers primarily use special techniques and methods to perform Regression tests.

1. Identify the changes: Identify the changes made to the system under test, such as code modifications, enhancements, bug fixes, etc.
2. Select the test cases: Select the test cases that cover the modified or impacted functionalities by analyzing the requirements and design documents, test cases, and bug reports.
3. Prioritize the test cases: Prioritize the selected test cases based on their criticality and impact on the system.
4. Create a regression test suite: Create a regression test suite that includes the selected test cases.
5. Execute the test cases: Execute the test cases in the regression test suite and compare the actual results with the expected results.
6. Report the defects: Report any defects identified during the test execution and ensure that they are fixed by the development team.
7. Repeat the cycle: Repeat the above steps whenever there are new changes made to the system under test to ensure that the system is functioning as expected.

Challenges of regression testing

• Large Scope and Coverage

There are challenges faced while listing out functionalities to be included in the test suite and possible scenarios. Regression testing may have limited test coverage, as it may not be possible to test all possible scenarios due to time and resource constraints.

• Maintenance and Optimization

Maintaining and optimizing the existing regression test suites is a significant task. When there are new modifications, the current regression test suites may need to be modified by adding, eliminating, or altering the existing test cases. All of this must be completed before the regression testing deadline.

• Validate database, cache, or message queue

To validate business correctness, it is not sufficient to simply validate the returned results from server. In some scenarios such as generating orders and making payments that require writing data to the database, message queues, cache, etc., it is necessary to validate whether the interaction requests between the business system and the underlying business system have changed, whether the content of the data written to the database by the business system is correct, whether the content of the asynchronous messages sent by the business system is correct, and so on.

Use the right testing tool

There are multiple ways to overcome the challenges in regression testing. One of the most straightforward approaches is record and replay testing. This low-code method involves using a tool to automate tests without writing test scripts. As a result, it is often considered the simplest way to perform regression testing.

There are different tools and frameworks that can be used to perform Regression Testing. Such as:

1. Selenium IDE: Selenium IDE is a user-friendly and free tool for recording and playing back web application tests. With its simple installation process and browser extension, you can quickly start recording your testing activities. However, a significant drawback is that the tool has limited capabilities in terms of scalability.

2. TestComplete: TestComplete is another pre-built tool that includes a record and replay functionality, as well as scripting capabilities and advanced features such as parallel or keyword-driven testing, object recognition engine, and robust reporting. These features can help your team enhance its testing capabilities and improve overall efficiency.

3. Katalon: Tthe record and replay feature is readily available and easy to use, with a simple interface. The tool also includes a built-in object repository that follows the Page-Object model, making it easy to organize and maintain test objects after recording tests. You can start utilizing this feature immediately without any additional setup.

4. AREX: It is an open-source automated regression testing platform, which allows one to record production traffic and replay it to testing environment. By comparing the recorded responses against replay, it can clarify the impact of code changes to realize efficient automated regression testing.

Regression testing with AREX

Record production traffic as test cases

Step 1: Install AREX

git clone https://github.com/arextest/deployments.git
cd deployments
docker-compose up -d

Step 2: Modify the way your production environment for the application under test runs to the following command

java -javaagent:/path/to/arex-agent-0.1.0.jar -Darex.service.name=your-service-name -Darex.storage.service.host=[storage.service.host:port](storage.service.host:port) -jar your-application.jar

Where：

• /path/to/arex-agent-0.1.0.jar needs to be changed to the actual path of the arex-agent-0.1.0.jar file in your server
• your-service-name needs to be changed to the name of your application under test
• [storage.service.host:port] needs to be changed to the address and port number of the data storage service
• your-application.jar needs to be changed to the jar package name of the application

Or change the environment variables in the environment

export JAVA_OPTS="-javaagent:/path/to/arex-agent-0.1.0.jar -Darex.config.path=/path/to/arex.agent.conf"

Replay traffic in the testing environment

Step 3: Visit the AREX UI from browser, and in the Replay tab, select the service with AREX Agent(i.e. your-service-name above).

Step 4: Select Replay, enter the Target Host of your test environment and start replaying all the recorded test cases.

After the replay is completed, a detailed replay report will be generated, where you can visually see the pass rate, API pass rate, and the details under each API.

Click the failed API to see the comparison results, and AREX visualized the comparison results to facilitate troubleshooting.