Regression Testing: A Complete Guide

In the fast-paced landscape of rapid software development, where upgrades and modifications are frequent, it is crucial to ensure the stability and quality of software products. Regression testing, as a critical component of the testing process, plays a vital role in achieving this goal.

What is regression testing, and why do we need it?

Regression testing is a type of software testing conducted to confirm that a recent change or upgrade in the application has not adversely affected the existing functionalities. It is performed to ensure that the modifications or fixes made in one code module do not impact other modules due to dependencies. A tester typically initiates regression testing after a developer incorporates a new functionality into the application or completes fixing a current error.

Example of regression tests

Suppose you are working on an e-commerce website that allows users to search for products and make purchases. You have recently made some changes to the search functionality and want to perform regression tests to ensure that the changes have not introduced any new issues.

1. Search functionality: First test the search functionality to ensure that it still works as expected. This would include testing the search bar, filters, sorting options, and search results.
2. Product pages: Then test the product pages to ensure they still display the correct information and images. This would include testing the product description, price, availability, and reviews.
3. Shopping cart: Test the cart to ensure it works as expected. This would include testing the ability to add and remove items, update quantities, and proceed to checkout.
4. Payment processing: Test the functionality to ensure it still works as expected. This would include testing the ability to enter payment information, select a payment method, and complete the purchase.
5. User accounts: Test the user accounts functionality to ensure it works as expected. This would include testing the ability to create an account, log in, view order history, and update account information.

Record & Replay Approach in regression testing

Today's software applications are incredibly complex, with numerous microservices, containers, and connections. As a result, it's challenging to manually script test cases for every possible scenario, making it difficult to ensure that applications are bug-free.

Despite the best efforts of QA teams, maintaining application stability and quality can be challenging. Writing and maintaining tests that fail to cover every user flow in the product is a time-consuming task, and workflows outside of these tests can break, leading to the generation of new bugs regularly.

Fortunately, the record and replay approach used by AREX offers a solution to this problem. AREX monitors every user interaction, automatically building integration tests without requiring teams to write a single line of code. This comprehensive approach ensures that every possible API regression, including logical, functional, and performance failures, is caught.

Compared to manually scripting and generating traffic, traffic replay has several advantages:

• Generates more realistic tests
• Reduces the cognitive load required to create tests
• Identifies bugs that may not be discovered in a controlled environment
• Leverages varied data

1. Record Application Traffic

In general, the invocation chain of a request includes the entry point and various synchronous or asynchronous dependent calls. The recording process in AREX involves linking the entry point and the dependent calls together using a Record-Id, to save as a complete test case. The AREX-Agent achieves this by performing bytecode enhancement on the code of the entry point and dependent calls. When the code is executed, the invocation process is intercepted, and the input parameters, return values, and exceptions of the calls are recorded and sent to the storage service.

2. Replay Traffic to test a new build

During the replay mode, AREX Schedule Service retrieves the recorded data of the real traffic requests in the production environment and then sends these requests to the tested application in test environment. At the same time, the Java Agent returns the recorded responses of external dependencies to the tested application.

AREX then compares the recorded responses with the responses of the version under test to verify the correctness of the system logic. This eliminates the need for QA teams to manually think and write all assertions, streamlining the testing process.

Best practices for regression testing

For the demo, we use the community-test application as an example to demonstrate the entire process of AREX automation regression testing, from configuring the AREX Agent to recording and playback, and issue identification.

Start the AREX Docker Project

The AREX project simplifies the setup, deployment, and execution of various services, including AREX-UI, Schedule Service, API Service, Storage Service, and others, by utilizing the existing pre-defined Docker Compose configuration file. With just a single command.

First, clone the deployments repo on GitHub:

cd arex git clone https://github.com/arextest/deployments.git

Then start AREX by simply running the docker compose up command in a terminal from the project root folder.

docker-compose up -d 

After installation, check the status and ports of each service by running docker-compose ps on the host running Docker.

Navigate to http://10.5.153.1:8088/ where the arex-front is running. Enter your email and you will receive a verification code to log in:

Configure AREX Java Agent

AOP (Aspect Oriented Programming) is a technique for building common, reusable routines that can be applied applicationwide. To use AREX Java Agent for recording traffic in a Java application, you essentially integrate the AREX Java Agent into your application. This agent will intercept and record the traffic, such as HTTP requests and responses. AOP is used here to weave the recording logic into your application without modifying the source code directly. It adds the recording aspect at runtime, enabling the AREX Java Agent to capture and log the traffic seamlessly as part of the application's operation.

This approach is efficient because it minimizes code intrusion while ensuring comprehensive traffic recording.

First, clone and compile the AREX Agent.

git clone https://github.com/arextest/arex-agent-java.git cd arex-agent-java mvn clean package

Compilation complete:

Step 1: Create an application in AREX

To set up the application, navigate to the AREX UI(http://10.5.153.1:8088/) and select Replay on the left-hand navigation bar. Click "+" to Create Application.

Step 2: Copy the Agent Script

After creating an application, the startup parameters for the Agent of the application will be automatically generated as shown below.

Replace the values in the pointed brackets as appropriate. You will use these parameters in the next step for the app config.

Step 3: Start Application with AREX Java Agent

Start the community-test application with the former AREX Java Agent script:

java -javaagent:/usr/local/tomcat/arex-agent-0.3.0.jar      -Darex.service.name=f27172b0e331a42a      -Darex.storage.service.host=10.5.153.1:8093      -jar community-service.jar

• '-javaagent:/usr/local/tomcat/arex-agent-0.3.0.jar' is the JAR file of the AREX Agent that we have compiled. This JAR file will be loaded into the application as a Java agent.
• -Darex.storage.service.host=10.5.153.1:8093: This is the address of the AREX Storage service, specified as 10.5.153.1:8093.

As you can see, the application is running successfully:

Record requests in a production environment

Now that arex-agent is successfully up and running in the application, the user does not need to perform any special operations or interventions during the recording process and can serve the application externally or access it manually in the usual way.

Code Updates

Now you've got your business requirements sorted and it's time to test a new version. Just follow these steps:

1. Pull the latest code.
2. Compile and package it up.
3. Heads up: Don't mess with the AREX Agent's config, especially the app name. AREX needs it to keep track of your app and handle the test cases. So, make sure the app name stays the same as before during compile and test.
4. Roll out the code to your test environment: Ship your compiled and packaged app to the testing grounds for some serious trial runs.

Replay Traffic to perform regression test

Now we need to perform a regression test on the new version of the application. Start replay testing and fill in the test environment (here is http://10.5.153.1:8080/):

Troubleshooting

If there are no discrepancies in the comparison between the recorded responses with the responses of the version under test, the test passes:

If the test failed, that means there is a discrepancy in the two responses. Then we need to troubleshooting.

The playback report will display all the interfaces and the test results, as shown below:

When compared, any deviations between the 2 versions is a possible breaking change in the version under test.

When you spot discrepancies, here's how to iron them out in the code:

1. Identify the issue and implement fixes.
2. Cycle back through the 'Code Update' steps: modify, test, deploy, and compare again.
3. If it turns out the discrepancy isn't an issue, label that section to bypass it in future playbacks.
4. Keep iterating this process until all discrepancies are either resolved or identified as within acceptable limits.
5. Once all fixes are confirmed, go ahead with releasing the updated version.