DEV Community: AREX Test

AREX or GoReplay: Choosing the Right Tool for Effective Traffic Replay

Jing — Thu, 21 Dec 2023 08:53:41 +0000

As software applications grow more complex, encompassing elements like microservices, containers, and intricate connections, predicting and manually scripting test cases for every conceivable scenario has turned into a formidable challenge. The conventional methodology, which involves the creation and validation of test cases, demands significant manual labor and the upkeep of extensive script libraries. This traditional approach is increasingly seen as labor-intensive and less feasible in the dynamic landscape of modern software development.

Nevertheless, the advent of Production traffic replay has revolutionized the process of testing and validating applications. This innovative approach facilitates the replication of actual production traffic, enabling the realistic simulation and thorough exercise of real-world scenarios. By capturing and replaying network traffic, this method offers a streamlined path to accelerate the testing process. More importantly, it instills greater confidence in the modifications applied to the application, ensuring they perform robustly in live environments.

This approach offers several distinct advantages compared to conventional test case generation:

Produces tests that are more representative of real-world scenarios.
Significantly lessens the cognitive effort needed to devise test cases.
Uncovers potential bugs that might remain undetected in a controlled testing environment.
Utilizes a diverse range of data for more comprehensive testing.

Several tools capable of capturing production traffic, such as GoReplay and AREX, are already available in the market. Both AREX and GoReplay highlight this feature prominently as a unique selling point. Therefore, this comparison will primarily focus on how each product leverages production traffic capture.

Traffic replay or traffic mirroring?

Gor implements production traffic replay through a technique known as traffic mirroring. This method involves shadowing live user traffic and redirecting it to a test environment. This can be done either in real-time or asynchronously using systems like Apache Kafka, file systems, and other such platforms.

AREX is a testing platform for traffic replay, which involves recording the live traffic from online environment and replaying it to the test environment for API testing.

The traffic replay mechanism employed by AREX, while similar to traffic mirroring, presents some key distinctions:

Persistence of Traffic Data: Unlike traffic mirroring, which doesn’t store the captured traffic, AREX's approach allows for the retention and replay of traffic at any desired moment. This feature is particularly beneficial for testing scenarios like Black Friday traffic surges.
Application Validation: Traffic mirroring lacks built-in application validation, requiring developers to write their own, often brittle and single-use, validation checks. In contrast, AREX addresses this gap.
Data Sanitization: Raw data captured through traffic mirroring typically requires sanitization to address issues such as authentication, timestamp adjustments, and the protection of sensitive information. AREX considers this essential step, ensuring the suitability of data for use in lower-level environments, a process often overlooked in basic traffic mirroring.

Ease of getting started

GoReplay is renowned for its straightforward setup process. Implementation is as simple as downloading a binary, involving fewer initial steps than AREX. However, when delving deeper into its usage, it becomes apparent that GoReplay lacks certain functionalities found in more specialized testing tools. This is especially noticeable when compared to AREX, which is purpose-built for API testing and offers a more comprehensive set of features.

A significant distinction lies in GoReplay's lack of a WebUI, necessitating interaction with the tool predominantly via the terminal. This contrasts sharply with AREX, which boasts a WebUI, offering a more intuitive and user-friendly interface. This graphical interface simplifies the management and configuration of the tool, enhancing the overall user experience.

Reliability

In the assessment of traffic replay solutions, reliability — the consistent replay of traffic — is a critical factor. It's crucial to understand that GoReplay operates differently from a proxy. Instead, it implements a system-level hook that records all HTTP socket traffic occurring on a specified port. Once this traffic is captured, replaying it involves using a command that is almost identical to the one used for recording:

// capture traffic 
gor --input-raw :8000 --output-file ./requests.gor 

// replay the traffic 
gor --input-raw :80 --output-http "http://target_server:8080"

While GoReplay natively integrates with Kafka for data storage, this setup presents notable limitations. Specifically, when using Kafka, data must be recorded and replayed simultaneously, which can compromise reliability.

In contrast, AREX emerges as a more comprehensive tool, offering seamless integration of traffic recording and playback with automated testing. It supports the replay of recorded data at any time and multiple times, enhancing both flexibility and reliability in its use.

Data sensitivity

Capturing production traffic often involves dealing with sensitive user information, necessitating stringent measures for data anonymization or removal during the capture process. In the case of Gor, its traffic mirroring approach results in the collection of raw data, which initially lacks sanitization, potentially raising security concerns during its operation.

Conversely, for users who place a high priority on data security, AREX stands out with its strong commitment to storing only desensitized data. This approach substantially increases the safety and privacy protection of the captured traffic.

API mocking

AREX distinguishes itself from GoReplay by offering the valuable ability to modify recorded data. This feature grants users the flexibility to tailor and adjust captured traffic to suit specific testing requirements. Such a capability is crucial for simulating diverse scenarios and edge cases, thereby elevating the comprehensiveness of your testing strategy.

On the other hand, GoReplay primarily concentrates on replaying captured traffic in its original form, lacking inherent functionality to alter the data during the replay phase.

Identify regressions and changes

With Gor, there's a necessity to author middlewares, for tasks such as comparing original and replayed data, monitoring latency fluctuations, and producing analytics from this data. This can add layers of complexity to the testing process.

In contrast, AREX not only provides traffic recording and playback features but also includes built-in comparison functionalities. These integrated features in AREX simplify the process of contrasting original and replayed data, monitoring performance changes, and extracting insightful analytics, streamlining the overall testing workflow.

AREX or Goreplay?

In conclusion, when deciding between AREX and GoReplay for traffic replay, the choice hinges on specific testing requirements and preferences. GoReplay offers simplicity and a straightforward approach, making it suitable for scenarios where basic traffic replay is needed without additional complexities. AREX, on the other hand, shines with its advanced features like data sanitization, traffic modification, and built-in comparison capabilities, catering to more intricate testing scenarios. Ultimately, the decision should align with your project’s needs, balancing simplicity against a comprehensive set of features for thorough and effective testing.

Community⤵️
View on GitHub
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

How to mock a method call with Java Agent?

Jing — Wed, 29 Nov 2023 06:00:59 +0000

Code example
The implementation in AREX
Record and replay in-memory data

AREX is an open-source automated regression testing platform based on real requests and data. It enables the accomplishment of bytecode injection with Java Agent to record real traffic in the production environment, and then replays the requests and injects mock data in the testing environment. This enables automatic recording, replaying, and comparison, providing convenience for interface regression testing.

In general, the basic principles of implementing mocking using Java Agent can be summarized as follows:

Start the application with an Java Agent loaded.
In the premain method of the Agent, use a Java bytecode manipulation library like ByteBuddy to modify the bytecode of the target class.
Locate the method or field in the target class where mocking needs to be applied within the bytecode.
Use the bytecode manipulation library to modify the implementation of the target method or field to incorporate the mocking logic. This can involve replacing the original method with a mock method or modifying the value of a field.
Redefine the modified bytecode as a new class and load it into the JVM.
During runtime, the modified class will be used, enabling the mocking functionality.

AREX Java Agent supports mocking for not only various technology frameworks, but also local time, cached data, and various memory data types. This allows for precise reproduction of the data environment during production execution during replay, without generating dirty data to your database.

This blog will delve into how AREX achieves automatic mocking of data during traffic replay from a code perspective.

Code example

Let's take a simple function as an example to understand the implementation. It's a function that converts a given IP string to an integer value. The code is as follows:

public Integer parseIp(String ip) {
    int result = 0;
    if (checkFormat(ip)) { // Check if the IP string is legal
        String[] ipArray = ip.split("\\.");
        for (int i = 0; i < ipArray.length; i++) {
            result = result << 8;
            result += Integer.parseInt(ipArray[i]);
        }
    }
    return result;
}

Now, let's explain from two perspectives, how to implement traffic playback with the function mock:

Record (collect traffic)

When this function is called, we collect the request parameters and corresponding results for later use in traffic replay. The code is as follows:

if (needRecord()) {
    // Data collection, save the parameters and results in DB
    DataService.save("parseIp", ip, result);
}

Replay

When traffic replay, the previously collected data is used to implement mocking of this function. The code is as follows:

if (needReplay()) {
    return DataService.query("parseIp", ip);
}

By examining the complete code, we can better understand its logic:

public Integer parseIp(String ip) {
    if (needReplay()) {
        // in the case of replay, the collected data is used as the return result, i.e., Mock
        return DataService.query("parseIp", ip);
    }

    int result = 0;
    if (checkFormat(ip)) {
        String[] ipArray = ip.split("\\.");
        for (int i = 0; i < ipArray.length; i++) {
            result = result << 8;
            result += Integer.parseInt(ipArray[i]);
        }
    }

    if (needRecord()) {
        // in the case of recording, save the parameters and results in DB
        DataService.save("pareseIp", ip, result);
    }
    return result;
}

The implementation in AREX

The implementation in AREX is similar but more complex. The developers don't need to add recording and replaying code in the business logic. The arex-agent automatically adds the necessary code in the required code. Let's take the Query of MyBatis3 as an example.

By reading the MyBatis source code, we should know that all Query operations are consolidated in the query method of the org.apache.ibatis.executor.BaseExecutor class (excluding Batch operations). The method signature is as follows:

public <E> List<E> query(MappedStatement ms, Object parameter, RowBounds rowBounds, ResultHandler resultHandler, CacheKey key, BoundSql boundSql) throws SQLException

This query method includes the SQL query and accompanying parameters. The function's result includes the data read from the database. It is obviously appropriate to collect data here, and during replay, the collected data can be used as the result to avoid actual operations performed on database.

Let's take a look at the code in AREX. For the sake of simplicity, some simplifications have been made, as shown below:

public class ExecutorInstrumentation extends TypeInstrumentation {
    @Override
    protected ElementMatcher<TypeDescription> typeMatcher() {
        // Full name of the class which need code injection
        return named("org.apache.ibatis.executor.BaseExecutor");
    }

    @Override
    public List<MethodInstrumentation> methodAdvices() {
        // The name of the method that requires code injection. Because the query method has multiple overloads, parameter verification is included.
        return Collections.singletonList(new MethodInstrumentation(
                        named("query").and(isPublic())
                                .and(takesArguments(6))
                                .and(takesArgument(0, named("org.apache.ibatis.mapping.MappedStatement")))
                                .and(takesArgument(1, Object.class))
                                .and(takesArgument(5, named("org.apache.ibatis.mapping.BoundSql"))),
                        QueryAdvice.class.getName())
        );
    }

    // injected code
    public static class QueryAdvice {

        @Advice.OnMethodEnter(skipOn = Advice.OnNonDefaultValue.class, suppress = Throwable.class)
        public static boolean onMethodEnter(@Advice.Argument(0) MappedStatement var1,
                                            @Advice.Argument(1) Object var2,
                                            @Advice.Argument(5) BoundSql boundSql,
                                            @Advice.Local("mockResult") MockResult mockResult) {
            RepeatedCollectManager.enter(); // prevent repeated collection of data caused by nested calls
            if (ContextManager.needReplay()) {
                mockResult = InternalExecutor.replay(var1, var2, boundSql, "query");
            }
            return mockResult != null;
        }

        @Advice.OnMethodExit(onThrowable = Throwable.class, suppress = Throwable.class)
        public static void onMethodExit(@Advice.Argument(0) MappedStatement var1,
                                  @Advice.Argument(1) Object var2,
                                  @Advice.Argument(5) BoundSql boundSql,
                                  @Advice.Thrown(readOnly = false) Throwable throwable,
                                  @Advice.Return(readOnly = false) List<?> result,
                                  @Advice.Local("mockResult") MockResult mockResult) {
            if (!RepeatedCollectManager.exitAndValidate()) {
                return;
            }

            if (mockResult != null) {
                if (mockResult.getThrowable() != null) {
                    throwable = mockResult.getThrowable();
                } else {
                    result = (List<?>) mockResult.getResult();
                }
                return;
            }            

            if (ContextManager.needRecord()) {
                InternalExecutor.record(var1, var2, boundSql, result, throwable, "query");
            }
        }
    }
}

Where QueryAdvice is the code that needs to be injected in the query method. The code injected through onMethodEnter will be executed at the beginning of the method, while the code injected by onMethodExit will be executed before the function returns the result.

It may be difficult to understand, so let's dump the code of BaseExecutor's query method after code injection and analyze it.

public <E> List<E> query(MappedStatement ms, Object parameter, RowBounds rowBounds, ResultHandler resultHandler, CacheKey key, BoundSql boundSql) throws SQLException {
        MockResult mockResult = null;
        boolean skipOk;
        try {
            RepeatedCollectManager.enter();
            if (ContextManager.needReplay()) {
                mockResult = InternalExecutor.replay(ms, parameter, boundSql, "query");
            }

            skipOk = mockResult != null;
        } catch (Throwable var28) {
            var28.printStackTrace();
            skipOk = false;
        }

        List result;
        Throwable throwable;
        if (skipOk) {
            // in the case of replay, the original query method body is no longer executed
            result = null;
        } else {
            try {
                // The source code of the query method in BaseExecutor(omitted here), and the only adjustment is to modify the code that is returned in the original method to assign the result to a variable named result
                result = list;
            } catch (Throwable var27) {
                throwable = var27;
                result = null;
            }
        }

        try {
            if (mockResult != null) {
                if (mockResult.getThrowable() != null) {
                    throwable = mockResult.getThrowable();
                } else {
                    result = (List)mockResult.getResult();
                }
            } else if (RepeatedCollectManager.exitAndValidate() && ContextManager.needRecord()) {
                InternalExecutor.record(ms, parameter, boundSql, result, throwable, "query");
            }
        } catch (Throwable var26) {
            var26.printStackTrace();
        }

        if (throwable != null) {
            throw throwable;
        } else {
            return result;
        }
    }

As you see, the code of onMethodEnter and onExit is inserted at the beginning and end respectively.

Record

When send a request to a service, AREX will decide whether to record this request based on the configured recording frequency. If recording is required, all 3rd dependencies accessed during this request will be recorded.

AREX will record this request by invoking the InternalExecutor.record(ms, parameter, boundSql, result, throwable, "query") method to store the returned result, core parameters, and other information into AREX's database.

Replay

When resend the recorded request, the function will not be executed actually. Instead, AREX will directly return the recorded result previously by invoking InternalExecutor.replay(ms, parameter, boundSql, "query").

Record and replay in-memory data

The previous example is idempotent function. For idempotent functions, since the returned result remains the same regardless of external factors, there is no need for collecting data and mocking during the recording and replay process.

For non-idempotent functions, the results returned by the request are different each time due to the influence of the external environment, such as local caches where different environment data may lead to different output results. In such cases, AREX can also record and mock the non-idempotent functions by configure it as dynamic classes.

Here, you can configure the class name, method name (optional, if not configured, it will be applied to all public methods with parameters and return values), and parameter types (optional) Once configured, the arex-agent will automatically inject similar code into the corresponding methods, enabling data collection and mocking during the replay process.

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

How to mock Apollot Configration with AREX Java Agent?

Jing — Thu, 16 Nov 2023 05:43:46 +0000

What is Apollo
- Principle of Apollo client implementation
Development Process
Recording Implementation
Replay Implementation
Version Differences
- Integration Testing

The current tools that support traffic replication for automated regression testing, such as tcpcopy and diffy, cannot be used for testing non-idempotent interfaces. This is because they record and replay traffic at the network layer outside of the application, which can only validate read-only pages. When it comes to testing interfaces that require writing to the database, it can result in dirty data and even affect the correctness of the business logic.

Unlike these tools, AREX implements traffic recording and playback internally within the application using AOP (Aspect-Oriented Programming). It utilizes Java Agent and bytecode enhancement technology to record the real traffic in the production environment. This approach avoids the impact of traffic replay on business data, making traffic replication regression testing suitable for various Java-based front-end and back-end business systems.

The basic principle of mocking in AREX is that the Java Agent intercepts the class loading process and substitutes the original class with a mock implementation.

Now most open-source components can be mocked with AREX Java Agent. This article will introduce how to mock Apollo Configuration with AREX Agent.

What is Apollo

Apollo is a reliable configuration management system. It can centrally manage the configurations of different applications and different clusters. It is suitable for microservice configuration management scenarios. And Configuration changes take effect in real-time.

Here's the official description of the Apollo base model:

Users modify and publish the configuration in the configuration center
The configuration center notifies Apollo clients of configuration updates
Apollo client pulls the latest configuration from the configuration center, updates the local configuration, and notifies the application

Principle of Apollo client implementation

The above diagram briefly describes the principle of Apollo client implementation:

The client and the server maintain a long connection to get the first push of configuration updates. (achieved through Http Long Polling)
The client also regularly pulls the latest application configuration from the Apollo Configuration Center server.
After the client gets the latest configuration of the application from the Apollo Configuration Center server, it will be saved in memory

Development Process

From the above figure, we can see that AREX only needs to support Apollo client recording and playback, i.e. Java application projects internally refer to the apollo-client component:

<dependency>
    <groupId>com.ctrip.framework.apollo</groupId>
    <artifactId>apollo-client</artifactId>
    <version>{apollo-client.version}</version>
</dependency>

Typically, there are three ways in which Apollo is commonly used in projects:

Spring Autowired annotation configBean (internally using EnableApolloConfig annotation)
Based on Apollo's built-in annotation ApolloConfig, such as the config object in the code
API Mode, such as the config1 object in the code below

@Autowired
ConfigBean configBean; // The first way, internally using EnableApolloConfig annotation

@ApolloConfig("TEST1.lucas")
private Config config; // The second way

private Config config1; // The third way, calling getAppConfig

public void test() {
    config1 = ConfigService.getAppConfig();

    System.out.println("timeout="+config.getProperty("timeout", "0"));
    System.out.println("switch="+config.getBooleanProperty("switch", false));
    System.out.println("json="+config.getProperty("json", ""));
    System.out.println("white.list="+config1.getProperty("flight.change.white.list", ""));
    System.out.println("configBean="+configBean);

    // Listening for Apollo Configuration Changes
    ConfigChangeListener changeListener = changeEvent -> {
        System.out.println("Changes for namespace:" + changeEvent.getNamespace());
    };
    config.addChangeListener(changeListener);
}

@Component
@Configuration
@EnableApolloConfig("TEST1.sofia")
public class ConfigBean {
    @Value("${age:0}")
    int age;

    @Value("${name:}")
    String name;

    @ApolloJsonValue("${resume:[]}")
    private List<JsonBean> jsonBean;

}

If AREX needs to implement the recording and playback of Apollo, it needs to be compatible with these three ways of use. By examining the Apollo source code, it is found that the first two modes, based on the annotations EnableApolloConfig and ApolloConfig, and the last mode, which calls the API, all ultimately create instances using ConfigService.getAppConfig(). This means that the underlying API is shared. Therefore, we can modify these underlying Apollo methods by inserting AREX bytecode, in order to achieve the goal of recording and playback.

Recording Implementation

All configuration items in Apollo are distinguished based on namespaces. In order to perform recording, we need to obtain all configuration instances, which means getting the config instance corresponding to each namespace. Further examination of the apollo-client source code reveals that the config instances are maintained in the Map<String, Config> m_configs of the DefaultConfigManager class.

However, several issues need to be considered:

The m_configs property is private and there is no related API to access it.
This instance is rarely called during business runtime, so it may not be possible to obtain m_configs through conventional Arex mocking methods.
After obtaining the m_configs instance, it is also necessary to obtain the m_configProperties in the Config class, as this is where the actual configuration data resides.

The dependency relationships in UML:

Therefore, it is advisable to use reflection to obtain all configurations and record them. This approach would only invoke reflection for recording during the initial startup or when configuration changes occur, which happens infrequently.

Another consideration is the timing of recording. Take the above code for example, the third way to use Apollo is to create the "config1" instance within the business interface "test()":

Config config1 = ConfigService.getAppConfig()

This can be regarded as an incremental configuration instance (compared to the first two ways where the full configuration instance is created at project startup using annotations). Therefore, we need to make sure that both can be recorded. The current approach is to perform recording at the "postHandle" point after the main entry servlet/Dubbo interface has completed the request and returned the result. This way, regardless of which way the "config" instance is created, we can obtain and record it.

If the Apollo configuration changes during recording, we can add modification code to the Apollo source code: com.ctrip.framework.apollo.internals.DefaultConfig#updateAndCalcConfigChanges method to listen for change events and reopen our recording switch. Turn on our recording switch so that we can record to the new configuration the next time.

When recording, it will generate a version number to distinguish the batch of test cases recorded in different periods, i.e., the version number acts as a batch concept, as shown in the timeline below:

Replay Implementation

Similar to the recording implementation, in the case of replay, by using reflection, the m_configProperties can be assigned values, and the mocked configuration can be used to override the real configuration.

There are several issues to consider:

How to trigger a configuration change listener method set by the application, such as the changeListener method in the Apollo usage above;
During replay, the long polling of Apollo for configuration changes may overwrite our configuration for replay, which needs to be avoided;
How to ensure the correctness of configuration data when replaying multiple versions of configurations?
How to restore the original configurations after replay?

Considering the above issues, the implementation of recording is not comprehensive for replaying and cannot meet these specific scenarios.

After a deep dive into the source code, we choose to modify the loadApolloConfig method in the com.ctrip.framework.apollo.internals.RemoteConfigRepository class. Before requesting the server configuration, we directly return our mocked configuration data. This allows us to leverage the existing mechanisms in Apollo to trigger the complete configuration update process.

Here's the solution:

Return the mocked configuration directly if the real Apollo-Server service is not called during playback.
After replay, the method is no longer mocked (the replay is considered complete if there is no replay activity for more than 1 minute), and then the normal logic is executed, i.e. the real configuration is used.

Since the long polling of Apollo is always running, if the replay activity is finished and Apollo detects that the server configuration is inconsistent with the configuration replayed by AREX, it will trigger the operation to update the local configuration, achieving the goal of restoring the original configuration.

Regarding the third point mentioned above, how to ensure the correctness of configuration data when replaying multiple versions of configurations?

The recording operation only occurs during the project startup and when there are changes in the configuration. The generated version number (UUID) is also used as the replay version number. If recorded multiple version numbers, the replay is done sequentially according to different numbers. This means that the version numbers generated by AREX are used to differentiate different versions of configurations. The implementation approach is to set the "releaseKey" attribute of the constructed Apollo configuration entity, com.ctrip.framework.apollo.core.dto.ApolloConfig , to our AREX version number every time the configuration is replayed. This ensures the correctness of playback for multiple versions of configuration data.

The "releaseKey" is a crucial field for communication between Apollo client and server. The server uses this field to determine if the configuration is consistent with the client. If not, return a new "releaseKey" value; otherwise, return a 304 status.

Version Differences

It is also necessary to be aware of the differences in source code between different versions of the decorated apollo-client.

Some methods or classes may have variations in different versions. Before deciding to modify underlying methods, it is better to examine the differences between different versions first. Otherwise, there is a possibility of failure due to inconsistencies between the Apollo client version of the user's project and the version modified by AREX Java Agent.

For example, the method we modify, com.ctrip.framework.apollo.internals.DefaultConfig#updateAndCalcConfigChanges has differences in the input parameters between versions 1.0.0 and 1.2.0.

v1.0.0

v1.2.0

When modifying this method, we need to ensure compatibility with such differences so that our injected bytecode can work properly across different versions.

It is recommended to choose a lower version when modifying with components in order to maintain as much compatibility with existing code as possible and to follow the open–closed principle.

Integration Testing

Now we need to perform integration testing between Agent and Schedule Service.

When a user starts to replay, the Schedule Service first groups all the configuration version numbers generated during recording. After grouping all the test cases within that time, the service switches the version number before each replay, informing the AREX agent that Apollo's configuration needs to be played back.

This response for version switch request is not considered as the replay result, but simply as a version pre-heating. The real replay process will be initiated once the configuration corresponding to the version number is successfully switched. The same process applies when switching to other version numbers, as shown in the following diagram.

The same batch of cases has the same version number, which only recorded or replayed once.

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

Reproduce production bugs locally with AREX

Jing — Tue, 07 Nov 2023 07:41:13 +0000

How to reproduce issues with AREX
Deploy and start AREX
- Step 1: Deploy AREX with Docker
- Step 2: Start the application with AREX Java Agent
Reproduce online issues locally
- Step 3: Record requests for bugs in the production environment
- Step 4: Debugging with local replay

In the software development cycle, software bugs are a natural occurrence. Reproducing and fixing issues in the development environment is a common activity for developers.

Reproducing production issues locally allows us to test and validate any proposed fixes before deploying them. This is essential for preventing regressions and ensuring the fix resolves the issue. It can also help identify and resolve other problems that may have been overlooked.

However, what often troubles developers is the differences in configuration and data between the production and local environments (such as data in the database, data in the cache, etc.), issues that occur in the production environment cannot be quickly reproduced in the local testing environment, making troubleshooting much more challenging.

How to reproduce issues with AREX

AREX is an automated regression testing tool, that can record traffic and data in the production environment by Java agent and replay them in the testing environment to identify the correlation between code changes and result differences.

AREX Java Agent can mock all external dependencies of the recorded real request during recording. This ensures that the data and behavior in the development environment are identical to the production environment, enabling issue reproduction and troubleshooting locally.

In this blog post, we’ll show you how to use AREX to reproduce issues. Let’s get started!

Deploy and start AREX

Step 1: Deploy AREX with Docker

git clone https://github.com/arextest/deployments.git 
cd deployments docker-compose up

Step 2: Start the application with AREX Java Agent

AREX Agent is the core component for traffic recording. Therefore, before using the recording feature, it is necessary to configure the AREX Agent for the application under test.

git clone git@github.com:arextest/arex-agent-java.git 
mvn clean install

After compilation, you will find a new folder named "arex-agent-jar" in the "arex-agent-java" directory, which contains two JAR files.

To configure the Java Agent for the application to be tested, add the Java Agent configuration to the startup of the Java application.

The complete environment variables are as follows:

JAVA_TOOL_OPTIONS='-javaagent:E:/github-arex/arex-agent-java/arex-agent-jar/arex-agent-0.1.0.jar'

Program arguments:

-Darex.service.name=arex-community-test6 -Darex.storage.service.host=10.5.153.1:8093 -Darex.enable.debug=true  -Dspring.datasource.url=jdbc:mysql://10.5.153.1:13306/community?useUnicode=true&characterEncoding=utf-8 -Dspring.datasource.username=arex_admin -Dspring.datasource.password=arex_admin_password -Dspring.redis.host=10.5.153.1 -Dspring.redis.port=16379

View the log at startup:

You will see the message "ArexJavaAgent installed" in the printout, indicating that the Agent has been successfully installed.

At the same time, you can see Java printing out command-line parameter information, such as Agent address and Storage service address, as indicated by the arrows in the above image.

Reproduce online issues locally

Step 3: Record requests for bugs in the production environment

A normal HTTP request and the response are as follows:

When a bug occurs in a production request, configure all the settings for the request in AREX, and click on "action.record" to enable recording this request, as shown in the figure below:

After clicking the button, a new key named "arex-force-record" with a value of "true" will be added to the Header List. Then save the changes.

Send the request and the response headers will generate a new key called "arex-record-id" with a value of "AREX-172-20-0-4-708656889122" (a unique value randomly generated by the AREX Agent). This indicates that the request has been successfully recorded and all the data that the request depends on in the production environment has been stored in AREX.

Step 4: Debugging with local replay

Create a new request and modify the environment of the URL to the local environment. In the AREX request headers, add a key "arex-record-id" with the value set to the previously generated "AREX-172-20-0-4-708656889122". After clicking save, send the request.

Now you can debug the issue step by step in your IDEA environment:

After debugging is completed, you can view the response.

The response message contains an "arex-replay-id" key with a value of "AREX-172-20-0-4-708658384473" (an ID randomly generated by the AREX Agent for analysis purposes). It means that the request has been successfully replayed.

The above is the complete process of using the AREX traffic recording feature to quickly reproduce and debug online issues locally.

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

A Real Automated Regression Testing Platform with Recording and Replay Testing (Open-Source)

Jing — Fri, 27 Oct 2023 09:07:45 +0000

As the company's businesses scale, the system is becoming increasingly complex. The R&D testing team is facing various challenges, including high business complexity, large data construction workload, full regression testing, high communication costs, large maintenance of test data, and automation of case management. Each of these will affect the efficiency and quality of the testing team, posing challenges to the software development process.

In summary, there are two difficulties: cost and complexity.

Cost: there will always be a trade-off between cost and quality. It's a quite challenge to ensure quality while iterating quickly and ensuring quality within limited inputs.
Complexity: when business rules accumulate over time, the complexity of business processes, rules, scenarios, and data processing increases exponentially after stacking, posing great challenges to the quality of testing work.

Despite the availability of many automation testing tools on the market today, there are still two problems:

Automated testing mainly focuses on automating execution. The maintenance work is still done manually, resulting in a less-than-optimal output ratio.
In scenarios where a large amount of test data needs to be constructed, regression testing involves a wide scope, releases are frequent, and data validation and writing are required, both manual and automated testing still face a huge workload in terms of maintaining test cases and data, and the pain points of testing have not been effectively addressed.

Replay Traffic Testing

Replay traffic refers to capturing the production traffic and replaying it in a test environment, allowing us to exercise updated systems to simulate actual production conditions.

We used this concept to build an automated testing platform AREX, which combines recording, playback, and comparison. (https://github.com/arextest).

Record: record the production requests as well as the data involved in the request processing.
Replay: replay the requests and mock the data involved in the invocation chain.
Comparative Analysis and Reporting: compare and analyze the responses from recording and replay, and get a comprehensive report for the analysis.

What is AREX?

AREX is an open-source automation test platform. It records real traffic in the production environment and mocks all invocations to third-party dependencies with Java Agent bytecode injection technology, then replays the request and mock data in the test environment. Furthermore, it compares the recorded and replayed response messages, and lists the differences, allowing developers and testers to quickly troubleshoot.

The process of replay traffic testing

Record

AOP (Aspect Oriented Programming) is a technique for building common, reusable routines that can be applied applicationwide. A Java Agent can be used to apply AOP principles by intercepting method calls and weaving in additional behavior. It provides the necessary infrastructure to implement AOP in Java applications.

In the flow of a request, the AREX Java Agent records the request information of Java applications in the production environment. It sends this information to the AREX Storage Service, which imports and stores the data in a MongoDB database.

Replay

During replay, the AREX Schedule Service retrieves the recorded data (requests) of the tested application from the database according to the user's specifications. It then sends interface requests to the target verification service. At the same time, the Java Agent returns the recorded responses of external dependencies (external requests/DB) to the tested application. After the target service completes the request logic, it returns the response message.

Comparative Analysis and Reporting

After replay testing, AREX compares the recorded response with the playback response to verify the correctness of the system logic. It then utilizes the comparison results to generate a testing report to review. Throughout this process, the AREX Cache Service (Redis) is responsible for caching mock data and comparison results during the replay process, improving the efficiency of the comparisons.

Technical challenges

Technology Stack of AREX Java Agent

Due to the good performance and code readability, we chose the Byte Buddy library to modify specific bytecode instructions from Java code.

In addition, we use SPI (Service Provider Interface), which is an API proposed to be implemented by a third-party provider. It can be used as an extension or replaceable by existing implementations. Our injection component is implemented through this plug-in model.

Tracing

While recording, AREX will capture the request and response messages of the main servlet and third-party dependencies as a complete test case. It's a challenge to trace these data.

To address this problem, the AREX Java agent generates a unique Record ID and saves it to the Thread Local variable. Injecting the agent code at the beginning and end of the application function, this code reads the value in the Thread Local variable and stores it together with the intercepted data.

Record and Replay Solution

Let's start with a simple function as an example to understand how to record and replay. Suppose we have the following function that converts a given IP string to an integer value:

public Integer parse(String ip) {
    int result = 0;
    if (checkFormat(ip)) { // Check whether IP address is legal
        String[] ipArray = ip.split("\\.");
        for (int i = 0; i < ipArray.length; i++) {
            result = result << 8;
            result += Integer.parseInt(ipArray[i]);
        }
    }
    return result;
}

Record (traffic collection)

When this function is called, we save the corresponding request parameters and the return result for later use in traffic replay. Here's the code:

if (needRecord()) {
    // Data collection, saving parameters and results in DB
    DataService.save("parseIp", ip, result);
}

Replay (traffic replay)

While replaying the traffic, we can use the previously collected data to automatically mock this function, without actually sending the request:

if (needReplay()) {
    return DataService.query("parseIp", ip);
}

By examining the complete code, we can better understand the logic :

public Integer parseIp(String ip) {
    if (needReplay()) {
        // replay scenes, use the collected data as the return result, which is essentially mock.
        return DataService.query("parseIp", ip);
    }

    int result = 0;
    if (checkFormat(ip)) {
        String[] ipArray = ip.split("\\.");
        for (int i = 0; i < ipArray.length; i++) {
            result = result << 8;
            result += Integer.parseInt(ipArray[i]);
        }
    }

    if (needRecord()) {
        // Recorded scenes, save the parameters and results to the database
        DataService.save("pareseIp", ip, result);
    }
    return result;
}

Compatible Component Versions

Popular components often have multiple versions that are used on different systems at the same time, and the implementations of the different versions may be very different or even incompatible.

To address this problem, AREX makes multiple versions compatible. In the application startup, the AREX Agent will capture all the dependent package information, such as the JAR package Manifest.MF file, from the Manifest to get the version information of the library, and then according to the version information to start the corresponding AREX injection code, thus realizing the realization of multiple versions of compatibility.

As shown in the following figure, the version range of the current injection script adaptation is set so that AREX can identify the versions of the components that the application depends on before these classes are loaded, and then later match the versions when the classes are loaded to ensure correct code injection.

Code Isolation

Since most scenarios are recorded in the production environment and replayed in the test environment, stability is of paramount importance. For the stability of the system and to prevent the Agent's code from affecting the code execution of the application under test, AREX has realized code isolation and interoperability.

The AREX core JAR is loaded in an independent ClassLoader, which is not interoperable with the user's application code. To ensure that the injected code can be accessed correctly at runtime, a simple modification to the ClassLoader is made.

Mock Time

Many business scenarios are time-sensitive. Imagine that during the replay of requests, the recorded time has expired, resulting in failed testing.

Here we implement currentTimeMillis() to proxy the original call of Java's currentTimeMillis(). While recording the traffic, capture the current time for each test case. By using a proxy for System.currentTimeMillis(), the time during replay will be replaced with simulated time that matches the time of recording, to mock time.

Mock Caches

Applications in real production environments often utilize different types of caches to enhance runtime performance. However, the variations in cached data can lead to inconsistent execution results.

These dynamic classes can be mocked in AREX, by accessing the local cache method configured as a dynamic class, equivalent to your customization of the method for mock, which will be recorded in the production environment where you configure the dynamic class method data, replay corresponding to the match out of the data returned.

What can you do with AREX?

Diversity validation

To verify the correctness of business logic after modifying the system, merely checking the return results is not enough. Usually, it is also necessary to verify the correctness of intermediate process data, such as whether the data content written to the database by the business system is correct.

In response to this, AREX also supports validating the data writing to the third-party dependencies.

During the recording and replay process, AREX will record the database requests sent to the outside by both the old and new versions of the system, and compare these two requests. If there are differences, they will be displayed in the report.

Since AREX mocks all requests to third-party dependencies, it supports the verification of data in databases, message queues, Redis, and even runtime memory data. Moreover, during the playback process, it does not generate calls to the database, so there is no dirty data.

Reproduce the Production Issues

In actual usage, AREX can also be used to quickly localize production issues.

After a production issue occurs, due to version differences, data differences, and other issues, it can be difficult for developers to reproduce on their local machines, and the cost of debugging is high and time-consuming.

By using AREX, you can force the recording of problematic cases (the response message will generate a unique Record ID) on the production environment. Then start your local development environment and add this Record ID to the request message header. This allows you to restore the recorded request and data on your local machine using the playback function, and then directly debug the production issue using local code.

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

Mock Testing: Write a Java Agent Plugin to Mock Your Custom Components

Jing — Tue, 17 Oct 2023 08:36:18 +0000

Mock Testing with Java agent
AREX Agent
Setting up the development environment
Step 1: Create DalClientModuleInstrumentation Class
Step 2: Bytecode Modification
Step 3: Implementation of recording and replay
Deployments
Debug

Mock Testing with Java agent

Mock testing is a software testing method used to simulate the behavior of certain components or dependencies in a system. The purpose of mock testing is to isolate the system under test and verify the correctness of interactions between the system and its dependencies.

Java Agent is a Java technology that allows application startup modification and enhancement of bytecode. With Java Agent, it is possible to dynamically modify the behavior of target classes at runtime, including intercepting method calls and modifying return values. This provides a possibility for implementing Mock.

By implementing a proxy class or using bytecode manipulation libraries like Byte Buddy or ASM, you can modify the behavior of the target classes. This allows you to return mock data or execute custom logic instead of the actual behavior of the dependencies.

By loading Java Agent into the target application, it can intercept and modify calls to open source components at application startup, thus enabling Mock of open source components. This can isolate the system under test during testing and verify the correctness of interactions between the system and its dependencies.

AREX Agent

AREX provides an out-of-box agent file that could be attached to any applications with Java 8+ and dynamically weaves solid bytecode into your existing code to record the real data of live traffic, and further use and replay it for mocking, testing, and debugging purposes.

Let's take a simple component DalClient as an example to illustrate how AREX Agent implements mocking.

Object result = dalClient.query(key)
if (result != null) {
    return result; // return the value when recording
} else {
    return rpc.search(); // no value to return thus call the remote interface when replaying

Setting up the development environment

Download the code and run mvn install.

git clone https://github.com/arextest/arex-agent-java.git 
mvn clean install // install arex-agent-java dependencies that may be needed into your local maven repository.

Create a new Java project from IntelliJ IDEA.

Now, add the AREX Agent dependencies to the pom file:

<dependencies>
    <!-- Some basic classes and tools used by arex-agent, such as recording and playback related components, configuration information, etc. -->         
    <dependency>
        <groupId>io.arex</groupId>
        <artifactId>arex-instrumentation-api</artifactId>
        <version>${arex.version}</version>
    </dependency>
    <!-- serialization related components -->         
    <dependency>
        <groupId>io.arex</groupId>
        <artifactId>arex-serializer</artifactId>
        <version>${arex.version}</version>
    </dependency>
    <!-- Replace it with the component you need to mock in the actual development. -->
    <dependency>
        <groupId>com.your.company.dal</groupId>
        <artifactId>dalclient</artifactId>
        <version>1.0.0</version>
        <scope>provided</scope>
    </dependency>
</dependencies>

Step 1: Create `DalClientModuleInstrumentation` Class

The arex-agent-java loads and instantiates plugins using the SPI (Service Provider Interface) mechanism. Therefore, we declare a class called DalClientModuleInstrumentation with the com.google.auto.service.AutoService annotation. This class serves as the entry point for modifying DalClient and will be recognized by the arex-agent-java (@AutoService).

@AutoService(ModuleInstrumentation.class)
public class DalClientModuleInstrumentation extends ModuleInstrumentation {
    public DalClientModuleInstrumentation() {
       // Plugin module name, you can specify a different version match if the code difference between different versions of your DalClient component is relatively large and to be supported in separate versions:
       // ModuleDescription.builder().name("dalclient").supportFrom(ComparableVersion.of("1.0")).supportTo(ComparableVersion.of("2.0")).build();
       super("plugin-dal");
    }

    @Override
    public List<TypeInstrumentation> instrumentationTypes() {
      // classes that have undergone bytecode modification to change the behavior of the DalClient component
      return singletonList(new DalClientInstrumentation()); 
    }
}

So when it's necessary to distinguish between version numbers?

If there're differences between two versions of the DalClient component, eg., the name of "invoke()" method in version 1.0.0 is changed to "invokeAll()" in version 2.0.0. In this case, the code modified by the Agent plugin cannot cover both versions of the DalClient framework simultaneously. Therefore, adapting the code specifically for different versions may be necessary.

The implementation details can refer to the adaptation logic for different Dubbo versions in the arex-instrumentation/dubbo/ module of the arex-agent-java project.

The version matching is implemented based on the contents of the META-INF/MANIFEST.MF file in the jar package of the component.

Step 2: Bytecode Modification

To modify the bytecode, create a new file named DalClientInstrumentation.java.

Modification of the DalClient source code is quite simple. Just find the underlying implementation methods, preferably using a common API, and then using bytecode manipulation tools like Byte Buddy to modify this API by adding our own code to implement the mocking functionality.

Here's the DalClient source code to be modified:

package com.your.company.dal;

public class DalClient {
    public Object query(String param) {
        return this.invoke(DalClient.Action.QUERY, param);
    }

    public Object insert(String param) {
        return this.invoke(DalClient.Action.INSERT, param);
    }

    private Object invoke(Action action, String param) {
      Object result;         
        switch (action) {
            case QUERY:
                result = "query:" + param;
            case INSERT:
                result = "insert:" + param;
            case UPDATE:
                result = "update:" + param;
            default:
                result = "unknown action:" + param;
        }     
        return result;
    }

    public static enum Action {
        QUERY,
        INSERT,
        UPDATE;
        private Action() {
        }
    }
}

In regular business projects, the DalClient is typically invoked using the dalClient.query(key) method. From the above source code, it can be observed that the underlying implementation is achieved through the invoke method.

Therefore, by modifying the invoke method and injecting recording and playback code, the functionality of the DalClientInstrumentation class can be summarized as follows:

public class DalClientInstrumentation extends TypeInstrumentation {
    @Override
    public ElementMatcher<TypeDescription> typeMatcher() {         
        // Path to the DalClient class to be modified
        return named("com.your.company.dal.DalClient"); 
    }

    @Override
    public List<MethodInstrumentation> methodAdvices() {
        ElementMatcher<MethodDescription> matcher = named("invoke") // the method tobe modified
                .and(takesArgument(0, named("com.your.company.dal.DalClient$Action"))) // The first parameter type of this method. This is to distinguish that there may be methods with the same name but different parameter types
                .and(takesArgument(1, named("java.lang.String"))); 
            // The InvokeAdvice class is the code we need to inject in the invoke method         
            return singletonList(new MethodInstrumentation(matcher, InvokeAdvice.class.getName())); 
    }
}

The MethodInstrumentation, ElementMatcher, named, takesArgument, and other methods in the above code are all the ByteBuddy API. AREX Agent edits bytecode with ByteBuddy for recording and replay. (See more details, refer to: https://bytebuddy.net/#/tutorial)

Step 3: Implementation of recording and replay

Here is the code injected into the DalClient#invoke method, implementing InvokeAdvice :

public static class InvokeAdvice {
    // OnMethodEnter is the operation that executes before the modified method (invoke) logic is called
    @Advice.OnMethodEnter(skipOn = Advice.OnNonDefaultValue.class, suppress = Throwable.class)
    public static boolean onEnter(@Advice.Argument(0) DalClient.Action action, // Obtaining a reference to the first parameter (action) of the modified method
                                  @Advice.Argument(1) String param, // Obtaining a reference to the second parameter (param) of the modified metho
                                  @Advice.Local("mockResult") MockResult mockResult) { // mockResult is the customized variables within this method 
        mockResult = DalClientAdvice.replay(action, param); // replay
        return mockResult != null && mockResult.notIgnoreMockResult();
    }

    // OnMethodExit is the operation that executes before the end of the modified method (invoke)
    @Advice.OnMethodExit(suppress = Throwable.class)   
    public static void onExit(@Advice.Argument(0) DalClient.Action action,
                              @Advice.Argument(1) String param,
                              @Advice.Local("mockResult") MockResult mockResult,
                              @Advice.Return(readOnly = false) Object result) { 
        // returned result of the method
        if (mockResult != null && mockResult.notIgnoreMockResult()) {
            result = mockResult.getResult(); // assigns the replayed result to the result variable
            return;
        }
        DalClientAdvice.record(action, param, result); // recording logic
    }
}

This class is designed to inject code before and after the modified invoke method is invoked, enabling the implementation of recording and replay functionality.

In which:

The skipOn = Advice.OnNonDefaultValue parameter indicates that if mockResult != null && mockResult.notIgnoreMockResult() is true (non-default value, where the default value for a boolean type is false), the original logic of the method will be skipped. In other words, the original method logic will not be executed, and instead, the mocked value will be returned. If it is false, the original method logic will be executed as usual.

The modified bytecode is shown below:

public class DalClientInstrumentation extends TypeInstrumentation {
    private Object invoke(DalClient.Action action, String param) {
        // replay          
        MockResult mockResult = DalClientAdvice.replay(action, param);
        if (mockResult != null && mockResult.notIgnoreMockResult()) {
            return mockResult.getResult();
        }

        // the original logic
        Object result;
        switch (action) {
            case QUERY:
                result = "query:" + param;
            case INSERT:
                result = "insert:" + param;
            case UPDATE:
                result = "update:" + param;
            default:
                result = "unknown action:" + param;
        }

        DalClientAdvice.record(action, param, result); // record

        return result;
    }
}

Similar to the Aspect-Oriented Programming (AOP), this class inserts our code before and after the method invocation. In the case of replay, it returns the mocked result without executing the original logic. For recording, it records the result before it returns.

The code of DalClientAdvice is shown below:

public class DalClientAdvice {
    // record
    public static void record(DalClient.Action action, String param, Object result) {
        if (ContextManager.needRecord()) {
            Mocker mocker = buildMocker(action, param);
            mocker.getTargetResponse().setBody(Serializer.serialize(result));
            MockUtils.recordMocker(mocker);
        }
    }

    // replay
    public static MockResult replay(DalClient.Action action, String param) {
        if (ContextManager.needReplay()) {
            Mocker mocker = buildMocker(action, param);
            Object result = MockUtils.replayBody(mocker);
            return MockResult.success(result); DalClientInstrumentation          }
        return null;
    }

    private static Mocker buildMocker(DalClient.Action action, String param) {
        Mocker mocker = MockUtils.createDatabase(action.name().toLowerCase());
        mocker.getTargetRequest().setBody(param);
        return mocker;
    }
}

The above is just a simple demo. To see more details, refer to the arex-instrumentation module of the arex-agent-java project for specific usage, which has a lot of implementations that modify various middleware.

Summarized in three steps:

Create a entry class called DalClientModuleInstrumentation, which will be loaded when arex-agent-java starts.
Create a class called DalClientInstrumentation to specify which class and method to modify, as well as when to execute the modifications, and inform the AREX Agent accordingly.
Create a class called DalClientAdvice to implement the actual recording and replay functionality, or any custom logic you want to implement.

Deployments

After completing the development, you can test whether your plugin works correctly by following these steps:

Run the command mvn clean compile package to generate the plugin JAR file (it will be located in the /target directory of your project by default).
In the arex-agent-java project, run the command mvn clean compile package to generate the arex-agent-jar directory (located in the root directory of the project).
Create a new folder named extensions in the arex-agent-jar directory to store the extension JAR files.
Place the generated plugin JAR file (e.g., plugin-dal-1.0.0.jar) into the extensions folder.

The AREX Agent loads all JAR files in the extensions folder as extension features during startup. The directory structure is as follows:

Now you can debug your plugin. Start your business application, making sure to attach the agent in the VM options:

The detailed parameters are as follows:

-javaagent:<your-project>\arex-agent-java\arex-agent-0.3.8.jar
-Darex.service.name=<your-service-name>
-Darex.storage.service.host=<arex-storage-service-ip:port>
-Darex.enable.debug=true

By starting the project in this way, the agent will be able to load the plugin package.

If the project references the internal DalClient component, you will be able to see the name of the plugin declared in DalClientModuleInstrumentation in the console after starting the project.

If the console outputs the log message [arex] installed instrumentation module: plugin-dal, it means that the plugin has been recognized and loaded successfully.

Next, test if the plugin is functioning correctly. As shown below, if an interface in the project invokes the query method of the DalClient component, you can observe whether the plugin can record or replay the invoke method of the DalClient component. If the recording is successful, the following log message will be printed:

Similarly, to test the replay functionality, include the header arex-record-id:AREX-10-32-179-120-2126724921 (the recordId generated during recording) in the request header. After the successful replay, the console will also output log messages related to [arex] replay category: Database, operation: query.

Debug

If the plugin is not working properly or not behaving as expected, you can troubleshoot from the following two points:

Check the bytecode in the arex-agent-jar/bytecode-dump folder。

The bytecode-dump folder holds the modified bytecode files where you can find the class files before and after modification here:

Drag and drop the above two files into your IDE to open them directly. Confirm whether the modifications were successful. You can compare the code logic before (left side) and after (right side) the plugin modifications, as shown in the comparison image below:

If the modifications were not successful, please refer to the source code of the project demo at GitHub.
If the code modifications were successful but the behavior is not as expected, you can debug the plugin source code in your IDE by following these steps:

Import the arex-agent-java project into the IDE of your project by following these steps: File → New → Module from Existing Sources

Select the local arex-agent-java project, and then choose Maven to import it.

Import the plugin project into the IDE of your project. Then you will have the source code for the business project, as well as the arex-agent-java and plugin-dal projects.

This way, you can set breakpoints in both of these projects to debug the code of the Agent and the plugin, as shown in the following image:

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

AREX: How to Debug API in the browser

Jing — Thu, 12 Oct 2023 07:00:59 +0000

What is AREX?
Challenge 1: Cross-Origin Restrictions
- Solutions
Challenge 2: API Debugging
- Solutions
Challenge 3: Binary serialization
- Solution
- Code Analysis
  - File to Base64
  - Base64 to File

Debugging APIs is vital in software development. It is a critical step to validate and test the effectiveness and correctness of application interfaces. However, traditional API debugging methods typically rely on separate tools or desktop applications, limiting the debugging process's flexibility and efficiency.

This article will use the open-source project AREX as an example to introduce how to implement API debugging functionality in the browser.

What is AREX?

AREX is an open-source automated regression testing platform based on real requests and data. It utilizes Java Agent technology and comparison techniques to achieve fast and effective regression testing through traffic recording and playback capabilities.

Challenge 1: Cross-Origin Restrictions

To debug API in the browser, the first challenge to overcome is handling the browser's cross-origin restrictions.

The browser cross-origin issue refers to the restrictions encountered when using JavaScript code to access resources from one domain's web pages to another domain's resources in web development. Browsers implement a security policy called the Same-Origin Policy to protect user information. This policy is a browser security feature that restricts how documents and scripts on one origin can interact with resources on another origin.

Due to the browser's security policy, HTTP requests cannot be sent from within the browser due to cross-origin restrictions.

Solutions

There are two common solutions to overcome this limitation: Chrome extension proxy and server-side proxy. Now we compare the two methods:

	Chrome extension	Server-side
local access	√	❌
speed	No request time loss	The overall process speed is influenced by the proxy interface.
Actual request	The request `Origin` will be modified to match the source of the Chrome extension	the same as the actual request

After weighing the options, AREX chose Chrome extension proxy.

The background of the Chrome extension has the ability to send cross-domain requests. Then we can intercept the requests received on the browser side and communicate with the background of the Chrome extension using window.postMessage (where communication also requires the content-script of a Chrome extension to serve as a data bridge).

The implementation is as follows:

In the client-side script:

Generate a random string and store it as a string in the variable tid.
Use the window.postMessage() method to send a message to other extensions. The message includes an identifier of type AREX_EXTENSION_REQUEST, tid, and the params parameter.
Add a message event listener receiveMessage to receive messages sent by other extensions.
In the receiveMessage function, check if the received message is of type AREX_EXTENSION_RES and if the tid matches the one sent in the previous message. If there is a match, remove the event listener.

In the content script:

Add a message event listener to receive messages sent from the page script or other extension programs.
Within the event listener, verify if the received message is of type AREX_EXTENSION_REQUEST. If so, utilize the chrome.runtime.sendMessage() method to dispatch the message to the background script.
After receiving a response from the background script, use the window.postMessage() method to send the response message back to the page script or other extension programs.

In the background script:

Use the chrome.runtime.onMessage.addListener() method to add a listener for receiving messages from content scripts or other extension programs.
Within the listener, you can handle the received messages and respond accordingly based on your requirements.

// arex
const tid = String(Math.random());
window.postMessage(
  {
    type: '__AREX_EXTENSION_REQUEST__',
    tid: tid,
    payload: params,
  },
  '*',
);
window.addEventListener('message', receiveMessage);
function receiveMessage(ev: any) {
  if (ev.data.type === '__AREX_EXTENSION_RES__' && ev.data.tid == tid) {
    window.removeEventListener('message', receiveMessage, false);
  }
}
// content-script.js
window.addEventListener("message", (ev) => {
  if (ev.data.type === "__AREX_EXTENSION_REQUEST__"){
    chrome.runtime.sendMessage(ev.data, res => {
      //  communicate with background
      window.postMessage(
        {
          type: "__AREX_EXTENSION_RES__",
          res,
          tid:ev.data.tid
        },
        "*"
      )
    })
  }
})
// background.js
chrome.runtime.onMessage.addListener((req, sender, sendResponse) => {

})

Challenge 2: API Debugging

The above has resolved the cross-origin restrictions. The next challenge is how to implement the API debugging feature in the browser.

Solutions

Postman is a mature API debugging tool. By standing on the shoulders of giant, we introduced the JavaScript sandbox of Postman in AREX. We utilize the sandbox to run pre-request scripts, post-request scripts, and assertions for API debugging.

Here is the flowchart of request in AREX :

When sending the request, the data in the form will be gathered together into a data structure as below:

export interface Request {
  id: string;
  name: string;
  method: string;
  endpoint: string;
  params: {key:string,value:string}[];
  headers: {key:string,value:string}[];
  preRequestScript: string;
  testScript: string;
  body: {contentType:string,body:string};
}

This is the data structure of AREX, which will be converted into Postman's. Then, invoke the PostmanRuntime.Runner() method, and pass the converted Postman data structure and the currently selected environment variables. The Runner will execute the preRequestScript and testScript . The preRequestScript executes *before* a *request* runs and can include other requests as well as operations on request parameters and environment variables. The testScript is after the request and can perform assertions on the response. Additionally, the scripts can use console.log to output data for debugging in the console.

var runner = new runtime.Runner(); // runtime = require('postman-runtime');

// A standard Postman collection object
var collection = new sdk.Collection();

runner.run(collection, {}, function (err, run) {
    run.start({
      assertion:function (){},
      prerequest:function (){}, 
      test:function (){}, 
      response:function (){} 
    });
});

The cross-origin issues also exist in the Postman sandbox. Due to the high integration of the Postman sandbox, we have adopted Ajax (Asynchronous JavaScript and XML) interception technique to ensure convenience and synchronization with PostmanRuntime. By intercepting Ajax requests on the browser side, we can modify requests, add custom logic, or perform other processing operations.

When the Postman sandbox sends a request, it carries a request header "postman-token". After intercepting Ajax's request, we assemble the request parameters and send them to the browser extension via window.postMessage. The browser extension then constructs a fetch request and returns the data to the Postman sandbox. Then Postman sandbox outputs the final results, including the response, test results, and console.log. The responseType should be specified as "arraybuffer".

The specific process:

Register a request handler using the xspy.onRequest() method. This handler accepts two parameters: request and sendResponse. The request parameter contains relevant information about the request, such as the method, URL, headers, request body, etc. sendResponse is a callback function used to send a response back to the requester.
In the handler, check if there is postman-token in the request headers to determine if the request is from Postman.

If the request is from Postman, use AgentAxios to send a new request with the same details. The response is stored in agentData. Create a dummyResponse object with relevant information from the original request. Set the status, headers, ajaxType, responseType, and response fields of dummyResponse based on agentData. Finally, return the dummyResponse using sendResponse().
If not, then call sendResponse() directly, indicating that no response should be returned.

xspy.onRequest(async (request: any, sendResponse: any) => {
  // check if is sent from postman
  if (request.headers['postman-token']) {
    const agentData: any = await AgentAxios({
      method: request.method,
      url: request.url,
      headers: request.headers,
      data: request.body,
    });
    const dummyResponse = {
      status: agentData.status,
      headers: agentData.headers.reduce((p: any, c: { key: any; value: any }) => {
        return {
          ...p,
          [c.key]: c.value,
        };
      }, {}),
      ajaxType: 'xhr',
      responseType: 'arraybuffer',
      response: new Buffer(JSON.stringify(agentData.data)),
    };
    sendResponse(dummyResponse);
  } else {
    sendResponse();
  }
});

Challenge 3: Binary serialization

For requests with data formats of x-www-form-urlencoded and Raw, handling them is relatively easy since they are regular JSON objects. However, for requests of form-data and binary, they need support for sending binary payload. Unfortunately, the postMessage communication method in Chrome extensions does not support directly passing binary files, which makes it impossible to handle these two types of requests directly.

Solution

To address this issue, AREX utilizes the base64 encoding technique. When a user selects a file, AREX converts the binary file into a base64 string for transmission. On the Chrome extension side, AREX decodes the base64 data and uses it to construct the actual fetch request. This approach bypasses the limitation of directly passing binary files.

This flowchart describes the process of converting binary files in FormData to Base64 strings, and then converting them back to files and further processing them through the Chrome proxy extension.

form-data binary: a FormData contains a binary file.
FileReader: Uses a FileReader object to read the binary file.
readAsDataURL base64 string: The FileReader uses the readAsDataURL method to read the binary file as a Base64 string.
Chrome extension proxy: After the Base64 string is read, it is passed to the Chrome proxy extension.
base64 string: Represents the Base64 string obtained after the binary file is read by the FileReader.
Uint8Array: In the Chrome extension proxy, the Base64 string is converted to a Uint8Array.
File: A new File object is created using the data from the Uint8Array.
fetch: The new File object is further processed using the fetch method or other means, such as uploading to a server or performing other operations.

Code Analysis

Now let's look at the code level:

// File to Base64
const toBase64 = (file: File): Promise<string> =>
  new Promise((resolve, reject) => {
    const reader = new FileReader();
    reader.readAsDataURL(file);
    reader.onload = () => resolve(reader.result as string);
    reader.onerror = reject;
  });
// Base64 to File
function base64ToFile(dataurl: string, filename: string) {
  const arr = dataurl.split(',') || [''],
    mime = arr[0].match(/:(.*?);/)?.[1],
    bstr = atob(arr[1]);
  let n = bstr.length;
  const u8arr = new Uint8Array(n);
  while (n--) {
    u8arr[n] = bstr.charCodeAt(n);
  }
  return new File([u8arr], filename, { type: mime });
}

export default base64ToFile;

File to Base64

The toBase64 function takes a File object as input and returns a Promise that resolves to a Base64 string representing the file.

Inside the function, a FileReader object is used to read the contents of the File object as a Data URL by calling reader.readAsDataURL(file).

The onload event handler is set on the FileReader object, which will be triggered when the file reading operation is complete. Inside the event handler, the reader.result property, which contains the result of the file reading operation, is resolved as a string and passed to the resolve function of the Promise.

If an error occurs during the file reading operation, the onerror event handler is triggered, and the reject function of the Promise is called with the error.

Base64 to File

The function base64ToFile takes two parameters: dataurl (a Base64 string) and filename (the name of the file). The function returns a File object that represents the file with the specified name and content.

Inside the function, the Base64 string is split into an array using the comma separator. The first element of the array contains the MIME type of the file, which is extracted using a regular expression. The second element of the array contains the Base64-encoded content of the file, which is decoded using the atob function.

The decoded content is then converted to a Uint8Array object, which is used to create a new File object. The File object is returned with the specified filename and MIME type.

The above is the internal mechanism of the entire AREX debugging process.

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

Production Traffic Replication with Java Agent

Jing — Mon, 09 Oct 2023 09:09:05 +0000

Production traffic replication is a technique used to record traffic in production environment, and replaying it in another. It is also considered the best solution for performing regression testing.

Record and playback testing tools

Depending on the location where the recording takes place, all the recording tools can be categorized as: web server-based recording, application layer-based recording, and network protocol stack-based recording.

Web server-based recording

HTTP request recording based on web server refers to the process of recording and replicating the requests and responses between the web server and client.

Advantages: Supports a diverse range of request formats and protocols.

Disadvantages: High maintenance cost, consumes a significant amount of online resources.

Internet protocol stack-based recording

Directly monitoring to network ports and recording by duplicating data packets.

Advantages: Low impact on the application

Disadvantages: More low-level implementation, higher cost. In addition, it cannot be used to test non-idempotent interfaces, because traffic replay can result in the generation of invalid or dirty data, potentially impacting the correctness of the business operations.

Representative tools: goReplay, tcpCopy, tcpReplay

Application layer-based recording

Since Spring Boot is widely used as a Java backend framework, we can leverage AOP (Aspect-Oriented Programming) to intercept the Controller layer and record traffic.

Advantages: Non-intrusive to the code, relatively quick and simple implementation. In addition to validating the returned responses from server, it also supports validation of data written to the service, validation of database, message queue, Redis data, and even validation of runtime in-memory data.

Disadvantages: May consume some online resources, potential impact on business

Representative tools: ngx_http_mirror_module, Java sandbox, AREX

AREX is an automated regression testing platform based on traffic recording and playback. It utilizes Java Agent and bytecode enhancement technology to capture the entry points of real request chains and the request and response data of their 3rd-party dependencies in the production environment. Then, in the testing environment, it simulates to replay these requests and verifies the correctness of the entire invocation chain logic one by one.

How dose AREX Java Agent work?

AOP (Aspect-Oriented Programming) is a programming paradigm that provides a way to modularize cross-cutting concerns, such as logging, security, and transaction management. It achieves this by separating these concerns from the core business logic, allowing you to add extra functionality to existing modules without directly modifying their code.

Java Agent is a mechanism in Java that allows you to dynamically modify the bytecode of classes at runtime. It provides a way to instrument and manipulate the behavior of Java applications, including adding aspects or intercepting method calls.

In the context of AOP, a Java Agent can be used to apply AOP principles by intercepting method calls and weaving in additional behavior. It provides the necessary infrastructure to implement AOP in Java applications.

As shown in the figure below, a request typically has a chain of calls consisting of an entry point and dependencies that are either synchronous or asynchronous.

The recording process is to connect the entry and dependency calls through a RecordId to form a complete test case. AREX-Agent enhances the bytecode of the entry and dependency calls, intercepts the call process when the code is executed, and records the entry parameter, return value, and exceptions of the call, and sends them to the storage service.

During playback in the test environment, the AREX Agent simulates requests using the real data recorded in the production environment. The AREX Agent determines whether playback is required by identifying the playback flag. If playback is required, the actual method invocation is not performed. Instead, the stored response data from the storage service is retrieved and returned as the response.



public Integer parseIp(String ip) {
    if (needReplay()) {
        //  In the playback scenario, the collected data is used as the return result, which is also known as Mocking.
        return DataService.query("parseIp", ip);
    }

    int result = 0;
    if (checkFormat(ip)) {
        String[] ipArray = ip.split("\\.");
        for (int i = 0; i < ipArray.length; i++) {
            result = result << 8;
            result += Integer.parseInt(ipArray[i]);
        }
    }

    if (needRecord()) {
        // In the recording scenario, the parameters and execution results are saved into the database.
        DataService.save("pareseIp", ip, result);
    }
    return result;
}

Taking the above function as an example:

At the beginning of the function, a decision is made whether to playback or not. If playback is necessary, the collected data is utilized as the return result, which is commonly referred to as Mocking.

At the end of the function, a decision is made whether to record or not. If recording is necessary, the intermediate data that the application needs to store is saved to the AREX database.

Technical challenges

The process of recording and replay is very complex. Next, we will dive into the challenges and technical details of AREX Agent.

ClassLoader Isolation

To ensure the AREX Agent code and dependencies do not have a conflict with the application code, the AREX Agent and application code are isolated by different class loaders. As shown in the figure below, AREX Agent overrides the findClass method by customizing AgentClassLoader to ensure that the classes used by AREX Agent will only be loaded by AgentClassLoader, so as to avoid conflicts with the application ClassLoader.

Meanwhile, in order to let the application ClassLoader recognize the recording and playback code of AREX Agent, AREX Agent injects the byte code needed for recording and playback into the application ClassLoader through the ByteBuddy ClassInjector to make sure there is no ClassNotFoundException/NoClassDefFoundError during recording and replay.

Tracing

When the data is recorded and replayed, the entry point of a request and the calls of each dependency will be linked together by a RecordId. When dealing with multi-threading and various asynchronous frameworks, there are significant challenges in maintaining data continuity. AREX Agent addresses the issue of passing RecordId across threads by enhancing the thread behavior, ensuring seamless transfer of RecordId between threads. The supported threads and thread pools are as follows:

Thread
ThreadPoolExecutor
ForkJoinTask
FutureTask
FutureCallback
Reactor Framework
……

Here's a simple example for better understanding of implementation.

When invoking java.util.concurrent.ThreadPoolExecutor#execute(Runnable runnable), the parameter AgentRunnableWrapper is used to wrap the AgentRunnableWrapper runnable. When constructing AgentRunnableWrapper, the current thread context is captured. In the run method, the subthread context is replaced, and after execution, the child thread context is restored. Here is an example code snippet:



executors.execute(Runnable runnable)
executors.submit(Callable callable)

public void execute(Runnable var1) {
var1 =RunnableWrapper.wrap(var1);
}

public class RunnableWrapper implements Runnable {
  private final Runnable runnable;
  private final TraceTransmitter traceTransmitter;

  private RunnableWrapper(Runnable runnable){
    this.runnable = runnable;
    //Capture the current thread context 
    this.traceTransmitter = TraceTransmitter.create();
    }

  @Override
  public void run(){
    //Replacing the subthread context 
    try (TraceTransmitter tm = traceTransmitter.transmit()){
      (runnable.run();
    }
    //Reducing the Atomic Thread Context 
  }
}

...

Component Version Compatibility

The components introduced by an application may have multiple versions, and different versions of the same component may be incompatible, such as changes in package, addition or removal of methods, etc. In order to support multiple versions of components, AREX Agent needs to identify the correct component version for bytecode enhancement, to avoid duplicate enhancement or enhancement of the wrong version.

AREX Agent identifies the Name and Version in the META-INF/MANIFEST.MF of the component JAR file, and matches the version during class loading to ensure that the correct version is used for bytecode enhancement.

Mock Local Cache



Object value = localCache.get(key)
// Cache is available during recording, but not available during playback in which case the code needs to query the database (db.query()).
if (value != null) {
    return value;
} else {
    return db.query();
}

As shown above, during recording, the code first attempts to retrieve the value associated with the given key from the local cache (localCache.get(key)). If the value is not null, it means that the corresponding data is available in the cache during recording, and it is directly returned.

However, during playback, the cache is not available. Therefore, if the value retrieved from the cache is null, it means that the data is not present in the cache during playback. In this case, the code needs to query the database (db.query()) to retrieve the data and return it as the result.

In a word, the execution flow of the replay request is often different from the recording due to inconsistent local cache data with the recording, resulting a low pass rate of replay testing. There are a few challenges to solve this problem:

It is challenging to achieve real-time synchronization between production and test cache data due to the isolation between them.
Local memory is implemented in various ways, and it is impossible to perceive each one individually.
Local memory data is typically fundamental data and can have a large volume. Recording this data can lead to significant performance overhead.

Currently, the solution adopted by AREX Agent is to only record the cache data used in the current request chain. This is achieved by allowing the application to configure dynamic classes to identify the recording, and during playback in the test environment, the cache data is automatically replaced to ensure consistency between the recorded and playback memory data. We are still researching the solution of recording large cache data.

Time Mock

Many business systems are time-sensitive, where accessing them at different times can result in different outcomes. If the recording and playback times are inconsistent, it can lead to playback failures. Additionally, modifying the machine time on the test server is not suitable as playback requests are concurrent, and many servers do not allow modification of the current time. Therefore, we need to implement mocking of the current time at the code level to address this issue.

The currently supported time types are as follows:

java.time.Instant
java.time.LocalDate
java.time.LocalTime
java.time.LocalDateTime
java.util.Date
java.util.Calendar
org.joda.time.DateTimeUtils
java.time.ZonedDateTime

public static native long currentTimeMillis() is an intrinsic function. When the JVM performs inline optimization on intrinsic functions, it replaces the existing bytecode with internal code (JIT), which causes the enhanced code by AREX Agent to become ineffective. The JDK performs inline operations on System.currentTimeMillis() and System.nanoTime() as follows:



// // https://hg.openjdk.org/jdk8u/jdk8u/hotspot/file/dae2d83e0ec2/src/share/vm/classfile/vmSymbols.hpp#l631
//------------------------inline_native_time_funcs--------------
// inline code for System.currentTimeMillis() and System.nanoTime()
// these have the same type and signature
bool LibraryCallKit::inline_native_time_funcs(address funcAddr, const char* funcName) {
  const TypeFunc* tf = OptoRuntime::void_long_Type();
  const TypePtr* no_memory_effects = NULL;
  Node* time = make_runtime_call(RC_LEAF, tf, funcAddr, funcName, no_memory_effects);
  Node* value = _gvn.transform(new ProjNode(time, TypeFunc::Parms+0));
#ifdef ASSERT
  Node* value_top = _gvn.transform(new ProjNode(time, TypeFunc::Parms+1));
  assert(value_top == top(), "second value must be top");
#endif
  set_result(value);
  return true;
}

AREX Agent has taken special care of this issue by replacing the code that uses the method System.currentTimeMillis() with AREX Agent's method of obtaining the time directly through the application configuration, avoiding inline optimizations.

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

The Coolest Record and Playback Testing tool to start automating tests

Jing — Wed, 27 Sep 2023 06:33:49 +0000

What Is Record and Playback Testing

Record and playback testing is regarded as a low-code method that automates software testing without writing test scripts.

When you manually send requests on an application under test (AUT) in the production environment, such as a web application, the tool will capture these requests and automatically converts them into test cases.

You can then “replay” these requests in test environment to make sure that they function correctly after any changes have been made to the software.

Advantages of Record and Playback Testing

Record and playback testing does not require knowledge of coding or the ability to execute programming languages, making it accessible to users of all skill levels. Even individuals with no programming background, such as product managers, designers, marketers, and others, can participate in.
It’s the easiest way for regression testing as it greatly reduces the time and effort needed to create test scripts. By recording requests as test cases, there may be no need to write or maintain a regression suite.
Replaying online traffic can perfectly simulate real user behavior, avoiding the discrepancies that may happen in manual writing test cases.

Challenges of Record and Playback Testing

Limited Functionality – Most record and playback testing tools are unable to verify whether the functionality is correctly executed. They can only reuse recorded steps to create another test case for playback.
Troubleshooting – Once a test played, if it failed, there was little information to help QA determine why it failed. Was it poorly written? Did it fail because of an element loaded slowly? Or was there actually a bug in the application?
Maintenance – The recorded tests might easily become broken when your application changes frequently. Then you need to re-record the tests or update the test scripts manually, dropping your testing efficiency.
Restrictive - The capabilities of most record and playback testing tools are limited to recording tests while performing manual actions on web pages. However, in reality, testing software involves more than just operations on a website.

Choosing the right testing tool

Over the years, the record and playback testing method has not been widely used due to the failure of many tools to address the aforementioned challenges.

Despite facing initial challenges, record and playback testing has evolved into a more advanced and intelligent application testing tool with the help of modern tools.

AREX is an open-source automation test platform. It record real requests in the production environment and mock all requests to third-party dependencies with Java Agent bytecode injection technology, then replay the request and mock data in the test environment. Furthermore, it compares the recorded and replayed response messages, and lists the differences, allowing developers and testers to quickly troubleshoot.

With AREX, you can accelerate your testing processes and identify defects and errors earlier in the development cycle, which can ultimately save time and resources. As an open source tool, AREX is constantly evolving and improving thanks to contributions from its community of users.

Low Cost

Low integration cost. No need to write test cases, while massive online requests can also ensure high coverage. Instrumentation code is simple enough for low performance loss.

Diversity validation

In addition to validating the returned responses from server, it also supports validation of data written to the service, validation of database, message queue, Redis data, and even validation of runtime in-memory data.

Intelligent Mock

Supports mocking for multiple popular technical frameworks, as well as local time and cache. This ensures accurate restoration of the production data environment during playback in testing environment.

One-click debugging online problems in your local environment.

Due to configuration and data differences between the online and local environments, it is often challenging to reproduce online issues quickly in a local testing environment, making troubleshooting more difficult. With AREX, as all third-party dependencies can be mocked, it becomes convenient to debug locally.

Safe and stable

Safe: Desensitize the data for some sensitive information to protect of sensitive private data when recording the real traffic in the production environment.

Stable: Code isolation. Automatically reduce or turn off data collection during high system activity, which ensures that system stability is not affected.

Try AREX: http://arextest.com/docs/chapter1/quick-installation

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

Complete Guide of Regression Testing: Definition, Tools & How to do

Jing — Thu, 21 Sep 2023 07:21:08 +0000

What is regression testing?

Regression testing refers to testing an application to ensure that the previously code remains operational as expected after changes or enhancements made to it. It is a quality control measure that helps developers and testers identify potential defects or problems that may occur with new code additions, design modifications, or system upgrades before a new feature markets.

It should be noted that regression testing is more than just simply re-testing. Re-testing concentrates solely on the test cases that have failed, whereas regression testing is used to examine the test cases that have passed to identify any unanticipated new bugs.

When to perform regression testing?

Typically, regression testing is performed after any changes have been made to the software to verify the changes have not introduced any new issues or affected any previously working functionality, like the following scenarios:

After making changes to the software.
After fixing a defect.
After updating to the operating system, browser, or other external applications occur that may affect product functionality.
After a major release.

How to perform Regression Testing?

Regression testing can be executed both manually and in an automated manner. Test Engineers primarily use special techniques and methods to perform Regression tests.

Identify the changes: Identify the changes made to the system under test, such as code modifications, enhancements, bug fixes, etc.
Select the test cases: Select the test cases that cover the modified or impacted functionalities by analyzing the requirements and design documents, test cases, and bug reports.
Prioritize the test cases: Prioritize the selected test cases based on their criticality and impact on the system.
Create a regression test suite: Create a regression test suite that includes the selected test cases.
Execute the test cases: Execute the test cases in the regression test suite and compare the actual results with the expected results.
Report the defects: Report any defects identified during the test execution and ensure that they are fixed by the development team.
Repeat the cycle: Repeat the above steps whenever there are new changes made to the system under test to ensure that the system is functioning as expected.

Challenges of regression testing

Large Scope and Coverage

There are challenges faced while listing out functionalities to be included in the test suite and possible scenarios. Regression testing may have limited test coverage, as it may not be possible to test all possible scenarios due to time and resource constraints.

Maintenance and Optimization

Maintaining and optimizing the existing regression test suites is a significant task. When there are new modifications, the current regression test suites may need to be modified by adding, eliminating, or altering the existing test cases. All of this must be completed before the regression testing deadline.

Validate database, cache, or message queue

To validate business correctness, it is not sufficient to simply validate the returned results from server. In some scenarios such as generating orders and making payments that require writing data to the database, message queues, cache, etc., it is necessary to validate whether the interaction requests between the business system and the underlying business system have changed, whether the content of the data written to the database by the business system is correct, whether the content of the asynchronous messages sent by the business system is correct, and so on.

Use the right testing tool

There are multiple ways to overcome the challenges in regression testing. One of the most straightforward approaches is record and replay testing. This low-code method involves using a tool to automate tests without writing test scripts. As a result, it is often considered the simplest way to perform regression testing.

There are different tools and frameworks that can be used to perform Regression Testing. Such as:

Selenium IDE: Selenium IDE is a user-friendly and free tool for recording and playing back web application tests. With its simple installation process and browser extension, you can quickly start recording your testing activities. However, a significant drawback is that the tool has limited capabilities in terms of scalability.
TestComplete: TestComplete is another pre-built tool that includes a record and replay functionality, as well as scripting capabilities and advanced features such as parallel or keyword-driven testing, object recognition engine, and robust reporting. These features can help your team enhance its testing capabilities and improve overall efficiency.
Katalon: Tthe record and replay feature is readily available and easy to use, with a simple interface. The tool also includes a built-in object repository that follows the Page-Object model, making it easy to organize and maintain test objects after recording tests. You can start utilizing this feature immediately without any additional setup.
AREX: It is an open-source automated regression testing platform, which allows one to record production traffic and replay it to testing environment. By comparing the recorded responses against replay, it can clarify the impact of code changes to realize efficient automated regression testing.

Regression testing with AREX

Record production traffic as test cases

Step 1: Install AREX

git clone https://github.com/arextest/deployments.git
cd deployments
docker-compose up -d

Step 2: Modify the way your production environment for the application under test runs to the following command

java -javaagent:/path/to/arex-agent-0.1.0.jar -Darex.service.name=your-service-name -Darex.storage.service.host=[storage.service.host:port](storage.service.host:port) -jar your-application.jar

Where：

/path/to/arex-agent-0.1.0.jar needs to be changed to the actual path of the arex-agent-0.1.0.jar file in your server
your-service-name needs to be changed to the name of your application under test
[storage.service.host:port] needs to be changed to the address and port number of the data storage service
your-application.jar needs to be changed to the jar package name of the application

Or change the environment variables in the environment

export JAVA_OPTS="-javaagent:/path/to/arex-agent-0.1.0.jar -Darex.config.path=/path/to/arex.agent.conf"

Replay traffic in the testing environment

Step 3: Visit the AREX UI from browser, and in the Replay tab, select the service with AREX Agent(i.e. your-service-name above).

Step 4: Select Replay, enter the Target Host of your test environment and start replaying all the recorded test cases.

After the replay is completed, a detailed replay report will be generated, where you can visually see the pass rate, API pass rate, and the details under each API.

Click the failed API to see the comparison results, and AREX visualized the comparison results to facilitate troubleshooting.

A Comprehensive Guide of API Testing with AREX

Jing — Mon, 18 Sep 2023 08:15:20 +0000

What is API?

API stands for Application Programming Interface. It’s a connection between different systems or applications that enables them to communicate with each other.

Just like ordering food at a restaurant. You only need to tell the waiter what you want, then the waiter will report it to chef and delivers the meal from kitchen. In this process, the waiter plays the role of an API. Similarly, when you use an API, you only need to call the required functions and services, without needing to know the underlying code implementation. Therefore, an API acts as a “middleman” between different applications, allowing them to communicate and interact with each other.

In contrast to monolithic architectures, microservice is favored by more developers due to the highly flexibility. It promote the decompositon of the application into smaller, isolated services, which allows for each core function within an application to exist independently.

The emergence of microservice architecture has also led to a surge in the number of APIs.

Why API Testing?

With the explosion growth of APIs, the quality of APIs becomes increasingly important. Once an API breaks down, the complete application and user experience are put at risk. Only proper API testing can secure the system from such downtime possibilities.

API testing involves testing the collection of APIs and checking if they meet expectations for functionality, reliability, performance, and security and returns the correct response. It helps developers detect and fix potential problems before deploying code to the production environment, thereby improving the availability and reliability of the entire system. Especially in microservice architecture, the applications always undergo frequent changes and updates.

HTTP Protocol

The most common Web Service APIs include SOAP, REST, and RPC. RESTful API is an API that conforms to the REST architecture style. It uses HTTP request methods to access resources and uses URIs (Uniform Resource Identifiers) to identify resources.

The Hypertext Transfer Protocol (HTTP) is an application-layer protocol mostly used in RESTful APIs for communication between web browsers and servers. Each HTTP request message contains the host information, the HTTP protocol version (HTTP/1.1, HTTP/2), the HTTP methods (GET/POST), the HTTP headers (content type, content length), the actual message that is being transferred to the server, and the body, which contains that message.

HTTP Requests

A correctly composed HTTP request contains three elements: the request line, header and message body.

Request line: It consists of the request method, the request URI (which specifies the URI of the resource that the client is requesting, including the path, query parameters, etc.), and the HTTP protocol version number.
Header: It contains information that a server can use to decide how to respond to the request.
Message body: It contains the specific content of the request, such as form data, JSON data, etc.

For example, the following is an example of an HTTP request message:

POST /api/v1/login HTTP/1.1
Host: example.com
Content-Type: application/json
Content-Length: 44

{"username": "user123", "password": "password123"}

URL

A URL(Uniform Resource Locator) is an address used to locate resources on the Internet. It usually consists of a protocol type, hostname, port number, path, and query string. For example:

https://www.example.com:8080/api/login?param1=value1&param2=value2

In which：

https is the protocol type
www.example.com is the server address
8080 is the port number that the server is listening on
/api/login is the path to the resource being accessed
param1=value1&param2=value2 is the query string

By requesting a URL, the client can send a request to the server and obtain the corresponding messages.

Request Method

The standard RESTful API only has four methods: GET, POST, PUT, and DELETE.

Request Headers

An HTTP request header is a component of a network packet sent by client to the server to request for a specific page or data on the Web server.

In the example above, the Request Header includes:

Host: The domain name of the server (for virtual hosting), and the TCP port number on which the server is listening.
User-Agent: The user agent string of the user agent, i.e. Firefox 58.0 browser on Windows 10 operating system.
Accept: Media type(s) that is/are acceptable for the response.
Content-Type: Type of data the browser will accept in return
Content-Length: The length of the request body in octets

Message body

A message body is data sent by the client to your API, mostly used in POST, PUT and PATCH methods.

The format and content of the message body are usually specified by the Content-Type field in the request header

These are three common types:

Raw. This can technically hold any kind of string, but generally includes XML or JSON, e.g. {"firstName":"Pied","lastName":"Piper"}
x-www-form-urlencoded. This is used for sending simple text values in a query string, e.g. key1=value1&key2=value2. All characters are url encoded, e.g. spaces are replaced by %20.
Binary. This is used for attaching images, videos, audio, and other non-text files.

HTTP Responses

HTTP responses, the answer from the server, consist of three parts: status line, header, and response body.

Status line

The start line of an HTTP response, called the status line, contains protocol version, status code and a status text for one to better understand the responses.

eg: HTTP/1.1 404 Not Found.

Headers

The response header allow the server to pass additional information about the response.

Message body

The message body of a response contains the requested information requested by the client in the format specified by the Accept field in the request header.

Types of API Testing

The aim of API testing is to test every interface in the system to verify that the interface meets the requirements and specifications in terms of functionality, performance, security, etc.

1. Functional Testing

Checks if the API works as inteded.

API functional testing mainly focuses on the functionality of an API. It primarily evaluates specific functions inside the codebase. API testing guarantees that the API returns the intended result for a given input and handles issues when the output is outside the acceptable parameters. Negative or positive tests are the sub-types of functional tests, where negative tests examine how an API responds to each sort of incorrect input.

2. Performance Testing

Verify the performance of the interface under high concurrency, large data volume, etc., including response time, throughput, number of concurrency, etc.

Stress testing: Simulate multi-user concurrent access to the interface and observe the performance of the interface, such as response time, throughput, error rate, etc. Some tools can be used for stress testing, such as JMeter, LoadRunner, etc.
Load testing: Test the performance of the interface under different loads. Load testing can be divided into two ways: static load and dynamic load. Static load refers to testing the performance of the interface under a predetermined number of concurrency, while dynamic load is to dynamically adjust the number of concurrency according to the actual load to test the performance of the interface under different loads.
Spike testing: Determine how an API responds to sudden, unexpected spikes in traffic.
Soak testing: Soak testing can be achieved by running for a long period of time to observe the stability and reliability of the interface over an extended period of time

3. Security Testing

Testing the security of the API, including its protection capability, authentication and authorization, data encryption, etc.

For security testing, different testing methods and techniques need to be used for different security risks and threats. It is also necessary to consider the security of the testing environment, such as the protection of test data and security management during the testing process. In order to improve testing efficiency and coverage, security testing tools can be used to assist testing, such as vulnerability scanning tools, code static analysis tools, etc.

How to Do API Testing

Reviewing the specifications of the API: Reviewing the specifications of the API implies studying its functioning, exploring its objectives, and what one can expect from it.
Determining the requirements of API testing: you must identify the API’s testing needs as this will necessitate an understanding of the API’s target consumer, its features and functionalities, the application’s workflow, as well as the aspects, priorities, and problems you’re testing for.
Defining the input parameters: You must define input parameters before executing an API. These parameters provide important information to the API for it to fulfill its job and are thus required for establishing if the API functions as planned.
Writing test cases for API testing: Positive tests are intended to verify the API’s fundamental functionality using mandatory parameters as well as additional capabilities utilizing optional options. Negative tests are used to see how the API reacts to disallowed actions with valid and invalid user input.
Selecting an API testing tool: Postman can be a great tool for monitoring API, creating automatic tests, identifying and removing bugs and running specific requests.

Send request to verify response

AREX is an open-source testing tool(https://github.com/arextest), a great Postman alternative for API testing. Here is a quick demonstration of testing an API with AREX.

First create a new request:

Add request URL

Enter the URL of the API you want to send the request to in the address field.

Select request method

Once the request is created, the request method is selected as GET by default, which means retrieve data from an API.

Or you can select other method from the dropdown.

Query parameters

A query string is a string of characters added to the end of a URL after ? and separated by & to pass additional information to the API.

A typical example like https://www.example.com/search?q=apple&category=fruits, with two pairs of key and value.

Set the query string parameter name in the KEY column.
Set the query string parameter value in the VALUE column.

Request headers

If you need to send specific request header information with your request, you can add request header by key-value pair.

Request Body

If you want to send data from the client to an API, you need to send the request body data with the request, commonly used in PUT, POST, and PATCH requests.

Writing Scripts

There are two types of scripts, Pre-request Script and Tests, which correspond to the two phases of API before request and after return response respectively.

Pre-request Script are JavaScript codes executed before the API request, which can be used to add authentication information, set request timeout, check the format of request parameters, etc. AREX provides commonly used predecessor scripts, which can be used directly by clicking on them.

Tests are JavaScript codes that are executed after the API request returns data. It is mainly used to test (assert) the correctness of the results returned by the request.

Once you have configured the request parameters, click Send to get the response.

Return Response

The response code, response time(the time in milliseconds it took for the response to arrive from the server) and response size can be seen at the top of the response.

Response Body is the body of the response, i.e. the content of the response returned from the server, and the data format of the content is JSON by default.

The Raw view allows you to see the raw response body content.

Headers shows the response header information.

If a Test script is set, you can view the results of executing the script in Result.

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

Test your application by using real traffic from production

Jing — Thu, 14 Sep 2023 02:42:44 +0000

Regression testing is a type of software testing that investigates the validity of previously-tested software. It ensures the software still functions as expected after it has been modified and integrated with other software, tools, and interfaces.

AREX is an open-source automated regression testing platform, which allows one to record production traffic and replay it to testing environment. By comparing the recorded responses against replay, it can clarify the impact of code changes to realize efficient automated regression testing:

Save 99% man-cost on regression: Our innovative record and replay approach eliminates the effort to write or maintain an automation suite. Testing without writing tests.
0 database contamination: Mock external sub-calls in the requests to isolate the third-database called during replay, without causing any dirty data to the database or cache of the system.
High-speed regression testing across environments: Automatically reproduces the production environment by mocking all the external dependencies. With just a single click to start, and the use of multi-threading and random execution, it achieves high-speed replay.

For QA engineers, they don't need to write or maintain regression suite by using AREX to record online traffic as test cases. For developers, it enables them to get quick feedback to debug failures.

What is Developer Testing (DevTest)

Developer testing is the act of regression testing source code by developers after the development of new features - instead of relying entirely on a team of QA experts to carry out testing.
Here are some significant benefits with regard to developer testing:

Higher code quality because new code is tested early and often.
Shortened time to market for new features.
The cost of change rises exponentially the longer it takes to find and then remove a defect.

How to do DevTest with AREX

Developers can use the standalone mode of AREX for quick self-testing of code changes before the test is submitted.

AREX Standalone mode refers to the independent operation of AREX in a local environment without the need to build storage services or other service components.

Install AREX on a Mac

Install AREX Standalone:

AREX_AGENT_MAJOR_VERSION=1.0.1 bash -c "$(curl https://raw.githubusercontent.com/arextest/deployments/dev/shell/arex.standlone.install.sh)"

As follows, the latest version of the executable file has been successfully downloaded and installed.

Install on Windows

Download the arex-standalone-all.zip from "Assets" on the release page and unzip it.
Run the arex-cli.bat script under arex-standalone-all folder.

Record

Before developing new features, start the stable version of the application, and run ./arex-cli.sh to attach to the application which you want to test.
For the first time to use, you need manually perform some actions to the application, so that AREX can record these requests as test cases.

Run ls -o 3 to view the recorded requests of /biz/getOrderInfo interface:

All requests are listed in the browser.

Click Detail to view the details:

Replay

When a new feature or change is expected to be tested, start the latest version of the application with new feature or changes, and run ./arex-cli.sh to attach Agent to the application.

Replay the recorded cases in the testing environment. After replay, AREX will compare the recorded responses against replay.

Run watch to check the differences between the two requests.

This help devs to identifies issues early, which allows them quickly test changes or new feature.

DEV Community: AREX Test

AREX or GoReplay: Choosing the Right Tool for Effective Traffic Replay

Traffic replay or traffic mirroring?

Ease of getting started

Reliability

Data sensitivity

API mocking

Identify regressions and changes

AREX or Goreplay?

How to mock a method call with Java Agent?

Code example

The implementation in AREX

Record and replay in-memory data

How to mock Apollot Configration with AREX Java Agent?

What is Apollo

Principle of Apollo client implementation

Development Process

Recording Implementation

Replay Implementation

Version Differences

Integration Testing

Reproduce production bugs locally with AREX

How to reproduce issues with AREX

Deploy and start AREX

Step 1: Deploy AREX with Docker

Step 2: Start the application with AREX Java Agent

Reproduce online issues locally

Step 3: Record requests for bugs in the production environment

Step 4: Debugging with local replay

A Real Automated Regression Testing Platform with Recording and Replay Testing (Open-Source)

Replay Traffic Testing

What is AREX?

The process of replay traffic testing

Record

Replay

Comparative Analysis and Reporting

Technical challenges

Technology Stack of AREX Java Agent

Tracing

Record and Replay Solution

Compatible Component Versions

Code Isolation

Mock Time

Mock Caches

What can you do with AREX?

Diversity validation

Reproduce the Production Issues

Mock Testing: Write a Java Agent Plugin to Mock Your Custom Components

Mock Testing with Java agent

AREX Agent

Setting up the development environment

Step 1: Create DalClientModuleInstrumentation Class

Step 2: Bytecode Modification

Step 3: Implementation of recording and replay

Deployments

Debug

AREX: How to Debug API in the browser

What is AREX?

Challenge 1: Cross-Origin Restrictions

Solutions

Challenge 2: API Debugging

Solutions

Challenge 3: Binary serialization

Solution

Code Analysis

File to Base64

Base64 to File

Production Traffic Replication with Java Agent

Record and playback testing tools

Web server-based recording

Internet protocol stack-based recording

Application layer-based recording

How dose AREX Java Agent work?

Technical challenges

ClassLoader Isolation

Tracing

Component Version Compatibility

Mock Local Cache

Time Mock

The Coolest Record and Playback Testing tool to start automating tests

Step 1: Create `DalClientModuleInstrumentation` Class