DEV Community: Jing

Automated API testing made easy with AREX

Jing — Mon, 24 Jun 2024 08:46:32 +0000

Traditional automation testing often requires significant human resources for test data preparation and script creation, and may not provide adequate coverage. To maintain the stability of online systems, both developers and testers face the following challenges:

After development, it can be challenging to quickly verify locally and identify initial issues.
Preparing test data, writing and maintaining automation scripts are time-consuming and may not provide adequate coverage.
It is difficult to verify the WRITE calls, and testing may produce dirty data, such as in our core trading system, which may write data to databases, message queues, Redis, etc. This data is often difficult to verify, and the data generated by testing is also difficult to clean up.
It's difficult to reproduce production issues locally for testing and debugging.

What is AREX?

AREX solves the challenges of automated testing by replicating real online traffic to the test environment for automated API testing.

AREX captures request parameters, return results, and some snapshot data during execution, such as database access parameters and results, as well as parameters and results for accessing remote servers, using AOP. It then sends the snapshot data to the test machine (the machine where code changes occur) to complete a replay process. By comparing the stored data, the data from calling backend requests, and the return results with the data from actual online requests, differences are identified to detect issues within the tested system.

AREX can record all operations of the application's underlying dependencies on external systems, including database operations and requests to external systems. During replay testing, when related methods are triggered, AREX extracts information from the recorded data and returns it directly to the application, avoiding interactions with the actual database or other dependencies, reducing reliance on specific environmental data, and focusing on validating the program's logic and functionality.

AREX also supports testing of write interfaces perfectly. For example, in critical scenarios such as order storage and calling third-party payment interfaces, the core mechanism of AREX traffic replay is to intercept and mock framework calls, using recorded data to replace actual data requests, ensuring that no real external interactions, such as database write operations or third-party service calls, occur during the testing process, effectively preventing the writing of dirty data during replay testing.

Workflow

1. Traffic Capture:

The AREX Java Agent, attached to Java applications in the production environment, records both inbound traffic to your API and outbound traffic to its dependencies, including the resulting responses.
Recorded data is forwarded to AREX Storage Service for storage in a MongoDB database.

2. Traffic Replay:

The Schedule Service replays the recorded application traffic back to the same application in the test environment, simulating production environment behavior.
The application in the test environment is also attached with the AREX Java Agent. When the application try to call to any dependencies like DBs, other services, AREX Agent will intercept and provide the previously recorded dependency response.

3. Result Verification and Reporting:

AREX compares API responses and outbound traffic to dependencies, like databases or external services, with previously recorded traffic, and then generates a report.

Core Advantages

✅ High Coverage Without Writing Tests

No code intrusion, minimal integration cost
No need to write test cases, a massive amount of online requests ensures high coverage

✅ Automate Testing with Mocks, No Need to Setup Test-environment

During replay, application avoids actual calls to the database and other downstream components by mocking dependencies. It utilizes previously captured requests and responses, eliminating the need to maintain active dependencies for testing.
Supports test WRITE calls, including validation of database, message queue, Redis data, and even runtime memory data without generating dirty data during testing. Replace external dependencies with mock data.
Supports automatic data collection and mocking for various mainstream technology frameworks, and supports local time, caching, and accurately reproduces the production data environment during replay.

✅ Secure and Stable

In terms of data security, it offers comprehensive permission control and traffic desensitization mechanisms.
Code isolation is implemented along with health management, and during system busy times, it intelligently reduces or stops data collection frequency, not affecting online applications.

✅ Lower test noise

Strategies to manage noise involve executing tests repeatedly to detect inconsistent fields and employing methods such as time mocking to prevent session token expiration.

Community⤵️

⭐ Star us on GitHub
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

Best Practice for Production Traffic Replication in Trip.com

Jing — Mon, 06 May 2024 03:02:40 +0000

Intro

AREX is an open-source API testing platform developed by Trip.com using the concept of production traffic replication. It focuses on the construction of core linkages for recording and playback, evolving from basic solution architecture to deep implementation verification across the corporation’s core business lines. Through continuous iterations and optimizations amidst the group’s complex business scenarios, AREX has accumulated a vast amount of experience and achieved tangible results. Since its adoption by Trip.com, over 4000 applications have been integrated with AREX, leading to improved delivery rates and a reduction in defects.

This article primarily discusses the range of challenges encountered and the solutions devised during the implementation of AREX within Trip.com, as well as how to utilize AREX to rapidly deploy a one-stop traffic recording and playback solution to reduce integration costs and accelerate implementation.

What is production traffic replication and replay?

Production traffic replication and replay is a technique used to capture network traffic in one environment — typically a production environment — and then replay that traffic in another environment.

It has a promising future for the performance, regression and automated testing. It provides a crucial solution for the technology team to cope with complex business and system architectures while ensuring system stability and improving efficiency in the R&D process.

However, deploying technological solutions is not without challenges. Teams often encounter difficulties in infrastructure development, disproportionate initial investment costs relative to early returns, and ambiguities surrounding practical application scenarios.

Sharp tools make good work

The most common open-source solution for “Production traffic replication” is based on Jvm-Sandbox-Repeater secondary development and transformation. The core principle is to record the real traffic on the line and then playback in the test environment to verify the correctness of the code logic. You may ask: Since there are mature solutions, why are we still “reinventing the wheel”?

Firstly, the components supported by JVM Sandbox are limited and cannot meet the needs of the middleware and frameworks widely used within Trip.com. Additionally, the underlying support of the JDK is not thorough enough, such as the asynchronous thread context transfer, which requires reliance on other third-party components.

Moreover, while Jvm-Sandbox-Repeater provides basic recording and playback functionality, to build a comprehensive regression testing platform, we also need a robust backend support system responsible for data collection, storage, and comparison tasks.

Lastly, the lack of official documentation and community activity put us at risk of not being able to get official support in time for subsequent secondary development.

So we decided to independently develop the traffic recording and replay tool:

Support for more middleware and components recording and replay, and the capability to simulate a variety of complex business cases, such as local caching, the time, and so on.
As a comprehensive solution, it also comes equipped with a complete set of supporting facilities, including a frontend interface, playback service, and report analysis, to achieve an all-in-one workflow from traffic collection and playback to comparison verification and report generation (as shown in the figure below).

Next, we will delve into the challenges encountered during the implementation process, targeted solutions, and internal application examples within our group, in the hope of providing you with substantial help and guidance.

Challenges Faced in Development

How to capture traffic in cross-thread and asynchronous cases

First of all, it needs to be made clear that the captured traffic we are referring to here is all the data involved in sending an API request, including not only the main endpoint but also the internal requests and responses of various frameworks such as Mybatis, Redis, Dubbo, and more.

However, many projects in our company utilize thread pools and asynchronous programming scenarios. For example, in a single request, the main process may fork multiple subtasks/threads to work in parallel. Some tasks may query Redis, some may call RPC interfaces, and others may perform database operations to fulfill different business scenarios. This also involves significant thread switching at the underlying level.

To ensure that operations executed in different threads within a single request are captured, we address this issue by using the approach of Trace propagation. This involves decorating various thread pools and asynchronous frameworks, using a recordId to pass between threads, thereby linking them together to complete a complete recording of a test case. For example, in Java, this can be achieved by decorating CompletableFuture, ThreadPoolExecutor, ForkJoinPool, as well as third-party thread pools used by Tomcat, Jetty, Netty, and asynchronous frameworks like Reactor and RXJava. This enables the propagation of the recordId across different threads.

How to avoid writing dirty data to the database during replay?

For example, in critical scenarios such as generating orders to the database and calling of third-party payment interfaces, it is necessary to use “mock” data during traffic replay to avoid actual data interactions. This approach prevents the generation of unnecessary data during testing and avoids disrupting normal business processes.

To achieve this, framework calls need to be intercepted and mocked, using recorded data instead of real data requests. This ensures that no actual external interactions, such as database writes or third-party service calls, occur during the testing process, effectively preventing the writing of dirty data during replay testing.

AREX Java Agent has supported most open-source frameworks such as Spring, Dubbo, Redis, Mybatis, and more. Please refer to the list below for the complete list:

Replay response differs from the recorded one, but not caused by a code bug

Login authentication and token expiration

During the actual process of traffic replay, we often encounter an issue: many web applications implement login authentication checks before accessing their interfaces. If the authentication fails or the login token has expired, the interface access will be denied, resulting in a large number of test cases failing during replay, which means replay response differs from the recorded one. While it is possible to address some of these issues by configuring a whitelist, we are seeking a more general solution.

The ideal solution would be to mock authentication frameworks such as Spring Security, Apache Shiro, JWT, etc., during the replay process. This would bypass the authentication and token verification steps, ensuring that the interfaces can be executed smoothly in the replay environment.

2. Time inconsistencies cause payment timeouts

If the current time during recording and replay are not consistent, it may result in unexpected differences in timeout logic. For example, in cases where we determine if an order has timed out and not been paid, we often use the condition “currentTime — orderCreateTime > 30 minutes” as the basis for judgment. If the order has not timed out during recording but during replay, half an hour later, due to the change in the system’s current time, it may mistakenly trigger the payment timeout processing logic.

To address this issue, we have proposed a solution: during the recording process, record the current time at that moment and only record it once. During the replay process, we use Mock to simulate classes related to the current time, such as Date, Calendar, LocalTime, joda.time, etc., so that the current time used during replay is actually the time recorded during the recording process. This ensures that time-related logic during replay is consistent with the recording, ensuring the accuracy and reliability of the test results.

3. Local cache

In applications, it is common practice to optimize performance by storing frequently accessed data in local caches for faster retrieval. However, in traffic recording and replay scenarios, the behavior of the cache can introduce variations in the replay results.

During recording, if the requested data is already cached, the system retrieves it directly from the cache, avoiding queries to the database or external interfaces. However, in the replay environment, without preloaded cached data, the same request may trigger database queries or external interface calls, resulting in new calls and causing replay failures.

To address this, we have supported for popular caching frameworks such as Guava Cache and Caffeine Cache, ensuring the requests to the cache during replay can return the expected results based on the recording state, avoiding unnecessary new calls.

For cases where custom caching frameworks are used, the AREX platform provides flexible configuration options that allow adaptation through dynamic classes. This means that even non-standard caching implementations can be compatible with the AREX platform and correctly support traffic replay.

Challenges Faced in Implementation

Installation and deployment should be simple, convenient, and easy to get started with

AREX is a comprehensive solution that includes not only the core recording and replay functionality but also complementary services such as frontend, scheduling, report analysis, and storage. Following the principle of out-of-the-box usability and quick integration, we offer multiple deployment options, including one-click deployment, non-container deployment, and private cloud deployment. Once installed, you only need to configure some basic parameters to automatically capture traffic and replay it to compare differences.

In addition, AREX also support a standalone mode, which allows you to quickly get started and experience the platform without the need for installation on your local machine.

2. Complying with the company’s risk control and data security requirements

When recording real production traffic, it is necessary to apply data masking rules to sensitive information to ensure reliable protection of sensitive and private data, especially in cases involving information security or sensitive commercial data. This involves transforming sensitive data into a masked or altered form to prevent unauthorized access and protect the privacy of sensitive information.

We have chosen to perform data masking during the data storage process to ensure the security of sensitive information. The specific implementation involves using the SPI (Service Provider Interface) mechanism to load external JAR packages and dynamically load encryption methods.

By utilizing the SPI mechanism, we can extend the functionality of the data storage process by loading custom encryption modules from external JAR packages. These modules can then be dynamically loaded and applied to the sensitive data before it is stored in the database.

3. Improve user experience and identify issues quickly

In actual usage, there is a huge number of test cases for recording and replaying. To reduce the workload of users when analyzing differences, we have implemented aggregation for test cases with the same differences. This speeds up the process of troubleshooting and issue identification.

With call chains, it is possible to quickly identify the scope of the problem and reduce interference by removing noise nodes such as timestamps, UUIDs, and IP addresses. This helps to minimize distractions and focus on the relevant information for issue identification.

If it is difficult to reproduce online issues locally in complex business applications, AREX also supports local debugging to quickly troubleshoot problems.

4. Is AREX mature, secure, and reliable.

AREX is based on Java Agent technology and utilizes the mature bytecode modification framework, ByteBuddy. It is secure, stable, and features code isolation and self-protection mechanisms. It intelligently reduces or disables data collection frequency during system busy periods. Moreover, it has been running steadily within Ctrip Group for over two years and has been thoroughly validated in production environments.

Best Practice

Currently, the AREX traffic recording and replay platform has been integrated as a standalone option into the company’s CI/CD system.

The initial onboarding process for AREX: When users onboard the traffic recording and replay feature for the first time, they simply need to select the Flight AREX Agent service in the CI Pipeline. This ensures that during the application packaging process into an image, the AREX startup script, arex-agent.sh, will be included in the release package.
Deployment and Agent loading: During the application deployment process, the previous script will first pull the latest arex-agent.jar and mount the AREX Agent by modifying the JVM options (-javaagent:/arex-agent.jar).
Version Control and Canary Release: After the startup script is executed, the corresponding arex-agent.jar version will be pulled based on the AppId of the application to achieve canary release and on-demand loading. For example, only certain specific applications will load the new Agent features.

Similarly, if it is the first replay, the operation is also simple:

Create a Pipeline: In GitLab or Jenkins, create a pipeline and in the ArexTest job script, call the playback URL provided by AREX and schedule the pipeline to run at regular intervals.

2. Automatically triggered traffic replay: Traffic replay is automatically triggered by developers after committing code.

3. Push the replay report and control release: After replay, AREX will push metrics such as the number of test cases, pass rate, failure rate, etc., to the relevant personnel for statistical analysis. Only when the pass rate meets the predetermined criteria, the code is allowed to be deployed to the production environment.

Landed results

Under the continuous iteration and optimization in complex business scenarios within the group, the AREX platform has accumulated a wealth of experience and achieved visible results. Since its implementation in Trip.com, it has been adopted by over 4000 applications, resulting in improved delivery rates and reduced defect counts.

Embracing Open Source

After long-term stable operation and verification of its reliability within Trip.com, we have decided to open-source the AREX platform in 2023, with the aim of helping more enterprises efficiently and cost-effectively implement traffic recording and replay technology solutions. You can find the open-source project at AREX GitHub repository.

In the past year, we have also been committed to building an open-source community. Currently, there are thousands of external users who have adopted and are using AREX, and we have received positive feedback from these users.

In the end

AREX is dedicated to ensure quality, reduce costs, and improve efficiency while meeting the demands of rapid iteration. This vision has been validated through the practices of Trip.com and numerous open-source users, bringing significant business value.

Looking ahead, we will continue to rely on the active community to respond to and address user inquiries, continuously optimizing AREX. We sincerely invite every developer to join the community, try it out, and witness the growth and progress of AREX together.

Community⤵️

⭐ Star us on GitHub
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

Try AREX for Shadow Testing with Zero Test Case Writing

Jing — Tue, 06 Feb 2024 08:05:50 +0000

As business systems evolve and become increasingly complex, quality engineers are constantly on the run. Traditional functional testing or automated testing practices no longer provide sufficient confidence in quality assurance, shadowed by the pain points of quality activities. These can be summarized as follows:

High script maintenance cost: The absence of comprehensive interface contract documentation makes the implementation of interface automation challenging, leading to high costs in writing and maintaining test cases due to iterative changes.
Tight project timelines: Insufficient testing regression.
Inadequate personnel experience & difficulty in test data creation: Script parameters fail to simulate real-world scenarios, making it harder to reproduce bugs.
Complex traffic: High cost in constructing requests, leading to incomplete coverage of test scenarios.
Business complexity: Difficulty in organizing business scenarios, with unrealistic traffic distribution.

Shadow Testing is a technique that aims to replicate the production environment (referred to as V-Current) in order to simulate real user traffic. This is achieved by creating a shadow version of the production environment. The purpose of this approach is to test new features in the V-Next environment. Once the testing phase is completed, the responses from both environments are compared to mitigate any potential risks before deploying the new feature to the production environment.
Shadow Testing has gained significant popularity within the testing community, mainly due to its effectiveness in conducting regression testing for large-scale services. It offers extensive coverage while keeping maintenance costs to a minimum.

Although similar in its objective, the AREX implementation of regression testing differs slightly from shadow testing.

Capture production traffic

A crucial part of shadow testing is testing with production data.

Java Agent is a powerful tool based on the JVM (Java Virtual Machine) that allows for the modification and enhancement of Java application bytecode at runtime, without the need to alter the application's source code. This capability makes Java Agent an ideal choice for implementing features such as monitoring, performance analysis, and traffic recording. By leveraging the Java Instrumentation API, agents can transform class files loaded by the JVM, enabling a wide range of dynamic analysis and manipulation tasks.

Understanding Traffic Capturing with Java Agent

Traffic capturing refers to the process of capturing the input and output data of an application during its execution. This includes network requests, responses, and potentially database queries, among other types of interactions. It is invaluable for debugging, performance optimization, security testing, and more, offering insights into the application's behavior under various conditions.

Bytecode Instrumentation

AREX Java Agent implements traffic recording primarily through bytecode instrumentation. Libraries such as ASM or Javassist are used to manipulate specific classes and methods at the bytecode level. This process, known as instrumentation, allows the agent to insert additional functionality for recording traffic without altering the existing business logic.

Data Capture

During instrumentation, Java Agent can inject extra code at the entry and exit points of methods to capture input parameters and return values. For network requests and responses, enhancing classes and methods related to networking libraries enables the capture of this data. This approach ensures that the traffic recording is comprehensive and includes all relevant data flows within the application.

Serialization and Storage

Once captured, the data needs to be serialized and stored for later access and analysis. Java Agent can be configured to use various serialization formats, such as JSON or XML, and storage mechanisms, including local files or remote databases. This flexibility allows developers to tailor the traffic recording to their specific needs, facilitating easier debugging and performance tuning.

Running tests and saving outputs

Once we have the input data, the next step is to build and run the test.

By leveraging the Java Agent to capture input parameters, we can accurately simulate requests to the application under test. Typically, the application being tested is also equipped with an agent. This setup allows for a nuanced approach to traffic replay where the primary interface calls are executed genuinely, while the sub-calls to third-party dependencies are not.

During replay, if a sub-call's input parameters match those recorded during the traffic recording phase, the recorded data is returned directly, effectively mocking the sub-call. This method of replaying traffic by using Java Agent to mock sub-calls based on identical input parameters ensures a high fidelity simulation of the production environment.

Advantages of This Approach

The primary benefit of this strategy is the ability to recreate the production environment's conditions in the test environment without making actual calls to third-party services. This not only helps in isolating the application under test but also significantly reduces the complexity and cost associated with setting up external dependencies for testing purposes. Moreover, it allows developers and testers to focus on the application's behavior in conditions that closely mirror those found in production, facilitating more accurate performance evaluations and debugging processes.

Comparing The Differences

Now that we have the test result and the result of the production version. We need to find the differences.

AREX not only compares the parameters and responses of the main-interface requests but also compares the request marameters of external dependencies.

A practical Example: Processing a Product Purchase Request

Consider a RESTful service with an endpoint /purchaseProduct that handles product purchase requests. When this endpoint is hit with an HTTP request, it triggers a method processPurchase in the service to handle the purchase logic, including database interactions to check and update product stock.

It involves both an entry point call and sub-calls to a database, providing a comprehensive example of traffic recording and replay using the AREX Java Agent.

/**
 * Processes a purchase request for a given product.
 * It involves an HTTP request as the entry point and a database operation as the sub-call.
 *
 * @param productId The ID of the product being purchased.
 * @param quantity The quantity of the product being bought.
 * @return A response indicating the success or failure of the purchase.
 */
@PostMapping("/purchaseProduct")
public ResponseEntity<?> processPurchase(@RequestParam Long productId, @RequestParam int quantity) {
    // Entrance Invocation: HTTP request to purchase a product
    
    // SubInvocation 1: Check product availability in the database
    int availableStock = productRepository.getStockForProduct(productId);
    
    if (availableStock >= quantity) {
        // SubInvocation 2: Update product stock in the database
        boolean isUpdated = productRepository.updateProductStock(productId, availableStock - quantity);
        if (isUpdated) {
            return ResponseEntity.ok("Purchase successful.");
        }
    }
    
    return ResponseEntity.status(HttpStatus.BAD_REQUEST).body("Insufficient stock.");
}

Step 1: Traffic Recording

Recording the Request: The Java Agent begins by recording the incoming HTTP request for a product purchase, capturing details such as the request method, URL, parameters (productId, quantity), and any headers or cookies if applicable.

Recording Database Operations: The agent then records the database queries made by the processPurchase method. This includes checking the product's available stock and attempting to update the stock if sufficient. Each database operation's request and result are captured.

Step 2: Traffic Replay

Replaying the Request: During the replay phase, the HTTP request to /purchaseProduct is made genuinely, reflecting a real-world interaction as would occur in production. This ensures that the test conditions closely mimic actual user behavior.

Mocking Database Operations: For database interactions, the agent checks if the product ID and quantity match the recorded data. If they do, it mocks the database responses based on the recorded outcomes, thereby simulating the database's behavior for both checking and updating stock without real database access.

Conclusion

To sum up, there are a lot of benefits to using AREX:

Zero impact on the production environment.
No need to generate and maintain test cases and test data.
Test real-life scenarios with real-life data.
Great test coverage from using production data.

AREX or GoReplay: Choosing the Right Tool for Effective Traffic Replay

Jing — Thu, 21 Dec 2023 08:53:41 +0000

As software applications grow more complex, encompassing elements like microservices, containers, and intricate connections, predicting and manually scripting test cases for every conceivable scenario has turned into a formidable challenge. The conventional methodology, which involves the creation and validation of test cases, demands significant manual labor and the upkeep of extensive script libraries. This traditional approach is increasingly seen as labor-intensive and less feasible in the dynamic landscape of modern software development.

Nevertheless, the advent of Production traffic replay has revolutionized the process of testing and validating applications. This innovative approach facilitates the replication of actual production traffic, enabling the realistic simulation and thorough exercise of real-world scenarios. By capturing and replaying network traffic, this method offers a streamlined path to accelerate the testing process. More importantly, it instills greater confidence in the modifications applied to the application, ensuring they perform robustly in live environments.

This approach offers several distinct advantages compared to conventional test case generation:

Produces tests that are more representative of real-world scenarios.
Significantly lessens the cognitive effort needed to devise test cases.
Uncovers potential bugs that might remain undetected in a controlled testing environment.
Utilizes a diverse range of data for more comprehensive testing.

Several tools capable of capturing production traffic, such as GoReplay and AREX, are already available in the market. Both AREX and GoReplay highlight this feature prominently as a unique selling point. Therefore, this comparison will primarily focus on how each product leverages production traffic capture.

Traffic replay or traffic mirroring?

Gor implements production traffic replay through a technique known as traffic mirroring. This method involves shadowing live user traffic and redirecting it to a test environment. This can be done either in real-time or asynchronously using systems like Apache Kafka, file systems, and other such platforms.

AREX is a testing platform for traffic replay, which involves recording the live traffic from online environment and replaying it to the test environment for API testing.

The traffic replay mechanism employed by AREX, while similar to traffic mirroring, presents some key distinctions:

Persistence of Traffic Data: Unlike traffic mirroring, which doesn’t store the captured traffic, AREX's approach allows for the retention and replay of traffic at any desired moment. This feature is particularly beneficial for testing scenarios like Black Friday traffic surges.
Application Validation: Traffic mirroring lacks built-in application validation, requiring developers to write their own, often brittle and single-use, validation checks. In contrast, AREX addresses this gap.
Data Sanitization: Raw data captured through traffic mirroring typically requires sanitization to address issues such as authentication, timestamp adjustments, and the protection of sensitive information. AREX considers this essential step, ensuring the suitability of data for use in lower-level environments, a process often overlooked in basic traffic mirroring.

Ease of getting started

GoReplay is renowned for its straightforward setup process. Implementation is as simple as downloading a binary, involving fewer initial steps than AREX. However, when delving deeper into its usage, it becomes apparent that GoReplay lacks certain functionalities found in more specialized testing tools. This is especially noticeable when compared to AREX, which is purpose-built for API testing and offers a more comprehensive set of features.

A significant distinction lies in GoReplay's lack of a WebUI, necessitating interaction with the tool predominantly via the terminal. This contrasts sharply with AREX, which boasts a WebUI, offering a more intuitive and user-friendly interface. This graphical interface simplifies the management and configuration of the tool, enhancing the overall user experience.

Reliability

In the assessment of traffic replay solutions, reliability — the consistent replay of traffic — is a critical factor. It's crucial to understand that GoReplay operates differently from a proxy. Instead, it implements a system-level hook that records all HTTP socket traffic occurring on a specified port. Once this traffic is captured, replaying it involves using a command that is almost identical to the one used for recording:

// capture traffic 
gor --input-raw :8000 --output-file ./requests.gor 

// replay the traffic 
gor --input-raw :80 --output-http "http://target_server:8080"

While GoReplay natively integrates with Kafka for data storage, this setup presents notable limitations. Specifically, when using Kafka, data must be recorded and replayed simultaneously, which can compromise reliability.

In contrast, AREX emerges as a more comprehensive tool, offering seamless integration of traffic recording and playback with automated testing. It supports the replay of recorded data at any time and multiple times, enhancing both flexibility and reliability in its use.

Data sensitivity

Capturing production traffic often involves dealing with sensitive user information, necessitating stringent measures for data anonymization or removal during the capture process. In the case of Gor, its traffic mirroring approach results in the collection of raw data, which initially lacks sanitization, potentially raising security concerns during its operation.

Conversely, for users who place a high priority on data security, AREX stands out with its strong commitment to storing only desensitized data. This approach substantially increases the safety and privacy protection of the captured traffic.

API mocking

AREX distinguishes itself from GoReplay by offering the valuable ability to modify recorded data. This feature grants users the flexibility to tailor and adjust captured traffic to suit specific testing requirements. Such a capability is crucial for simulating diverse scenarios and edge cases, thereby elevating the comprehensiveness of your testing strategy.

On the other hand, GoReplay primarily concentrates on replaying captured traffic in its original form, lacking inherent functionality to alter the data during the replay phase.

Identify regressions and changes

With Gor, there's a necessity to author middlewares, for tasks such as comparing original and replayed data, monitoring latency fluctuations, and producing analytics from this data. This can add layers of complexity to the testing process.

In contrast, AREX not only provides traffic recording and playback features but also includes built-in comparison functionalities. These integrated features in AREX simplify the process of contrasting original and replayed data, monitoring performance changes, and extracting insightful analytics, streamlining the overall testing workflow.

AREX or Goreplay?

In conclusion, when deciding between AREX and GoReplay for traffic replay, the choice hinges on specific testing requirements and preferences. GoReplay offers simplicity and a straightforward approach, making it suitable for scenarios where basic traffic replay is needed without additional complexities. AREX, on the other hand, shines with its advanced features like data sanitization, traffic modification, and built-in comparison capabilities, catering to more intricate testing scenarios. Ultimately, the decision should align with your project’s needs, balancing simplicity against a comprehensive set of features for thorough and effective testing.

Community⤵️
View on GitHub
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

Regression Testing: A Complete Guide

Jing — Thu, 07 Dec 2023 07:38:01 +0000

In the fast-paced landscape of rapid software development, where upgrades and modifications are frequent, it is crucial to ensure the stability and quality of software products. Regression testing, as a critical component of the testing process, plays a vital role in achieving this goal.

What is regression testing, and why do we need it?

Regression testing is a type of software testing conducted to confirm that a recent change or upgrade in the application has not adversely affected the existing functionalities. It is performed to ensure that the modifications or fixes made in one code module do not impact other modules due to dependencies. A tester typically initiates regression testing after a developer incorporates a new functionality into the application or completes fixing a current error.

Example of regression tests

Suppose you are working on an e-commerce website that allows users to search for products and make purchases. You have recently made some changes to the search functionality and want to perform regression tests to ensure that the changes have not introduced any new issues.

Search functionality: First test the search functionality to ensure that it still works as expected. This would include testing the search bar, filters, sorting options, and search results.
Product pages: Then test the product pages to ensure they still display the correct information and images. This would include testing the product description, price, availability, and reviews.
Shopping cart: Test the cart to ensure it works as expected. This would include testing the ability to add and remove items, update quantities, and proceed to checkout.
Payment processing: Test the functionality to ensure it still works as expected. This would include testing the ability to enter payment information, select a payment method, and complete the purchase.
User accounts: Test the user accounts functionality to ensure it works as expected. This would include testing the ability to create an account, log in, view order history, and update account information.

Record & Replay Approach in regression testing

Today's software applications are incredibly complex, with numerous microservices, containers, and connections. As a result, it's challenging to manually script test cases for every possible scenario, making it difficult to ensure that applications are bug-free.

Despite the best efforts of QA teams, maintaining application stability and quality can be challenging. Writing and maintaining tests that fail to cover every user flow in the product is a time-consuming task, and workflows outside of these tests can break, leading to the generation of new bugs regularly.

Fortunately, the record and replay approach used by AREX offers a solution to this problem. AREX monitors every user interaction, automatically building integration tests without requiring teams to write a single line of code. This comprehensive approach ensures that every possible API regression, including logical, functional, and performance failures, is caught.

Compared to manually scripting and generating traffic, traffic replay has several advantages:

Generates more realistic tests
Reduces the cognitive load required to create tests
Identifies bugs that may not be discovered in a controlled environment
Leverages varied data

1. Record Application Traffic

In general, the invocation chain of a request includes the entry point and various synchronous or asynchronous dependent calls. The recording process in AREX involves linking the entry point and the dependent calls together using a Record-Id, to save as a complete test case. The AREX-Agent achieves this by performing bytecode enhancement on the code of the entry point and dependent calls. When the code is executed, the invocation process is intercepted, and the input parameters, return values, and exceptions of the calls are recorded and sent to the storage service.

2. Replay Traffic to test a new build

During the replay mode, AREX Schedule Service retrieves the recorded data of the real traffic requests in the production environment and then sends these requests to the tested application in test environment. At the same time, the Java Agent returns the recorded responses of external dependencies to the tested application.

AREX then compares the recorded responses with the responses of the version under test to verify the correctness of the system logic. This eliminates the need for QA teams to manually think and write all assertions, streamlining the testing process.

Best practices for regression testing

For the demo, we use the community-test application as an example to demonstrate the entire process of AREX automation regression testing, from configuring the AREX Agent to recording and playback, and issue identification.

Start the AREX Docker Project

The AREX project simplifies the setup, deployment, and execution of various services, including AREX-UI, Schedule Service, API Service, Storage Service, and others, by utilizing the existing pre-defined Docker Compose configuration file. With just a single command.

First, clone the deployments repo on GitHub:

cd arex git clone https://github.com/arextest/deployments.git

Then start AREX by simply running the docker compose up command in a terminal from the project root folder.

docker-compose up -d

After installation, check the status and ports of each service by running docker-compose ps on the host running Docker.

Navigate to http://10.5.153.1:8088/ where the arex-front is running. Enter your email and you will receive a verification code to log in:

Configure AREX Java Agent

AOP (Aspect Oriented Programming) is a technique for building common, reusable routines that can be applied applicationwide. To use AREX Java Agent for recording traffic in a Java application, you essentially integrate the AREX Java Agent into your application. This agent will intercept and record the traffic, such as HTTP requests and responses. AOP is used here to weave the recording logic into your application without modifying the source code directly. It adds the recording aspect at runtime, enabling the AREX Java Agent to capture and log the traffic seamlessly as part of the application's operation.

This approach is efficient because it minimizes code intrusion while ensuring comprehensive traffic recording.

First, clone and compile the AREX Agent.

git clone https://github.com/arextest/arex-agent-java.git cd arex-agent-java mvn clean package

Compilation complete:

Step 1: Create an application in AREX

To set up the application, navigate to the AREX UI(http://10.5.153.1:8088/) and select Replay on the left-hand navigation bar. Click "+" to Create Application.

Step 2: Copy the Agent Script

After creating an application, the startup parameters for the Agent of the application will be automatically generated as shown below.

Replace the values in the pointed brackets as appropriate. You will use these parameters in the next step for the app config.

Step 3: Start Application with AREX Java Agent

Start the community-test application with the former AREX Java Agent script:

java -javaagent:/usr/local/tomcat/arex-agent-0.3.0.jar      -Darex.service.name=f27172b0e331a42a      -Darex.storage.service.host=10.5.153.1:8093      -jar community-service.jar

'-javaagent:/usr/local/tomcat/arex-agent-0.3.0.jar' is the JAR file of the AREX Agent that we have compiled. This JAR file will be loaded into the application as a Java agent.
-Darex.storage.service.host=10.5.153.1:8093: This is the address of the AREX Storage service, specified as 10.5.153.1:8093.

As you can see, the application is running successfully:

Record requests in a production environment

Now that arex-agent is successfully up and running in the application, the user does not need to perform any special operations or interventions during the recording process and can serve the application externally or access it manually in the usual way.

Code Updates

Now you've got your business requirements sorted and it's time to test a new version. Just follow these steps:

Pull the latest code.
Compile and package it up.
Heads up: Don't mess with the AREX Agent's config, especially the app name. AREX needs it to keep track of your app and handle the test cases. So, make sure the app name stays the same as before during compile and test.
Roll out the code to your test environment: Ship your compiled and packaged app to the testing grounds for some serious trial runs.

Replay Traffic to perform regression test

Now we need to perform a regression test on the new version of the application. Start replay testing and fill in the test environment (here is http://10.5.153.1:8080/):

Troubleshooting

If there are no discrepancies in the comparison between the recorded responses with the responses of the version under test, the test passes:

If the test failed, that means there is a discrepancy in the two responses. Then we need to troubleshooting.

The playback report will display all the interfaces and the test results, as shown below:

When compared, any deviations between the 2 versions is a possible breaking change in the version under test.

When you spot discrepancies, here's how to iron them out in the code:

Identify the issue and implement fixes.
Cycle back through the 'Code Update' steps: modify, test, deploy, and compare again.
If it turns out the discrepancy isn't an issue, label that section to bypass it in future playbacks.
Keep iterating this process until all discrepancies are either resolved or identified as within acceptable limits.
Once all fixes are confirmed, go ahead with releasing the updated version.

How to mock a method call with Java Agent?

Jing — Wed, 29 Nov 2023 06:00:59 +0000

Code example
The implementation in AREX
Record and replay in-memory data

AREX is an open-source automated regression testing platform based on real requests and data. It enables the accomplishment of bytecode injection with Java Agent to record real traffic in the production environment, and then replays the requests and injects mock data in the testing environment. This enables automatic recording, replaying, and comparison, providing convenience for interface regression testing.

In general, the basic principles of implementing mocking using Java Agent can be summarized as follows:

Start the application with an Java Agent loaded.
In the premain method of the Agent, use a Java bytecode manipulation library like ByteBuddy to modify the bytecode of the target class.
Locate the method or field in the target class where mocking needs to be applied within the bytecode.
Use the bytecode manipulation library to modify the implementation of the target method or field to incorporate the mocking logic. This can involve replacing the original method with a mock method or modifying the value of a field.
Redefine the modified bytecode as a new class and load it into the JVM.
During runtime, the modified class will be used, enabling the mocking functionality.

AREX Java Agent supports mocking for not only various technology frameworks, but also local time, cached data, and various memory data types. This allows for precise reproduction of the data environment during production execution during replay, without generating dirty data to your database.

This blog will delve into how AREX achieves automatic mocking of data during traffic replay from a code perspective.

Code example

Let's take a simple function as an example to understand the implementation. It's a function that converts a given IP string to an integer value. The code is as follows:

public Integer parseIp(String ip) {
    int result = 0;
    if (checkFormat(ip)) { // Check if the IP string is legal
        String[] ipArray = ip.split("\\.");
        for (int i = 0; i < ipArray.length; i++) {
            result = result << 8;
            result += Integer.parseInt(ipArray[i]);
        }
    }
    return result;
}

Now, let's explain from two perspectives, how to implement traffic playback with the function mock:

Record (collect traffic)

When this function is called, we collect the request parameters and corresponding results for later use in traffic replay. The code is as follows:

if (needRecord()) {
    // Data collection, save the parameters and results in DB
    DataService.save("parseIp", ip, result);
}

Replay

When traffic replay, the previously collected data is used to implement mocking of this function. The code is as follows:

if (needReplay()) {
    return DataService.query("parseIp", ip);
}

By examining the complete code, we can better understand its logic:

public Integer parseIp(String ip) {
    if (needReplay()) {
        // in the case of replay, the collected data is used as the return result, i.e., Mock
        return DataService.query("parseIp", ip);
    }

    int result = 0;
    if (checkFormat(ip)) {
        String[] ipArray = ip.split("\\.");
        for (int i = 0; i < ipArray.length; i++) {
            result = result << 8;
            result += Integer.parseInt(ipArray[i]);
        }
    }

    if (needRecord()) {
        // in the case of recording, save the parameters and results in DB
        DataService.save("pareseIp", ip, result);
    }
    return result;
}

The implementation in AREX

The implementation in AREX is similar but more complex. The developers don't need to add recording and replaying code in the business logic. The arex-agent automatically adds the necessary code in the required code. Let's take the Query of MyBatis3 as an example.

By reading the MyBatis source code, we should know that all Query operations are consolidated in the query method of the org.apache.ibatis.executor.BaseExecutor class (excluding Batch operations). The method signature is as follows:

public <E> List<E> query(MappedStatement ms, Object parameter, RowBounds rowBounds, ResultHandler resultHandler, CacheKey key, BoundSql boundSql) throws SQLException

This query method includes the SQL query and accompanying parameters. The function's result includes the data read from the database. It is obviously appropriate to collect data here, and during replay, the collected data can be used as the result to avoid actual operations performed on database.

Let's take a look at the code in AREX. For the sake of simplicity, some simplifications have been made, as shown below:

public class ExecutorInstrumentation extends TypeInstrumentation {
    @Override
    protected ElementMatcher<TypeDescription> typeMatcher() {
        // Full name of the class which need code injection
        return named("org.apache.ibatis.executor.BaseExecutor");
    }

    @Override
    public List<MethodInstrumentation> methodAdvices() {
        // The name of the method that requires code injection. Because the query method has multiple overloads, parameter verification is included.
        return Collections.singletonList(new MethodInstrumentation(
                        named("query").and(isPublic())
                                .and(takesArguments(6))
                                .and(takesArgument(0, named("org.apache.ibatis.mapping.MappedStatement")))
                                .and(takesArgument(1, Object.class))
                                .and(takesArgument(5, named("org.apache.ibatis.mapping.BoundSql"))),
                        QueryAdvice.class.getName())
        );
    }

    // injected code
    public static class QueryAdvice {

        @Advice.OnMethodEnter(skipOn = Advice.OnNonDefaultValue.class, suppress = Throwable.class)
        public static boolean onMethodEnter(@Advice.Argument(0) MappedStatement var1,
                                            @Advice.Argument(1) Object var2,
                                            @Advice.Argument(5) BoundSql boundSql,
                                            @Advice.Local("mockResult") MockResult mockResult) {
            RepeatedCollectManager.enter(); // prevent repeated collection of data caused by nested calls
            if (ContextManager.needReplay()) {
                mockResult = InternalExecutor.replay(var1, var2, boundSql, "query");
            }
            return mockResult != null;
        }

        @Advice.OnMethodExit(onThrowable = Throwable.class, suppress = Throwable.class)
        public static void onMethodExit(@Advice.Argument(0) MappedStatement var1,
                                  @Advice.Argument(1) Object var2,
                                  @Advice.Argument(5) BoundSql boundSql,
                                  @Advice.Thrown(readOnly = false) Throwable throwable,
                                  @Advice.Return(readOnly = false) List<?> result,
                                  @Advice.Local("mockResult") MockResult mockResult) {
            if (!RepeatedCollectManager.exitAndValidate()) {
                return;
            }

            if (mockResult != null) {
                if (mockResult.getThrowable() != null) {
                    throwable = mockResult.getThrowable();
                } else {
                    result = (List<?>) mockResult.getResult();
                }
                return;
            }            

            if (ContextManager.needRecord()) {
                InternalExecutor.record(var1, var2, boundSql, result, throwable, "query");
            }
        }
    }
}

Where QueryAdvice is the code that needs to be injected in the query method. The code injected through onMethodEnter will be executed at the beginning of the method, while the code injected by onMethodExit will be executed before the function returns the result.

It may be difficult to understand, so let's dump the code of BaseExecutor's query method after code injection and analyze it.

public <E> List<E> query(MappedStatement ms, Object parameter, RowBounds rowBounds, ResultHandler resultHandler, CacheKey key, BoundSql boundSql) throws SQLException {
        MockResult mockResult = null;
        boolean skipOk;
        try {
            RepeatedCollectManager.enter();
            if (ContextManager.needReplay()) {
                mockResult = InternalExecutor.replay(ms, parameter, boundSql, "query");
            }

            skipOk = mockResult != null;
        } catch (Throwable var28) {
            var28.printStackTrace();
            skipOk = false;
        }

        List result;
        Throwable throwable;
        if (skipOk) {
            // in the case of replay, the original query method body is no longer executed
            result = null;
        } else {
            try {
                // The source code of the query method in BaseExecutor(omitted here), and the only adjustment is to modify the code that is returned in the original method to assign the result to a variable named result
                result = list;
            } catch (Throwable var27) {
                throwable = var27;
                result = null;
            }
        }

        try {
            if (mockResult != null) {
                if (mockResult.getThrowable() != null) {
                    throwable = mockResult.getThrowable();
                } else {
                    result = (List)mockResult.getResult();
                }
            } else if (RepeatedCollectManager.exitAndValidate() && ContextManager.needRecord()) {
                InternalExecutor.record(ms, parameter, boundSql, result, throwable, "query");
            }
        } catch (Throwable var26) {
            var26.printStackTrace();
        }

        if (throwable != null) {
            throw throwable;
        } else {
            return result;
        }
    }

As you see, the code of onMethodEnter and onExit is inserted at the beginning and end respectively.

Record

When send a request to a service, AREX will decide whether to record this request based on the configured recording frequency. If recording is required, all 3rd dependencies accessed during this request will be recorded.

AREX will record this request by invoking the InternalExecutor.record(ms, parameter, boundSql, result, throwable, "query") method to store the returned result, core parameters, and other information into AREX's database.

Replay

When resend the recorded request, the function will not be executed actually. Instead, AREX will directly return the recorded result previously by invoking InternalExecutor.replay(ms, parameter, boundSql, "query").

Record and replay in-memory data

The previous example is idempotent function. For idempotent functions, since the returned result remains the same regardless of external factors, there is no need for collecting data and mocking during the recording and replay process.

For non-idempotent functions, the results returned by the request are different each time due to the influence of the external environment, such as local caches where different environment data may lead to different output results. In such cases, AREX can also record and mock the non-idempotent functions by configure it as dynamic classes.

Here, you can configure the class name, method name (optional, if not configured, it will be applied to all public methods with parameters and return values), and parameter types (optional) Once configured, the arex-agent will automatically inject similar code into the corresponding methods, enabling data collection and mocking during the replay process.

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

How to mock Apollot Configration with AREX Java Agent?

Jing — Thu, 16 Nov 2023 05:43:46 +0000

What is Apollo
- Principle of Apollo client implementation
Development Process
Recording Implementation
Replay Implementation
Version Differences
- Integration Testing

The current tools that support traffic replication for automated regression testing, such as tcpcopy and diffy, cannot be used for testing non-idempotent interfaces. This is because they record and replay traffic at the network layer outside of the application, which can only validate read-only pages. When it comes to testing interfaces that require writing to the database, it can result in dirty data and even affect the correctness of the business logic.

Unlike these tools, AREX implements traffic recording and playback internally within the application using AOP (Aspect-Oriented Programming). It utilizes Java Agent and bytecode enhancement technology to record the real traffic in the production environment. This approach avoids the impact of traffic replay on business data, making traffic replication regression testing suitable for various Java-based front-end and back-end business systems.

The basic principle of mocking in AREX is that the Java Agent intercepts the class loading process and substitutes the original class with a mock implementation.

Now most open-source components can be mocked with AREX Java Agent. This article will introduce how to mock Apollo Configuration with AREX Agent.

What is Apollo

Apollo is a reliable configuration management system. It can centrally manage the configurations of different applications and different clusters. It is suitable for microservice configuration management scenarios. And Configuration changes take effect in real-time.

Here's the official description of the Apollo base model:

Users modify and publish the configuration in the configuration center
The configuration center notifies Apollo clients of configuration updates
Apollo client pulls the latest configuration from the configuration center, updates the local configuration, and notifies the application

Principle of Apollo client implementation

The above diagram briefly describes the principle of Apollo client implementation:

The client and the server maintain a long connection to get the first push of configuration updates. (achieved through Http Long Polling)
The client also regularly pulls the latest application configuration from the Apollo Configuration Center server.
After the client gets the latest configuration of the application from the Apollo Configuration Center server, it will be saved in memory

Development Process

From the above figure, we can see that AREX only needs to support Apollo client recording and playback, i.e. Java application projects internally refer to the apollo-client component:

<dependency>
    <groupId>com.ctrip.framework.apollo</groupId>
    <artifactId>apollo-client</artifactId>
    <version>{apollo-client.version}</version>
</dependency>

Typically, there are three ways in which Apollo is commonly used in projects:

Spring Autowired annotation configBean (internally using EnableApolloConfig annotation)
Based on Apollo's built-in annotation ApolloConfig, such as the config object in the code
API Mode, such as the config1 object in the code below

@Autowired
ConfigBean configBean; // The first way, internally using EnableApolloConfig annotation

@ApolloConfig("TEST1.lucas")
private Config config; // The second way

private Config config1; // The third way, calling getAppConfig

public void test() {
    config1 = ConfigService.getAppConfig();

    System.out.println("timeout="+config.getProperty("timeout", "0"));
    System.out.println("switch="+config.getBooleanProperty("switch", false));
    System.out.println("json="+config.getProperty("json", ""));
    System.out.println("white.list="+config1.getProperty("flight.change.white.list", ""));
    System.out.println("configBean="+configBean);

    // Listening for Apollo Configuration Changes
    ConfigChangeListener changeListener = changeEvent -> {
        System.out.println("Changes for namespace:" + changeEvent.getNamespace());
    };
    config.addChangeListener(changeListener);
}

@Component
@Configuration
@EnableApolloConfig("TEST1.sofia")
public class ConfigBean {
    @Value("${age:0}")
    int age;

    @Value("${name:}")
    String name;

    @ApolloJsonValue("${resume:[]}")
    private List<JsonBean> jsonBean;

}

If AREX needs to implement the recording and playback of Apollo, it needs to be compatible with these three ways of use. By examining the Apollo source code, it is found that the first two modes, based on the annotations EnableApolloConfig and ApolloConfig, and the last mode, which calls the API, all ultimately create instances using ConfigService.getAppConfig(). This means that the underlying API is shared. Therefore, we can modify these underlying Apollo methods by inserting AREX bytecode, in order to achieve the goal of recording and playback.

Recording Implementation

All configuration items in Apollo are distinguished based on namespaces. In order to perform recording, we need to obtain all configuration instances, which means getting the config instance corresponding to each namespace. Further examination of the apollo-client source code reveals that the config instances are maintained in the Map<String, Config> m_configs of the DefaultConfigManager class.

However, several issues need to be considered:

The m_configs property is private and there is no related API to access it.
This instance is rarely called during business runtime, so it may not be possible to obtain m_configs through conventional Arex mocking methods.
After obtaining the m_configs instance, it is also necessary to obtain the m_configProperties in the Config class, as this is where the actual configuration data resides.

The dependency relationships in UML:

Therefore, it is advisable to use reflection to obtain all configurations and record them. This approach would only invoke reflection for recording during the initial startup or when configuration changes occur, which happens infrequently.

Another consideration is the timing of recording. Take the above code for example, the third way to use Apollo is to create the "config1" instance within the business interface "test()":

Config config1 = ConfigService.getAppConfig()

This can be regarded as an incremental configuration instance (compared to the first two ways where the full configuration instance is created at project startup using annotations). Therefore, we need to make sure that both can be recorded. The current approach is to perform recording at the "postHandle" point after the main entry servlet/Dubbo interface has completed the request and returned the result. This way, regardless of which way the "config" instance is created, we can obtain and record it.

If the Apollo configuration changes during recording, we can add modification code to the Apollo source code: com.ctrip.framework.apollo.internals.DefaultConfig#updateAndCalcConfigChanges method to listen for change events and reopen our recording switch. Turn on our recording switch so that we can record to the new configuration the next time.

When recording, it will generate a version number to distinguish the batch of test cases recorded in different periods, i.e., the version number acts as a batch concept, as shown in the timeline below:

Replay Implementation

Similar to the recording implementation, in the case of replay, by using reflection, the m_configProperties can be assigned values, and the mocked configuration can be used to override the real configuration.

There are several issues to consider:

How to trigger a configuration change listener method set by the application, such as the changeListener method in the Apollo usage above;
During replay, the long polling of Apollo for configuration changes may overwrite our configuration for replay, which needs to be avoided;
How to ensure the correctness of configuration data when replaying multiple versions of configurations?
How to restore the original configurations after replay?

Considering the above issues, the implementation of recording is not comprehensive for replaying and cannot meet these specific scenarios.

After a deep dive into the source code, we choose to modify the loadApolloConfig method in the com.ctrip.framework.apollo.internals.RemoteConfigRepository class. Before requesting the server configuration, we directly return our mocked configuration data. This allows us to leverage the existing mechanisms in Apollo to trigger the complete configuration update process.

Here's the solution:

Return the mocked configuration directly if the real Apollo-Server service is not called during playback.
After replay, the method is no longer mocked (the replay is considered complete if there is no replay activity for more than 1 minute), and then the normal logic is executed, i.e. the real configuration is used.

Since the long polling of Apollo is always running, if the replay activity is finished and Apollo detects that the server configuration is inconsistent with the configuration replayed by AREX, it will trigger the operation to update the local configuration, achieving the goal of restoring the original configuration.

Regarding the third point mentioned above, how to ensure the correctness of configuration data when replaying multiple versions of configurations?

The recording operation only occurs during the project startup and when there are changes in the configuration. The generated version number (UUID) is also used as the replay version number. If recorded multiple version numbers, the replay is done sequentially according to different numbers. This means that the version numbers generated by AREX are used to differentiate different versions of configurations. The implementation approach is to set the "releaseKey" attribute of the constructed Apollo configuration entity, com.ctrip.framework.apollo.core.dto.ApolloConfig , to our AREX version number every time the configuration is replayed. This ensures the correctness of playback for multiple versions of configuration data.

The "releaseKey" is a crucial field for communication between Apollo client and server. The server uses this field to determine if the configuration is consistent with the client. If not, return a new "releaseKey" value; otherwise, return a 304 status.

Version Differences

It is also necessary to be aware of the differences in source code between different versions of the decorated apollo-client.

Some methods or classes may have variations in different versions. Before deciding to modify underlying methods, it is better to examine the differences between different versions first. Otherwise, there is a possibility of failure due to inconsistencies between the Apollo client version of the user's project and the version modified by AREX Java Agent.

For example, the method we modify, com.ctrip.framework.apollo.internals.DefaultConfig#updateAndCalcConfigChanges has differences in the input parameters between versions 1.0.0 and 1.2.0.

v1.0.0

v1.2.0

When modifying this method, we need to ensure compatibility with such differences so that our injected bytecode can work properly across different versions.

It is recommended to choose a lower version when modifying with components in order to maintain as much compatibility with existing code as possible and to follow the open–closed principle.

Integration Testing

Now we need to perform integration testing between Agent and Schedule Service.

When a user starts to replay, the Schedule Service first groups all the configuration version numbers generated during recording. After grouping all the test cases within that time, the service switches the version number before each replay, informing the AREX agent that Apollo's configuration needs to be played back.

This response for version switch request is not considered as the replay result, but simply as a version pre-heating. The real replay process will be initiated once the configuration corresponding to the version number is successfully switched. The same process applies when switching to other version numbers, as shown in the following diagram.

The same batch of cases has the same version number, which only recorded or replayed once.

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

Reproduce production bugs locally with AREX

Jing — Tue, 07 Nov 2023 07:41:13 +0000

How to reproduce issues with AREX
Deploy and start AREX
- Step 1: Deploy AREX with Docker
- Step 2: Start the application with AREX Java Agent
Reproduce online issues locally
- Step 3: Record requests for bugs in the production environment
- Step 4: Debugging with local replay

In the software development cycle, software bugs are a natural occurrence. Reproducing and fixing issues in the development environment is a common activity for developers.

Reproducing production issues locally allows us to test and validate any proposed fixes before deploying them. This is essential for preventing regressions and ensuring the fix resolves the issue. It can also help identify and resolve other problems that may have been overlooked.

However, what often troubles developers is the differences in configuration and data between the production and local environments (such as data in the database, data in the cache, etc.), issues that occur in the production environment cannot be quickly reproduced in the local testing environment, making troubleshooting much more challenging.

How to reproduce issues with AREX

AREX is an automated regression testing tool, that can record traffic and data in the production environment by Java agent and replay them in the testing environment to identify the correlation between code changes and result differences.

AREX Java Agent can mock all external dependencies of the recorded real request during recording. This ensures that the data and behavior in the development environment are identical to the production environment, enabling issue reproduction and troubleshooting locally.

In this blog post, we’ll show you how to use AREX to reproduce issues. Let’s get started!

Deploy and start AREX

Step 1: Deploy AREX with Docker

git clone https://github.com/arextest/deployments.git 
cd deployments docker-compose up

Step 2: Start the application with AREX Java Agent

AREX Agent is the core component for traffic recording. Therefore, before using the recording feature, it is necessary to configure the AREX Agent for the application under test.

git clone git@github.com:arextest/arex-agent-java.git 
mvn clean install

After compilation, you will find a new folder named "arex-agent-jar" in the "arex-agent-java" directory, which contains two JAR files.

To configure the Java Agent for the application to be tested, add the Java Agent configuration to the startup of the Java application.

The complete environment variables are as follows:

JAVA_TOOL_OPTIONS='-javaagent:E:/github-arex/arex-agent-java/arex-agent-jar/arex-agent-0.1.0.jar'

Program arguments:

-Darex.service.name=arex-community-test6 -Darex.storage.service.host=10.5.153.1:8093 -Darex.enable.debug=true  -Dspring.datasource.url=jdbc:mysql://10.5.153.1:13306/community?useUnicode=true&characterEncoding=utf-8 -Dspring.datasource.username=arex_admin -Dspring.datasource.password=arex_admin_password -Dspring.redis.host=10.5.153.1 -Dspring.redis.port=16379

View the log at startup:

You will see the message "ArexJavaAgent installed" in the printout, indicating that the Agent has been successfully installed.

At the same time, you can see Java printing out command-line parameter information, such as Agent address and Storage service address, as indicated by the arrows in the above image.

Reproduce online issues locally

Step 3: Record requests for bugs in the production environment

A normal HTTP request and the response are as follows:

When a bug occurs in a production request, configure all the settings for the request in AREX, and click on "action.record" to enable recording this request, as shown in the figure below:

After clicking the button, a new key named "arex-force-record" with a value of "true" will be added to the Header List. Then save the changes.

Send the request and the response headers will generate a new key called "arex-record-id" with a value of "AREX-172-20-0-4-708656889122" (a unique value randomly generated by the AREX Agent). This indicates that the request has been successfully recorded and all the data that the request depends on in the production environment has been stored in AREX.

Step 4: Debugging with local replay

Create a new request and modify the environment of the URL to the local environment. In the AREX request headers, add a key "arex-record-id" with the value set to the previously generated "AREX-172-20-0-4-708656889122". After clicking save, send the request.

Now you can debug the issue step by step in your IDEA environment:

After debugging is completed, you can view the response.

The response message contains an "arex-replay-id" key with a value of "AREX-172-20-0-4-708658384473" (an ID randomly generated by the AREX Agent for analysis purposes). It means that the request has been successfully replayed.

The above is the complete process of using the AREX traffic recording feature to quickly reproduce and debug online issues locally.

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

A Real Automated Regression Testing Platform with Recording and Replay Testing (Open-Source)

Jing — Fri, 27 Oct 2023 09:07:45 +0000

As the company's businesses scale, the system is becoming increasingly complex. The R&D testing team is facing various challenges, including high business complexity, large data construction workload, full regression testing, high communication costs, large maintenance of test data, and automation of case management. Each of these will affect the efficiency and quality of the testing team, posing challenges to the software development process.

In summary, there are two difficulties: cost and complexity.

Cost: there will always be a trade-off between cost and quality. It's a quite challenge to ensure quality while iterating quickly and ensuring quality within limited inputs.
Complexity: when business rules accumulate over time, the complexity of business processes, rules, scenarios, and data processing increases exponentially after stacking, posing great challenges to the quality of testing work.

Despite the availability of many automation testing tools on the market today, there are still two problems:

Automated testing mainly focuses on automating execution. The maintenance work is still done manually, resulting in a less-than-optimal output ratio.
In scenarios where a large amount of test data needs to be constructed, regression testing involves a wide scope, releases are frequent, and data validation and writing are required, both manual and automated testing still face a huge workload in terms of maintaining test cases and data, and the pain points of testing have not been effectively addressed.

Replay Traffic Testing

Replay traffic refers to capturing the production traffic and replaying it in a test environment, allowing us to exercise updated systems to simulate actual production conditions.

We used this concept to build an automated testing platform AREX, which combines recording, playback, and comparison. (https://github.com/arextest).

Record: record the production requests as well as the data involved in the request processing.
Replay: replay the requests and mock the data involved in the invocation chain.
Comparative Analysis and Reporting: compare and analyze the responses from recording and replay, and get a comprehensive report for the analysis.

What is AREX?

AREX is an open-source automation test platform. It records real traffic in the production environment and mocks all invocations to third-party dependencies with Java Agent bytecode injection technology, then replays the request and mock data in the test environment. Furthermore, it compares the recorded and replayed response messages, and lists the differences, allowing developers and testers to quickly troubleshoot.

The process of replay traffic testing

Record

AOP (Aspect Oriented Programming) is a technique for building common, reusable routines that can be applied applicationwide. A Java Agent can be used to apply AOP principles by intercepting method calls and weaving in additional behavior. It provides the necessary infrastructure to implement AOP in Java applications.

In the flow of a request, the AREX Java Agent records the request information of Java applications in the production environment. It sends this information to the AREX Storage Service, which imports and stores the data in a MongoDB database.

Replay

During replay, the AREX Schedule Service retrieves the recorded data (requests) of the tested application from the database according to the user's specifications. It then sends interface requests to the target verification service. At the same time, the Java Agent returns the recorded responses of external dependencies (external requests/DB) to the tested application. After the target service completes the request logic, it returns the response message.

Comparative Analysis and Reporting

After replay testing, AREX compares the recorded response with the playback response to verify the correctness of the system logic. It then utilizes the comparison results to generate a testing report to review. Throughout this process, the AREX Cache Service (Redis) is responsible for caching mock data and comparison results during the replay process, improving the efficiency of the comparisons.

Technical challenges

Technology Stack of AREX Java Agent

Due to the good performance and code readability, we chose the Byte Buddy library to modify specific bytecode instructions from Java code.

In addition, we use SPI (Service Provider Interface), which is an API proposed to be implemented by a third-party provider. It can be used as an extension or replaceable by existing implementations. Our injection component is implemented through this plug-in model.

Tracing

While recording, AREX will capture the request and response messages of the main servlet and third-party dependencies as a complete test case. It's a challenge to trace these data.

To address this problem, the AREX Java agent generates a unique Record ID and saves it to the Thread Local variable. Injecting the agent code at the beginning and end of the application function, this code reads the value in the Thread Local variable and stores it together with the intercepted data.

Record and Replay Solution

Let's start with a simple function as an example to understand how to record and replay. Suppose we have the following function that converts a given IP string to an integer value:

public Integer parse(String ip) {
    int result = 0;
    if (checkFormat(ip)) { // Check whether IP address is legal
        String[] ipArray = ip.split("\\.");
        for (int i = 0; i < ipArray.length; i++) {
            result = result << 8;
            result += Integer.parseInt(ipArray[i]);
        }
    }
    return result;
}

Record (traffic collection)

When this function is called, we save the corresponding request parameters and the return result for later use in traffic replay. Here's the code:

if (needRecord()) {
    // Data collection, saving parameters and results in DB
    DataService.save("parseIp", ip, result);
}

Replay (traffic replay)

While replaying the traffic, we can use the previously collected data to automatically mock this function, without actually sending the request:

if (needReplay()) {
    return DataService.query("parseIp", ip);
}

By examining the complete code, we can better understand the logic :

public Integer parseIp(String ip) {
    if (needReplay()) {
        // replay scenes, use the collected data as the return result, which is essentially mock.
        return DataService.query("parseIp", ip);
    }

    int result = 0;
    if (checkFormat(ip)) {
        String[] ipArray = ip.split("\\.");
        for (int i = 0; i < ipArray.length; i++) {
            result = result << 8;
            result += Integer.parseInt(ipArray[i]);
        }
    }

    if (needRecord()) {
        // Recorded scenes, save the parameters and results to the database
        DataService.save("pareseIp", ip, result);
    }
    return result;
}

Compatible Component Versions

Popular components often have multiple versions that are used on different systems at the same time, and the implementations of the different versions may be very different or even incompatible.

To address this problem, AREX makes multiple versions compatible. In the application startup, the AREX Agent will capture all the dependent package information, such as the JAR package Manifest.MF file, from the Manifest to get the version information of the library, and then according to the version information to start the corresponding AREX injection code, thus realizing the realization of multiple versions of compatibility.

As shown in the following figure, the version range of the current injection script adaptation is set so that AREX can identify the versions of the components that the application depends on before these classes are loaded, and then later match the versions when the classes are loaded to ensure correct code injection.

Code Isolation

Since most scenarios are recorded in the production environment and replayed in the test environment, stability is of paramount importance. For the stability of the system and to prevent the Agent's code from affecting the code execution of the application under test, AREX has realized code isolation and interoperability.

The AREX core JAR is loaded in an independent ClassLoader, which is not interoperable with the user's application code. To ensure that the injected code can be accessed correctly at runtime, a simple modification to the ClassLoader is made.

Mock Time

Many business scenarios are time-sensitive. Imagine that during the replay of requests, the recorded time has expired, resulting in failed testing.

Here we implement currentTimeMillis() to proxy the original call of Java's currentTimeMillis(). While recording the traffic, capture the current time for each test case. By using a proxy for System.currentTimeMillis(), the time during replay will be replaced with simulated time that matches the time of recording, to mock time.

Mock Caches

Applications in real production environments often utilize different types of caches to enhance runtime performance. However, the variations in cached data can lead to inconsistent execution results.

These dynamic classes can be mocked in AREX, by accessing the local cache method configured as a dynamic class, equivalent to your customization of the method for mock, which will be recorded in the production environment where you configure the dynamic class method data, replay corresponding to the match out of the data returned.

What can you do with AREX?

Diversity validation

To verify the correctness of business logic after modifying the system, merely checking the return results is not enough. Usually, it is also necessary to verify the correctness of intermediate process data, such as whether the data content written to the database by the business system is correct.

In response to this, AREX also supports validating the data writing to the third-party dependencies.

During the recording and replay process, AREX will record the database requests sent to the outside by both the old and new versions of the system, and compare these two requests. If there are differences, they will be displayed in the report.

Since AREX mocks all requests to third-party dependencies, it supports the verification of data in databases, message queues, Redis, and even runtime memory data. Moreover, during the playback process, it does not generate calls to the database, so there is no dirty data.

Reproduce the Production Issues

In actual usage, AREX can also be used to quickly localize production issues.

After a production issue occurs, due to version differences, data differences, and other issues, it can be difficult for developers to reproduce on their local machines, and the cost of debugging is high and time-consuming.

By using AREX, you can force the recording of problematic cases (the response message will generate a unique Record ID) on the production environment. Then start your local development environment and add this Record ID to the request message header. This allows you to restore the recorded request and data on your local machine using the playback function, and then directly debug the production issue using local code.

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

Mock Testing: Write a Java Agent Plugin to Mock Your Custom Components

Jing — Tue, 17 Oct 2023 08:36:18 +0000

Mock Testing with Java agent
AREX Agent
Setting up the development environment
Step 1: Create DalClientModuleInstrumentation Class
Step 2: Bytecode Modification
Step 3: Implementation of recording and replay
Deployments
Debug

Mock Testing with Java agent

Mock testing is a software testing method used to simulate the behavior of certain components or dependencies in a system. The purpose of mock testing is to isolate the system under test and verify the correctness of interactions between the system and its dependencies.

Java Agent is a Java technology that allows application startup modification and enhancement of bytecode. With Java Agent, it is possible to dynamically modify the behavior of target classes at runtime, including intercepting method calls and modifying return values. This provides a possibility for implementing Mock.

By implementing a proxy class or using bytecode manipulation libraries like Byte Buddy or ASM, you can modify the behavior of the target classes. This allows you to return mock data or execute custom logic instead of the actual behavior of the dependencies.

By loading Java Agent into the target application, it can intercept and modify calls to open source components at application startup, thus enabling Mock of open source components. This can isolate the system under test during testing and verify the correctness of interactions between the system and its dependencies.

AREX Agent

AREX provides an out-of-box agent file that could be attached to any applications with Java 8+ and dynamically weaves solid bytecode into your existing code to record the real data of live traffic, and further use and replay it for mocking, testing, and debugging purposes.

Let's take a simple component DalClient as an example to illustrate how AREX Agent implements mocking.

Object result = dalClient.query(key)
if (result != null) {
    return result; // return the value when recording
} else {
    return rpc.search(); // no value to return thus call the remote interface when replaying

Setting up the development environment

Download the code and run mvn install.

git clone https://github.com/arextest/arex-agent-java.git 
mvn clean install // install arex-agent-java dependencies that may be needed into your local maven repository.

Create a new Java project from IntelliJ IDEA.

Now, add the AREX Agent dependencies to the pom file:

<dependencies>
    <!-- Some basic classes and tools used by arex-agent, such as recording and playback related components, configuration information, etc. -->         
    <dependency>
        <groupId>io.arex</groupId>
        <artifactId>arex-instrumentation-api</artifactId>
        <version>${arex.version}</version>
    </dependency>
    <!-- serialization related components -->         
    <dependency>
        <groupId>io.arex</groupId>
        <artifactId>arex-serializer</artifactId>
        <version>${arex.version}</version>
    </dependency>
    <!-- Replace it with the component you need to mock in the actual development. -->
    <dependency>
        <groupId>com.your.company.dal</groupId>
        <artifactId>dalclient</artifactId>
        <version>1.0.0</version>
        <scope>provided</scope>
    </dependency>
</dependencies>

Step 1: Create `DalClientModuleInstrumentation` Class

The arex-agent-java loads and instantiates plugins using the SPI (Service Provider Interface) mechanism. Therefore, we declare a class called DalClientModuleInstrumentation with the com.google.auto.service.AutoService annotation. This class serves as the entry point for modifying DalClient and will be recognized by the arex-agent-java (@AutoService).

@AutoService(ModuleInstrumentation.class)
public class DalClientModuleInstrumentation extends ModuleInstrumentation {
    public DalClientModuleInstrumentation() {
       // Plugin module name, you can specify a different version match if the code difference between different versions of your DalClient component is relatively large and to be supported in separate versions:
       // ModuleDescription.builder().name("dalclient").supportFrom(ComparableVersion.of("1.0")).supportTo(ComparableVersion.of("2.0")).build();
       super("plugin-dal");
    }

    @Override
    public List<TypeInstrumentation> instrumentationTypes() {
      // classes that have undergone bytecode modification to change the behavior of the DalClient component
      return singletonList(new DalClientInstrumentation()); 
    }
}

So when it's necessary to distinguish between version numbers?

If there're differences between two versions of the DalClient component, eg., the name of "invoke()" method in version 1.0.0 is changed to "invokeAll()" in version 2.0.0. In this case, the code modified by the Agent plugin cannot cover both versions of the DalClient framework simultaneously. Therefore, adapting the code specifically for different versions may be necessary.

The implementation details can refer to the adaptation logic for different Dubbo versions in the arex-instrumentation/dubbo/ module of the arex-agent-java project.

The version matching is implemented based on the contents of the META-INF/MANIFEST.MF file in the jar package of the component.

Step 2: Bytecode Modification

To modify the bytecode, create a new file named DalClientInstrumentation.java.

Modification of the DalClient source code is quite simple. Just find the underlying implementation methods, preferably using a common API, and then using bytecode manipulation tools like Byte Buddy to modify this API by adding our own code to implement the mocking functionality.

Here's the DalClient source code to be modified:

package com.your.company.dal;

public class DalClient {
    public Object query(String param) {
        return this.invoke(DalClient.Action.QUERY, param);
    }

    public Object insert(String param) {
        return this.invoke(DalClient.Action.INSERT, param);
    }

    private Object invoke(Action action, String param) {
      Object result;         
        switch (action) {
            case QUERY:
                result = "query:" + param;
            case INSERT:
                result = "insert:" + param;
            case UPDATE:
                result = "update:" + param;
            default:
                result = "unknown action:" + param;
        }     
        return result;
    }

    public static enum Action {
        QUERY,
        INSERT,
        UPDATE;
        private Action() {
        }
    }
}

In regular business projects, the DalClient is typically invoked using the dalClient.query(key) method. From the above source code, it can be observed that the underlying implementation is achieved through the invoke method.

Therefore, by modifying the invoke method and injecting recording and playback code, the functionality of the DalClientInstrumentation class can be summarized as follows:

public class DalClientInstrumentation extends TypeInstrumentation {
    @Override
    public ElementMatcher<TypeDescription> typeMatcher() {         
        // Path to the DalClient class to be modified
        return named("com.your.company.dal.DalClient"); 
    }

    @Override
    public List<MethodInstrumentation> methodAdvices() {
        ElementMatcher<MethodDescription> matcher = named("invoke") // the method tobe modified
                .and(takesArgument(0, named("com.your.company.dal.DalClient$Action"))) // The first parameter type of this method. This is to distinguish that there may be methods with the same name but different parameter types
                .and(takesArgument(1, named("java.lang.String"))); 
            // The InvokeAdvice class is the code we need to inject in the invoke method         
            return singletonList(new MethodInstrumentation(matcher, InvokeAdvice.class.getName())); 
    }
}

The MethodInstrumentation, ElementMatcher, named, takesArgument, and other methods in the above code are all the ByteBuddy API. AREX Agent edits bytecode with ByteBuddy for recording and replay. (See more details, refer to: https://bytebuddy.net/#/tutorial)

Step 3: Implementation of recording and replay

Here is the code injected into the DalClient#invoke method, implementing InvokeAdvice :

public static class InvokeAdvice {
    // OnMethodEnter is the operation that executes before the modified method (invoke) logic is called
    @Advice.OnMethodEnter(skipOn = Advice.OnNonDefaultValue.class, suppress = Throwable.class)
    public static boolean onEnter(@Advice.Argument(0) DalClient.Action action, // Obtaining a reference to the first parameter (action) of the modified method
                                  @Advice.Argument(1) String param, // Obtaining a reference to the second parameter (param) of the modified metho
                                  @Advice.Local("mockResult") MockResult mockResult) { // mockResult is the customized variables within this method 
        mockResult = DalClientAdvice.replay(action, param); // replay
        return mockResult != null && mockResult.notIgnoreMockResult();
    }

    // OnMethodExit is the operation that executes before the end of the modified method (invoke)
    @Advice.OnMethodExit(suppress = Throwable.class)   
    public static void onExit(@Advice.Argument(0) DalClient.Action action,
                              @Advice.Argument(1) String param,
                              @Advice.Local("mockResult") MockResult mockResult,
                              @Advice.Return(readOnly = false) Object result) { 
        // returned result of the method
        if (mockResult != null && mockResult.notIgnoreMockResult()) {
            result = mockResult.getResult(); // assigns the replayed result to the result variable
            return;
        }
        DalClientAdvice.record(action, param, result); // recording logic
    }
}

This class is designed to inject code before and after the modified invoke method is invoked, enabling the implementation of recording and replay functionality.

In which:

The skipOn = Advice.OnNonDefaultValue parameter indicates that if mockResult != null && mockResult.notIgnoreMockResult() is true (non-default value, where the default value for a boolean type is false), the original logic of the method will be skipped. In other words, the original method logic will not be executed, and instead, the mocked value will be returned. If it is false, the original method logic will be executed as usual.

The modified bytecode is shown below:

public class DalClientInstrumentation extends TypeInstrumentation {
    private Object invoke(DalClient.Action action, String param) {
        // replay          
        MockResult mockResult = DalClientAdvice.replay(action, param);
        if (mockResult != null && mockResult.notIgnoreMockResult()) {
            return mockResult.getResult();
        }

        // the original logic
        Object result;
        switch (action) {
            case QUERY:
                result = "query:" + param;
            case INSERT:
                result = "insert:" + param;
            case UPDATE:
                result = "update:" + param;
            default:
                result = "unknown action:" + param;
        }

        DalClientAdvice.record(action, param, result); // record

        return result;
    }
}

Similar to the Aspect-Oriented Programming (AOP), this class inserts our code before and after the method invocation. In the case of replay, it returns the mocked result without executing the original logic. For recording, it records the result before it returns.

The code of DalClientAdvice is shown below:

public class DalClientAdvice {
    // record
    public static void record(DalClient.Action action, String param, Object result) {
        if (ContextManager.needRecord()) {
            Mocker mocker = buildMocker(action, param);
            mocker.getTargetResponse().setBody(Serializer.serialize(result));
            MockUtils.recordMocker(mocker);
        }
    }

    // replay
    public static MockResult replay(DalClient.Action action, String param) {
        if (ContextManager.needReplay()) {
            Mocker mocker = buildMocker(action, param);
            Object result = MockUtils.replayBody(mocker);
            return MockResult.success(result); DalClientInstrumentation          }
        return null;
    }

    private static Mocker buildMocker(DalClient.Action action, String param) {
        Mocker mocker = MockUtils.createDatabase(action.name().toLowerCase());
        mocker.getTargetRequest().setBody(param);
        return mocker;
    }
}

The above is just a simple demo. To see more details, refer to the arex-instrumentation module of the arex-agent-java project for specific usage, which has a lot of implementations that modify various middleware.

Summarized in three steps:

Create a entry class called DalClientModuleInstrumentation, which will be loaded when arex-agent-java starts.
Create a class called DalClientInstrumentation to specify which class and method to modify, as well as when to execute the modifications, and inform the AREX Agent accordingly.
Create a class called DalClientAdvice to implement the actual recording and replay functionality, or any custom logic you want to implement.

Deployments

After completing the development, you can test whether your plugin works correctly by following these steps:

Run the command mvn clean compile package to generate the plugin JAR file (it will be located in the /target directory of your project by default).
In the arex-agent-java project, run the command mvn clean compile package to generate the arex-agent-jar directory (located in the root directory of the project).
Create a new folder named extensions in the arex-agent-jar directory to store the extension JAR files.
Place the generated plugin JAR file (e.g., plugin-dal-1.0.0.jar) into the extensions folder.

The AREX Agent loads all JAR files in the extensions folder as extension features during startup. The directory structure is as follows:

Now you can debug your plugin. Start your business application, making sure to attach the agent in the VM options:

The detailed parameters are as follows:

-javaagent:<your-project>\arex-agent-java\arex-agent-0.3.8.jar
-Darex.service.name=<your-service-name>
-Darex.storage.service.host=<arex-storage-service-ip:port>
-Darex.enable.debug=true

By starting the project in this way, the agent will be able to load the plugin package.

If the project references the internal DalClient component, you will be able to see the name of the plugin declared in DalClientModuleInstrumentation in the console after starting the project.

If the console outputs the log message [arex] installed instrumentation module: plugin-dal, it means that the plugin has been recognized and loaded successfully.

Next, test if the plugin is functioning correctly. As shown below, if an interface in the project invokes the query method of the DalClient component, you can observe whether the plugin can record or replay the invoke method of the DalClient component. If the recording is successful, the following log message will be printed:

Similarly, to test the replay functionality, include the header arex-record-id:AREX-10-32-179-120-2126724921 (the recordId generated during recording) in the request header. After the successful replay, the console will also output log messages related to [arex] replay category: Database, operation: query.

Debug

If the plugin is not working properly or not behaving as expected, you can troubleshoot from the following two points:

Check the bytecode in the arex-agent-jar/bytecode-dump folder。

The bytecode-dump folder holds the modified bytecode files where you can find the class files before and after modification here:

Drag and drop the above two files into your IDE to open them directly. Confirm whether the modifications were successful. You can compare the code logic before (left side) and after (right side) the plugin modifications, as shown in the comparison image below:

If the modifications were not successful, please refer to the source code of the project demo at GitHub.
If the code modifications were successful but the behavior is not as expected, you can debug the plugin source code in your IDE by following these steps:

Import the arex-agent-java project into the IDE of your project by following these steps: File → New → Module from Existing Sources

Select the local arex-agent-java project, and then choose Maven to import it.

Import the plugin project into the IDE of your project. Then you will have the source code for the business project, as well as the arex-agent-java and plugin-dal projects.

This way, you can set breakpoints in both of these projects to debug the code of the Agent and the plugin, as shown in the following image:

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

AREX: How to Debug API in the browser

Jing — Thu, 12 Oct 2023 07:00:59 +0000

What is AREX?
Challenge 1: Cross-Origin Restrictions
- Solutions
Challenge 2: API Debugging
- Solutions
Challenge 3: Binary serialization
- Solution
- Code Analysis
  - File to Base64
  - Base64 to File

Debugging APIs is vital in software development. It is a critical step to validate and test the effectiveness and correctness of application interfaces. However, traditional API debugging methods typically rely on separate tools or desktop applications, limiting the debugging process's flexibility and efficiency.

This article will use the open-source project AREX as an example to introduce how to implement API debugging functionality in the browser.

What is AREX?

AREX is an open-source automated regression testing platform based on real requests and data. It utilizes Java Agent technology and comparison techniques to achieve fast and effective regression testing through traffic recording and playback capabilities.

Challenge 1: Cross-Origin Restrictions

To debug API in the browser, the first challenge to overcome is handling the browser's cross-origin restrictions.

The browser cross-origin issue refers to the restrictions encountered when using JavaScript code to access resources from one domain's web pages to another domain's resources in web development. Browsers implement a security policy called the Same-Origin Policy to protect user information. This policy is a browser security feature that restricts how documents and scripts on one origin can interact with resources on another origin.

Due to the browser's security policy, HTTP requests cannot be sent from within the browser due to cross-origin restrictions.

Solutions

There are two common solutions to overcome this limitation: Chrome extension proxy and server-side proxy. Now we compare the two methods:

	Chrome extension	Server-side
local access	√	❌
speed	No request time loss	The overall process speed is influenced by the proxy interface.
Actual request	The request `Origin` will be modified to match the source of the Chrome extension	the same as the actual request

After weighing the options, AREX chose Chrome extension proxy.

The background of the Chrome extension has the ability to send cross-domain requests. Then we can intercept the requests received on the browser side and communicate with the background of the Chrome extension using window.postMessage (where communication also requires the content-script of a Chrome extension to serve as a data bridge).

The implementation is as follows:

In the client-side script:

Generate a random string and store it as a string in the variable tid.
Use the window.postMessage() method to send a message to other extensions. The message includes an identifier of type AREX_EXTENSION_REQUEST, tid, and the params parameter.
Add a message event listener receiveMessage to receive messages sent by other extensions.
In the receiveMessage function, check if the received message is of type AREX_EXTENSION_RES and if the tid matches the one sent in the previous message. If there is a match, remove the event listener.

In the content script:

Add a message event listener to receive messages sent from the page script or other extension programs.
Within the event listener, verify if the received message is of type AREX_EXTENSION_REQUEST. If so, utilize the chrome.runtime.sendMessage() method to dispatch the message to the background script.
After receiving a response from the background script, use the window.postMessage() method to send the response message back to the page script or other extension programs.

In the background script:

Use the chrome.runtime.onMessage.addListener() method to add a listener for receiving messages from content scripts or other extension programs.
Within the listener, you can handle the received messages and respond accordingly based on your requirements.

// arex
const tid = String(Math.random());
window.postMessage(
  {
    type: '__AREX_EXTENSION_REQUEST__',
    tid: tid,
    payload: params,
  },
  '*',
);
window.addEventListener('message', receiveMessage);
function receiveMessage(ev: any) {
  if (ev.data.type === '__AREX_EXTENSION_RES__' && ev.data.tid == tid) {
    window.removeEventListener('message', receiveMessage, false);
  }
}
// content-script.js
window.addEventListener("message", (ev) => {
  if (ev.data.type === "__AREX_EXTENSION_REQUEST__"){
    chrome.runtime.sendMessage(ev.data, res => {
      //  communicate with background
      window.postMessage(
        {
          type: "__AREX_EXTENSION_RES__",
          res,
          tid:ev.data.tid
        },
        "*"
      )
    })
  }
})
// background.js
chrome.runtime.onMessage.addListener((req, sender, sendResponse) => {

})

Challenge 2: API Debugging

The above has resolved the cross-origin restrictions. The next challenge is how to implement the API debugging feature in the browser.

Solutions

Postman is a mature API debugging tool. By standing on the shoulders of giant, we introduced the JavaScript sandbox of Postman in AREX. We utilize the sandbox to run pre-request scripts, post-request scripts, and assertions for API debugging.

Here is the flowchart of request in AREX :

When sending the request, the data in the form will be gathered together into a data structure as below:

export interface Request {
  id: string;
  name: string;
  method: string;
  endpoint: string;
  params: {key:string,value:string}[];
  headers: {key:string,value:string}[];
  preRequestScript: string;
  testScript: string;
  body: {contentType:string,body:string};
}

This is the data structure of AREX, which will be converted into Postman's. Then, invoke the PostmanRuntime.Runner() method, and pass the converted Postman data structure and the currently selected environment variables. The Runner will execute the preRequestScript and testScript . The preRequestScript executes *before* a *request* runs and can include other requests as well as operations on request parameters and environment variables. The testScript is after the request and can perform assertions on the response. Additionally, the scripts can use console.log to output data for debugging in the console.

var runner = new runtime.Runner(); // runtime = require('postman-runtime');

// A standard Postman collection object
var collection = new sdk.Collection();

runner.run(collection, {}, function (err, run) {
    run.start({
      assertion:function (){},
      prerequest:function (){}, 
      test:function (){}, 
      response:function (){} 
    });
});

The cross-origin issues also exist in the Postman sandbox. Due to the high integration of the Postman sandbox, we have adopted Ajax (Asynchronous JavaScript and XML) interception technique to ensure convenience and synchronization with PostmanRuntime. By intercepting Ajax requests on the browser side, we can modify requests, add custom logic, or perform other processing operations.

When the Postman sandbox sends a request, it carries a request header "postman-token". After intercepting Ajax's request, we assemble the request parameters and send them to the browser extension via window.postMessage. The browser extension then constructs a fetch request and returns the data to the Postman sandbox. Then Postman sandbox outputs the final results, including the response, test results, and console.log. The responseType should be specified as "arraybuffer".

The specific process:

Register a request handler using the xspy.onRequest() method. This handler accepts two parameters: request and sendResponse. The request parameter contains relevant information about the request, such as the method, URL, headers, request body, etc. sendResponse is a callback function used to send a response back to the requester.
In the handler, check if there is postman-token in the request headers to determine if the request is from Postman.

If the request is from Postman, use AgentAxios to send a new request with the same details. The response is stored in agentData. Create a dummyResponse object with relevant information from the original request. Set the status, headers, ajaxType, responseType, and response fields of dummyResponse based on agentData. Finally, return the dummyResponse using sendResponse().
If not, then call sendResponse() directly, indicating that no response should be returned.

xspy.onRequest(async (request: any, sendResponse: any) => {
  // check if is sent from postman
  if (request.headers['postman-token']) {
    const agentData: any = await AgentAxios({
      method: request.method,
      url: request.url,
      headers: request.headers,
      data: request.body,
    });
    const dummyResponse = {
      status: agentData.status,
      headers: agentData.headers.reduce((p: any, c: { key: any; value: any }) => {
        return {
          ...p,
          [c.key]: c.value,
        };
      }, {}),
      ajaxType: 'xhr',
      responseType: 'arraybuffer',
      response: new Buffer(JSON.stringify(agentData.data)),
    };
    sendResponse(dummyResponse);
  } else {
    sendResponse();
  }
});

Challenge 3: Binary serialization

For requests with data formats of x-www-form-urlencoded and Raw, handling them is relatively easy since they are regular JSON objects. However, for requests of form-data and binary, they need support for sending binary payload. Unfortunately, the postMessage communication method in Chrome extensions does not support directly passing binary files, which makes it impossible to handle these two types of requests directly.

Solution

To address this issue, AREX utilizes the base64 encoding technique. When a user selects a file, AREX converts the binary file into a base64 string for transmission. On the Chrome extension side, AREX decodes the base64 data and uses it to construct the actual fetch request. This approach bypasses the limitation of directly passing binary files.

This flowchart describes the process of converting binary files in FormData to Base64 strings, and then converting them back to files and further processing them through the Chrome proxy extension.

form-data binary: a FormData contains a binary file.
FileReader: Uses a FileReader object to read the binary file.
readAsDataURL base64 string: The FileReader uses the readAsDataURL method to read the binary file as a Base64 string.
Chrome extension proxy: After the Base64 string is read, it is passed to the Chrome proxy extension.
base64 string: Represents the Base64 string obtained after the binary file is read by the FileReader.
Uint8Array: In the Chrome extension proxy, the Base64 string is converted to a Uint8Array.
File: A new File object is created using the data from the Uint8Array.
fetch: The new File object is further processed using the fetch method or other means, such as uploading to a server or performing other operations.

Code Analysis

Now let's look at the code level:

// File to Base64
const toBase64 = (file: File): Promise<string> =>
  new Promise((resolve, reject) => {
    const reader = new FileReader();
    reader.readAsDataURL(file);
    reader.onload = () => resolve(reader.result as string);
    reader.onerror = reject;
  });
// Base64 to File
function base64ToFile(dataurl: string, filename: string) {
  const arr = dataurl.split(',') || [''],
    mime = arr[0].match(/:(.*?);/)?.[1],
    bstr = atob(arr[1]);
  let n = bstr.length;
  const u8arr = new Uint8Array(n);
  while (n--) {
    u8arr[n] = bstr.charCodeAt(n);
  }
  return new File([u8arr], filename, { type: mime });
}

export default base64ToFile;

File to Base64

The toBase64 function takes a File object as input and returns a Promise that resolves to a Base64 string representing the file.

Inside the function, a FileReader object is used to read the contents of the File object as a Data URL by calling reader.readAsDataURL(file).

The onload event handler is set on the FileReader object, which will be triggered when the file reading operation is complete. Inside the event handler, the reader.result property, which contains the result of the file reading operation, is resolved as a string and passed to the resolve function of the Promise.

If an error occurs during the file reading operation, the onerror event handler is triggered, and the reject function of the Promise is called with the error.

Base64 to File

The function base64ToFile takes two parameters: dataurl (a Base64 string) and filename (the name of the file). The function returns a File object that represents the file with the specified name and content.

Inside the function, the Base64 string is split into an array using the comma separator. The first element of the array contains the MIME type of the file, which is extracted using a regular expression. The second element of the array contains the Base64-encoded content of the file, which is decoded using the atob function.

The decoded content is then converted to a Uint8Array object, which is used to create a new File object. The File object is returned with the specified filename and MIME type.

The above is the internal mechanism of the entire AREX debugging process.

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List

Production Traffic Replication with Java Agent

Jing — Mon, 09 Oct 2023 09:09:05 +0000

Production traffic replication is a technique used to record traffic in production environment, and replaying it in another. It is also considered the best solution for performing regression testing.

Record and playback testing tools

Depending on the location where the recording takes place, all the recording tools can be categorized as: web server-based recording, application layer-based recording, and network protocol stack-based recording.

Web server-based recording

HTTP request recording based on web server refers to the process of recording and replicating the requests and responses between the web server and client.

Advantages: Supports a diverse range of request formats and protocols.

Disadvantages: High maintenance cost, consumes a significant amount of online resources.

Internet protocol stack-based recording

Directly monitoring to network ports and recording by duplicating data packets.

Advantages: Low impact on the application

Disadvantages: More low-level implementation, higher cost. In addition, it cannot be used to test non-idempotent interfaces, because traffic replay can result in the generation of invalid or dirty data, potentially impacting the correctness of the business operations.

Representative tools: goReplay, tcpCopy, tcpReplay

Application layer-based recording

Since Spring Boot is widely used as a Java backend framework, we can leverage AOP (Aspect-Oriented Programming) to intercept the Controller layer and record traffic.

Advantages: Non-intrusive to the code, relatively quick and simple implementation. In addition to validating the returned responses from server, it also supports validation of data written to the service, validation of database, message queue, Redis data, and even validation of runtime in-memory data.

Disadvantages: May consume some online resources, potential impact on business

Representative tools: ngx_http_mirror_module, Java sandbox, AREX

AREX is an automated regression testing platform based on traffic recording and playback. It utilizes Java Agent and bytecode enhancement technology to capture the entry points of real request chains and the request and response data of their 3rd-party dependencies in the production environment. Then, in the testing environment, it simulates to replay these requests and verifies the correctness of the entire invocation chain logic one by one.

How dose AREX Java Agent work?

AOP (Aspect-Oriented Programming) is a programming paradigm that provides a way to modularize cross-cutting concerns, such as logging, security, and transaction management. It achieves this by separating these concerns from the core business logic, allowing you to add extra functionality to existing modules without directly modifying their code.

Java Agent is a mechanism in Java that allows you to dynamically modify the bytecode of classes at runtime. It provides a way to instrument and manipulate the behavior of Java applications, including adding aspects or intercepting method calls.

In the context of AOP, a Java Agent can be used to apply AOP principles by intercepting method calls and weaving in additional behavior. It provides the necessary infrastructure to implement AOP in Java applications.

As shown in the figure below, a request typically has a chain of calls consisting of an entry point and dependencies that are either synchronous or asynchronous.

The recording process is to connect the entry and dependency calls through a RecordId to form a complete test case. AREX-Agent enhances the bytecode of the entry and dependency calls, intercepts the call process when the code is executed, and records the entry parameter, return value, and exceptions of the call, and sends them to the storage service.

During playback in the test environment, the AREX Agent simulates requests using the real data recorded in the production environment. The AREX Agent determines whether playback is required by identifying the playback flag. If playback is required, the actual method invocation is not performed. Instead, the stored response data from the storage service is retrieved and returned as the response.



public Integer parseIp(String ip) {
    if (needReplay()) {
        //  In the playback scenario, the collected data is used as the return result, which is also known as Mocking.
        return DataService.query("parseIp", ip);
    }

    int result = 0;
    if (checkFormat(ip)) {
        String[] ipArray = ip.split("\\.");
        for (int i = 0; i < ipArray.length; i++) {
            result = result << 8;
            result += Integer.parseInt(ipArray[i]);
        }
    }

    if (needRecord()) {
        // In the recording scenario, the parameters and execution results are saved into the database.
        DataService.save("pareseIp", ip, result);
    }
    return result;
}

Taking the above function as an example:

At the beginning of the function, a decision is made whether to playback or not. If playback is necessary, the collected data is utilized as the return result, which is commonly referred to as Mocking.

At the end of the function, a decision is made whether to record or not. If recording is necessary, the intermediate data that the application needs to store is saved to the AREX database.

Technical challenges

The process of recording and replay is very complex. Next, we will dive into the challenges and technical details of AREX Agent.

ClassLoader Isolation

To ensure the AREX Agent code and dependencies do not have a conflict with the application code, the AREX Agent and application code are isolated by different class loaders. As shown in the figure below, AREX Agent overrides the findClass method by customizing AgentClassLoader to ensure that the classes used by AREX Agent will only be loaded by AgentClassLoader, so as to avoid conflicts with the application ClassLoader.

Meanwhile, in order to let the application ClassLoader recognize the recording and playback code of AREX Agent, AREX Agent injects the byte code needed for recording and playback into the application ClassLoader through the ByteBuddy ClassInjector to make sure there is no ClassNotFoundException/NoClassDefFoundError during recording and replay.

Tracing

When the data is recorded and replayed, the entry point of a request and the calls of each dependency will be linked together by a RecordId. When dealing with multi-threading and various asynchronous frameworks, there are significant challenges in maintaining data continuity. AREX Agent addresses the issue of passing RecordId across threads by enhancing the thread behavior, ensuring seamless transfer of RecordId between threads. The supported threads and thread pools are as follows:

Thread
ThreadPoolExecutor
ForkJoinTask
FutureTask
FutureCallback
Reactor Framework
……

Here's a simple example for better understanding of implementation.

When invoking java.util.concurrent.ThreadPoolExecutor#execute(Runnable runnable), the parameter AgentRunnableWrapper is used to wrap the AgentRunnableWrapper runnable. When constructing AgentRunnableWrapper, the current thread context is captured. In the run method, the subthread context is replaced, and after execution, the child thread context is restored. Here is an example code snippet:



executors.execute(Runnable runnable)
executors.submit(Callable callable)

public void execute(Runnable var1) {
var1 =RunnableWrapper.wrap(var1);
}

public class RunnableWrapper implements Runnable {
  private final Runnable runnable;
  private final TraceTransmitter traceTransmitter;

  private RunnableWrapper(Runnable runnable){
    this.runnable = runnable;
    //Capture the current thread context 
    this.traceTransmitter = TraceTransmitter.create();
    }

  @Override
  public void run(){
    //Replacing the subthread context 
    try (TraceTransmitter tm = traceTransmitter.transmit()){
      (runnable.run();
    }
    //Reducing the Atomic Thread Context 
  }
}

...

Component Version Compatibility

The components introduced by an application may have multiple versions, and different versions of the same component may be incompatible, such as changes in package, addition or removal of methods, etc. In order to support multiple versions of components, AREX Agent needs to identify the correct component version for bytecode enhancement, to avoid duplicate enhancement or enhancement of the wrong version.

AREX Agent identifies the Name and Version in the META-INF/MANIFEST.MF of the component JAR file, and matches the version during class loading to ensure that the correct version is used for bytecode enhancement.

Mock Local Cache



Object value = localCache.get(key)
// Cache is available during recording, but not available during playback in which case the code needs to query the database (db.query()).
if (value != null) {
    return value;
} else {
    return db.query();
}

As shown above, during recording, the code first attempts to retrieve the value associated with the given key from the local cache (localCache.get(key)). If the value is not null, it means that the corresponding data is available in the cache during recording, and it is directly returned.

However, during playback, the cache is not available. Therefore, if the value retrieved from the cache is null, it means that the data is not present in the cache during playback. In this case, the code needs to query the database (db.query()) to retrieve the data and return it as the result.

In a word, the execution flow of the replay request is often different from the recording due to inconsistent local cache data with the recording, resulting a low pass rate of replay testing. There are a few challenges to solve this problem:

It is challenging to achieve real-time synchronization between production and test cache data due to the isolation between them.
Local memory is implemented in various ways, and it is impossible to perceive each one individually.
Local memory data is typically fundamental data and can have a large volume. Recording this data can lead to significant performance overhead.

Currently, the solution adopted by AREX Agent is to only record the cache data used in the current request chain. This is achieved by allowing the application to configure dynamic classes to identify the recording, and during playback in the test environment, the cache data is automatically replaced to ensure consistency between the recorded and playback memory data. We are still researching the solution of recording large cache data.

Time Mock

Many business systems are time-sensitive, where accessing them at different times can result in different outcomes. If the recording and playback times are inconsistent, it can lead to playback failures. Additionally, modifying the machine time on the test server is not suitable as playback requests are concurrent, and many servers do not allow modification of the current time. Therefore, we need to implement mocking of the current time at the code level to address this issue.

The currently supported time types are as follows:

java.time.Instant
java.time.LocalDate
java.time.LocalTime
java.time.LocalDateTime
java.util.Date
java.util.Calendar
org.joda.time.DateTimeUtils
java.time.ZonedDateTime

public static native long currentTimeMillis() is an intrinsic function. When the JVM performs inline optimization on intrinsic functions, it replaces the existing bytecode with internal code (JIT), which causes the enhanced code by AREX Agent to become ineffective. The JDK performs inline operations on System.currentTimeMillis() and System.nanoTime() as follows:



// // https://hg.openjdk.org/jdk8u/jdk8u/hotspot/file/dae2d83e0ec2/src/share/vm/classfile/vmSymbols.hpp#l631
//------------------------inline_native_time_funcs--------------
// inline code for System.currentTimeMillis() and System.nanoTime()
// these have the same type and signature
bool LibraryCallKit::inline_native_time_funcs(address funcAddr, const char* funcName) {
  const TypeFunc* tf = OptoRuntime::void_long_Type();
  const TypePtr* no_memory_effects = NULL;
  Node* time = make_runtime_call(RC_LEAF, tf, funcAddr, funcName, no_memory_effects);
  Node* value = _gvn.transform(new ProjNode(time, TypeFunc::Parms+0));
#ifdef ASSERT
  Node* value_top = _gvn.transform(new ProjNode(time, TypeFunc::Parms+1));
  assert(value_top == top(), "second value must be top");
#endif
  set_result(value);
  return true;
}

AREX Agent has taken special care of this issue by replacing the code that uses the method System.currentTimeMillis() with AREX Agent's method of obtaining the time directly through the application configuration, avoiding inline optimizations.

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List