AREX: How to Debug API in the browser

• What is AREX?
• Challenge 1: Cross-Origin Restrictions
  • Solutions
• Challenge 2: API Debugging
  • Solutions
• Challenge 3: Binary serialization
  • Solution
  • Code Analysis
    • File to Base64
    • Base64 to File

Debugging APIs is vital in software development. It is a critical step to validate and test the effectiveness and correctness of application interfaces. However, traditional API debugging methods typically rely on separate tools or desktop applications, limiting the debugging process's flexibility and efficiency.

This article will use the open-source project AREX as an example to introduce how to implement API debugging functionality in the browser.

What is AREX?

AREX is an open-source automated regression testing platform based on real requests and data. It utilizes Java Agent technology and comparison techniques to achieve fast and effective regression testing through traffic recording and playback capabilities.

Challenge 1: Cross-Origin Restrictions

To debug API in the browser, the first challenge to overcome is handling the browser's cross-origin restrictions.

The browser cross-origin issue refers to the restrictions encountered when using JavaScript code to access resources from one domain's web pages to another domain's resources in web development. Browsers implement a security policy called the Same-Origin Policy to protect user information. This policy is a browser security feature that restricts how documents and scripts on one origin can interact with resources on another origin.

Due to the browser's security policy, HTTP requests cannot be sent from within the browser due to cross-origin restrictions.

Solutions

There are two common solutions to overcome this limitation: Chrome extension proxy and server-side proxy. Now we compare the two methods:

	Chrome extension	Server-side
local access	√	❌
speed	No request time loss	The overall process speed is influenced by the proxy interface.
Actual request	The request Origin will be modified to match the source of the Chrome extension	the same as the actual request

After weighing the options, AREX chose Chrome extension proxy.

The background of the Chrome extension has the ability to send cross-domain requests. Then we can intercept the requests received on the browser side and communicate with the background of the Chrome extension using window.postMessage (where communication also requires the content-script of a Chrome extension to serve as a data bridge).

The implementation is as follows:

• In the client-side script:

1. Generate a random string and store it as a string in the variable tid.
2. Use the window.postMessage() method to send a message to other extensions. The message includes an identifier of type AREX_EXTENSION_REQUEST, tid, and the params parameter.
3. Add a message event listener receiveMessage to receive messages sent by other extensions.
4. In the receiveMessage function, check if the received message is of type AREX_EXTENSION_RES and if the tid matches the one sent in the previous message. If there is a match, remove the event listener.

• In the content script:

1. Add a message event listener to receive messages sent from the page script or other extension programs.
2. Within the event listener, verify if the received message is of type AREX_EXTENSION_REQUEST. If so, utilize the chrome.runtime.sendMessage() method to dispatch the message to the background script.
3. After receiving a response from the background script, use the window.postMessage() method to send the response message back to the page script or other extension programs.

• In the background script:

1. Use the chrome.runtime.onMessage.addListener() method to add a listener for receiving messages from content scripts or other extension programs.
2. Within the listener, you can handle the received messages and respond accordingly based on your requirements.

// arex
const tid = String(Math.random());
window.postMessage(
  {
    type: '__AREX_EXTENSION_REQUEST__',
    tid: tid,
    payload: params,
  },
  '*',
);
window.addEventListener('message', receiveMessage);
function receiveMessage(ev: any) {
  if (ev.data.type === '__AREX_EXTENSION_RES__' && ev.data.tid == tid) {
    window.removeEventListener('message', receiveMessage, false);
  }
}
// content-script.js
window.addEventListener("message", (ev) => {
  if (ev.data.type === "__AREX_EXTENSION_REQUEST__"){
    chrome.runtime.sendMessage(ev.data, res => {
      //  communicate with background
      window.postMessage(
        {
          type: "__AREX_EXTENSION_RES__",
          res,
          tid:ev.data.tid
        },
        "*"
      )
    })
  }
})
// background.js
chrome.runtime.onMessage.addListener((req, sender, sendResponse) => {

})

Challenge 2: API Debugging

The above has resolved the cross-origin restrictions. The next challenge is how to implement the API debugging feature in the browser.

Solutions

Postman is a mature API debugging tool. By standing on the shoulders of giant, we introduced the JavaScript sandbox of Postman in AREX. We utilize the sandbox to run pre-request scripts, post-request scripts, and assertions for API debugging.

Here is the flowchart of request in AREX :

When sending the request, the data in the form will be gathered together into a data structure as below:

export interface Request {
  id: string;
  name: string;
  method: string;
  endpoint: string;
  params: {key:string,value:string}[];
  headers: {key:string,value:string}[];
  preRequestScript: string;
  testScript: string;
  body: {contentType:string,body:string};
}

This is the data structure of AREX, which will be converted into Postman's. Then, invoke the PostmanRuntime.Runner() method, and pass the converted Postman data structure and the currently selected environment variables. The Runner will execute the preRequestScript and testScript . The preRequestScript executes *before* a *request* runs and can include other requests as well as operations on request parameters and environment variables. The testScript is after the request and can perform assertions on the response. Additionally, the scripts can use console.log to output data for debugging in the console.

var runner = new runtime.Runner(); // runtime = require('postman-runtime');

// A standard Postman collection object
var collection = new sdk.Collection();

runner.run(collection, {}, function (err, run) {
    run.start({
      assertion:function (){},
      prerequest:function (){}, 
      test:function (){}, 
      response:function (){} 
    });
});

The cross-origin issues also exist in the Postman sandbox. Due to the high integration of the Postman sandbox, we have adopted Ajax (Asynchronous JavaScript and XML) interception technique to ensure convenience and synchronization with PostmanRuntime. By intercepting Ajax requests on the browser side, we can modify requests, add custom logic, or perform other processing operations.

When the Postman sandbox sends a request, it carries a request header "postman-token". After intercepting Ajax's request, we assemble the request parameters and send them to the browser extension via window.postMessage. The browser extension then constructs a fetch request and returns the data to the Postman sandbox. Then Postman sandbox outputs the final results, including the response, test results, and console.log. The responseType should be specified as "arraybuffer".

The specific process:

1. Register a request handler using the xspy.onRequest() method. This handler accepts two parameters: request and sendResponse. The request parameter contains relevant information about the request, such as the method, URL, headers, request body, etc. sendResponse is a callback function used to send a response back to the requester.
2. In the handler, check if there is postman-token in the request headers to determine if the request is from Postman.

• If the request is from Postman, use AgentAxios to send a new request with the same details. The response is stored in agentData. Create a dummyResponse object with relevant information from the original request. Set the status, headers, ajaxType, responseType, and response fields of dummyResponse based on agentData. Finally, return the dummyResponse using sendResponse().

• If not, then call sendResponse() directly, indicating that no response should be returned.
  

xspy.onRequest(async (request: any, sendResponse: any) => {
  // check if is sent from postman
  if (request.headers['postman-token']) {
    const agentData: any = await AgentAxios({
      method: request.method,
      url: request.url,
      headers: request.headers,
      data: request.body,
    });
    const dummyResponse = {
      status: agentData.status,
      headers: agentData.headers.reduce((p: any, c: { key: any; value: any }) => {
        return {
          ...p,
          [c.key]: c.value,
        };
      }, {}),
      ajaxType: 'xhr',
      responseType: 'arraybuffer',
      response: new Buffer(JSON.stringify(agentData.data)),
    };
    sendResponse(dummyResponse);
  } else {
    sendResponse();
  }
});

Challenge 3: Binary serialization

For requests with data formats of x-www-form-urlencoded and Raw, handling them is relatively easy since they are regular JSON objects. However, for requests of form-data and binary, they need support for sending binary payload. Unfortunately, the postMessage communication method in Chrome extensions does not support directly passing binary files, which makes it impossible to handle these two types of requests directly.

Solution

To address this issue, AREX utilizes the base64 encoding technique. When a user selects a file, AREX converts the binary file into a base64 string for transmission. On the Chrome extension side, AREX decodes the base64 data and uses it to construct the actual fetch request. This approach bypasses the limitation of directly passing binary files.

This flowchart describes the process of converting binary files in FormData to Base64 strings, and then converting them back to files and further processing them through the Chrome proxy extension.

1. form-data binary: a FormData contains a binary file.
2. FileReader: Uses a FileReader object to read the binary file.
3. readAsDataURL base64 string: The FileReader uses the readAsDataURL method to read the binary file as a Base64 string.
4. Chrome extension proxy: After the Base64 string is read, it is passed to the Chrome proxy extension.
5. base64 string: Represents the Base64 string obtained after the binary file is read by the FileReader.
6. Uint8Array: In the Chrome extension proxy, the Base64 string is converted to a Uint8Array.
7. File: A new File object is created using the data from the Uint8Array.
8. fetch: The new File object is further processed using the fetch method or other means, such as uploading to a server or performing other operations.

Code Analysis

Now let's look at the code level:

// File to Base64
const toBase64 = (file: File): Promise<string> =>
  new Promise((resolve, reject) => {
    const reader = new FileReader();
    reader.readAsDataURL(file);
    reader.onload = () => resolve(reader.result as string);
    reader.onerror = reject;
  });
// Base64 to File
function base64ToFile(dataurl: string, filename: string) {
  const arr = dataurl.split(',') || [''],
    mime = arr[0].match(/:(.*?);/)?.[1],
    bstr = atob(arr[1]);
  let n = bstr.length;
  const u8arr = new Uint8Array(n);
  while (n--) {
    u8arr[n] = bstr.charCodeAt(n);
  }
  return new File([u8arr], filename, { type: mime });
}
​
export default base64ToFile;

File to Base64

The toBase64 function takes a File object as input and returns a Promise that resolves to a Base64 string representing the file.

Inside the function, a FileReader object is used to read the contents of the File object as a Data URL by calling reader.readAsDataURL(file).

The onload event handler is set on the FileReader object, which will be triggered when the file reading operation is complete. Inside the event handler, the reader.result property, which contains the result of the file reading operation, is resolved as a string and passed to the resolve function of the Promise.

If an error occurs during the file reading operation, the onerror event handler is triggered, and the reject function of the Promise is called with the error.

Base64 to File

The function base64ToFile takes two parameters: dataurl (a Base64 string) and filename (the name of the file). The function returns a File object that represents the file with the specified name and content.

Inside the function, the Base64 string is split into an array using the comma separator. The first element of the array contains the MIME type of the file, which is extracted using a regular expression. The second element of the array contains the Base64-encoded content of the file, which is decoded using the atob function.

The decoded content is then converted to a Uint8Array object, which is used to create a new File object. The File object is returned with the specified filename and MIME type.

The above is the internal mechanism of the entire AREX debugging process.

---

Community⤵️
🐦 Follow us on Twitter
📝 Join AREX Slack
📧 Join the Mailing List