Intercepting HTTP Requests with NodeJS

Intro

As part of a project at work, I had to develop a way to intercept and store HTTP traffic for any given backend application (microservice in this case). This would have been a fairly straightforward task, but our backend is composed of many services (and many repos). Therefore, the solution had to be as seamless as possible so that it could be easily integrated into any of the services.

TLDR;

Using @mswjs/interceptors makes it straightforward to intercept HTTP traffic on your backend app.

Intercepting HTTP Traffic

For my use case, there were two options I could think of for capturing the HTTP traffic:

1. Create a library that wraps an HTTP client library like Axios
2. Somehow intercept all HTTP traffic

In an ideal world, I would have chosen option 1 since it would be the simplest. Unfortunately, the project I work on consists of many microservices owned by different teams. Therefore, it would make it difficult for everyone to go back and refactor their code to use this new library.

Therefore, my only option was really option 2.

First attempt

My first attempt was ok, but far from perfect. After trying to intercept the traffic directly through the low-level http module, I opted for a more higher-level solution. My idea was to monkey patch Axios's request methods to inject my own logic before a request is sent and after the response is received.

function _instrumentAxios(axiosInstance: AxiosInstance) {
   axiosInstance.request = _instrumentHttpRequest(axiosInstance.request, axiosInstance);
   axiosInstance.get = _instrumentHttpRequest(axiosInstance.get, axiosInstance, "get");
   axiosInstance.post = _instrumentHttpRequest(axiosInstance.post, axiosInstance, "post");
   axiosInstance.put = _instrumentHttpRequest(axiosInstance.put, axiosInstance, "put");
   axiosInstance.patch = _instrumentHttpRequest(axiosInstance.patch, axiosInstance, "patch");
   axiosInstance.delete = _instrumentHttpRequest(axiosInstance.delete, axiosInstance, "delete");
   axiosInstance.options = _instrumentHttpRequest(axiosInstance.options, axiosInstance, "options");
}

function _instrumentHttpRequest(originalFunction: Function, thisArgument: any, method?: string) {
   return async function() {
      const {method: cleanedMethod, url, config: requestConfig, data} = _parseAxiosArguments(arguments, method);

      const requestEvent: HttpRequestEvent = {
         url,
         method: cleanedMethod,
         body: data,
         headers: requestConfig?.headers,
      };


      // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
      doSomethingWithRequest(requestEvent);

      const res = await originalFunction.apply(thisArgument, arguments);


      const responseEvent: HttpResponseEvent = {
         url,
         method: cleanedMethod,
         body: res.data,
         headers: res.headers,
         statusCode: res.status,
      };

      doSomethingWithResponse(responseEvent);

      return res;
   };
}

This method worked fine, but then I accidentally bumped into a cleaner approach while reading the Axios docs.

Second attempt

To my surprise, Axios actually offers an API for intercepting requests and responses!

import {createInterceptor, InterceptorApi, IsomorphicRequest, IsomorphicResponse} from "@mswjs/interceptors";
import {interceptXMLHttpRequest} from "@mswjs/interceptors/lib/interceptors/XMLHttpRequest";
import {interceptClientRequest} from "@mswjs/interceptors/lib/interceptors/ClientRequest";

function _instrumentAxios(axiosInstance: AxiosInstance) {
   axiosInstance.interceptors.request.use(_instrumentHttpRequest);
   axiosInstance.interceptors.response.use(_instrumentHttpResponse);
}

function _instrumentHttpRequest(requestConfig: AxiosRequestConfig): AxiosRequestConfig {
   const method = String(requestConfig.method);
   const headers = requestConfig.headers && {
      ...requestConfig.headers.common,
      ...requestConfig.headers[method],
   };

   const requestEvent: HttpRequestEvent = {
      headers,
      method,
      url: String(requestConfig.url),
      body: requestConfig.data,
   };


   // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
   doSomethingWithRequest(requestEvent);

   return requestConfig;
}

function _instrumentHttpResponse(response: AxiosResponse): AxiosResponse {
   const responseEvent: HttpResponseEvent = {
      url: String(response.request?.url),
      method: String(response.request?.method),
      body: response.data,
      headers: response.headers,
      statusCode: response.status,
   };


   // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
   doSomethingWithResponse(responseEvent);

   return response;
}

Ah! Much better. However, there is another complication to this approach that's also present in the first attempt: the interception has to be set for every Axios instance; this makes for a less than ideal developer experience. I initially assumed everyone used the default axios instance. However, it turns out that it's also possible to create new instances via axios.create(). So back to the drawing board 😔

Final solution

Before attempting to mess with the low-level http module, I decided to look for some existing solutions. And after digging around for some time, I stumbled into @mswjs/interceptors. This library is very well documented and is TypeScript friendly.

function _instrumentHTTPTraffic() {
    const interceptor = createInterceptor({
      resolver: () => {}, // Required even if not used
      modules: [interceptXMLHttpRequest, interceptClientRequest],
   });

   interceptor.on("request", _handleHttpRequest);

   interceptor.on("response", _handleHttpResponse);

   interceptor.apply();
}

function _handleHttpRequest(request: IsomorphicRequest): void {
   const url = request.url.toString();
   const method = String(request.method);
   const headers = request.headers.raw();

   const requestEvent: HttpRequestEvent = {
      headers,
      method,
      url: request.url.toString(),
      body: request.body,
   };


   // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
   doSomethingWithRequest(requestEvent);
}

function _handleHttpResponse(request: IsomorphicRequest, response: IsomorphicResponse): void {
   const url = request.url.toString();
   const headers = request.headers.raw();


   const responseEvent: HttpResponseEvent = {
      url: request.url.toString(),
      method: request.method,
      body: response.body,
      headers: response.headers.raw(),
      statusCode: response.status,
   };

   // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
   doSomethingWithResponse(responseEvent);
}

Caviats

Although the final solution is more general and is also agnostic to the client HTTP library used, there are some downsides:

• Since all HTTP traffic going thorugh the app is intercepted, there needs to be some logic in place to know which requests to ignore. E.g., instrumentation tools like NewRelic regularly send requests to capture metadata. This can add a lot of noise if not handled properly
• Dependence on another library. Whether this is a big deal depends on what the interception is used for. Probably not a big deal for most projects

Comments

[Artem Zakharchenko]

Thank you for writing this, Henry!

An interception predicate is, actually, an interesting idea. The interceptors library was intentionally designed to capture all outgoing traffic and delegate any kind of predicates to the higher scope (you). I'm curious if there'd be any performance boost was there some preliminary predicate applied.

[Henry Williams]

Awesome. I'll check it out!

[Voomzi Repository]

Henry how does this intercept the request? Is there some code that's missing from this example?