A trick to improve speed when you are interfacing with a slow API

TLDR; I created a small npm package that acts as a wrapper around node-fetch, and returns the same promise for the same request, until it resolves. You can visit the repo of this package here. Below, I explain my motivation, and how I tackled the issue.

So here's the scenario:

You have a system that interfaces with a really slow third-party API. User Bob, needs some data, so your system performs a request to the third-party API, and waits for a response. In the meantime, user Alice needs the same date and the system performs the same request to the API on behalf of her. Both users are now waiting for two requests that the only difference they have, is the execution time.

If a request to this API has an average response time of 1 second, both users will wait 1 second. Also, you would need to occupy resources in your system and the third-party API for more than 1 second, and for 2 seconds at most!

The solution

What if you could have both users, Bob and Alice, wait for the same request? Then, although Bob will still wait for the request for 1 second, Alice will use Bob's request, and wait less time for the response.

To achieve that, we'll need a promise-cache subsystem. This subsystem will consist of a data structure to store our requests' promises and of a way to retrieve them/delete them when they are not needed.

The data structure

We need a data structure to store our promises inside. This data structure needs to be able to store and retrieve a new promise in one operation (O(1)). So, the best choice would be a key/value store. Javascript, offers two such structures, the basic object and the Map() instance. The most preferrable data structure for our use-case among the two is the Map().

So, let's create it:

const promiseCache: Map<string, Promise<Response>> = new Map();

The retrieval/storage

Now, let's create a function that wraps around the request function and retrieves the same promise for the same request, if it exists. If it doesn't, it performs a new request and stores it in the cache.

function memoizedRequest(url: string) {
    const key = url;
    if (promiseCache.has(key)) {
        return promiseCache.get(key);
    }

    const promise = request(url);
    promiseCache.set(key, promise);

    return promise;
}

With this, we have achieved the basic function of our promise-cache subsystem. When our system performs a request using the memoizedRequest function, and the request has already happened, it returns the same promise.

But, we haven't yet implemented the mechanism for the deletion of the promise from the cache when the promise resolves (when the request returns results)

The deletion - cache invalidation

For this, we'll create a function that awaits for the promise to resolve and then delete the promise from the cache.

async function promiseInvalidator(key: string, promise: Promise<any>) {
    await promise;
    promiseCache.delete(key);

    return promise;
}

And then we'll modify our memoizedRequest function to include this invalidation function:

function memoizedRequest(url: string) {
    const key = url;
    if (promiseCache.has(key)) {
        return promiseCache.get(key);
    }

    const promise = promiseInvalidator(key, request(url));
    promiseCache.set(key, promise);

    return promise;
}

But what happens with more complicated requests?

Not all requests can be differentiated by just the url they are performed on. There are many other parameters that make a request different (eg: headers, body etc).

For that, we'll need to refine our promise-cache's key and add an options object on our function:

function memoizedRequest(url: string, options: RequestOptions) {
    const key = url + JSON.stringify(options);
    if (promiseCache.has(key)) {
        return promiseCache.get(key);
    }

    const promise = promiseInvalidator(key, request(url));
    promiseCache.set(key, promise);

    return promise;
}

Now, only the requests that use exactly the same options will return the same promise until they resolve.

With this, we implemented all the basic functionality of our package. But we haven't taken into account the possibility of a request failure. Let's add this on our code, by making the promiseInvalidator function to always remove the promise from the cache either when it resolves, or when it rejects.

async function promiseInvalidator(key: string, promise: Promise<any>) {
    try {
        await promise;
    } finally {
        promiseCache.delete(key);
    }

    return promise;
}

More improvements

This implementation has a small drawback, that can prove serious on a production system. All the requests' data, are stored within the key of our data store, highly increasing the memory requirements of our application, especially when our requests contain a lot of data. The solution to this is to use a hash function on our key, to assign a unique value to each different request, without needing to include all the actual of the request.

const key = hasher(url + JSON.stringify(options));

Caveats

This solution, isn't applicable to any situation. To use this solution, you need to ensure that the API you are interfacing with, is not providing different responses for two different requests in the amount of time it will take for those requests to resolve.

The package

If you don't want to code this for yourself, I created a simple npm package that does all of the above, as a wrapper to node-fetch (or any other fetch-like function you choose).

import memoizedNodeFetch from 'memoized-node-fetch';

const fetch = memoizedNodeFetch();

(async () => {
    const fetch1 = fetch('https://jsonplaceholder.typicode.com/todos/1');
    const fetch2 = fetch('https://jsonplaceholder.typicode.com/todos/1');

    // This should return true because both requests return the same promise.
    console.log(fetch1 === fetch2);

    const res1 = await fetch1;
    const res2 = await fetch2;

    console.log(await res1.json());
    console.log(await res2.json());
})();

You can see all of the above work, on its Github repository here:

https://github.com/chrispanag/memoized-node-fetch

PS. 1: Although this can be used in the front-end, I can't find a very useful use-case for it, especially when you have other packages such as react-query/swr, that although they perform a different function than the above, can sometimes remove the need for it.

PS. 2: Special thanks to the other two contributors of this repository (ferrybig and Bonjur for their invaluable input and suggestions!