Promise wait for multiple functions and retry some failed function

So, I found myself in a situation where I need to create an asynchronous function that fetches data from multiple third party's APIs, daily. My concern is we can't control the third party's up and downtime. And its (function) being automatically executed is making me more confused. (well, I just want a happy and quiet weekend (: )

Multiple asynchronous requests from 1 URL, could overload its server. When it happens, the server will be restarted and offline for a few seconds (maybe minutes or more), and all the requests will result failed.

In some cases, you need to crawl many websites, because not all websites offer an API. Crawling many big websites, simultaneously, at one time, could take quite a long time. The more time it needs to connect to the website, the more chance for request timeout to happen. Especially, if your application has limited bandwidth. This could lead to a request timeout.

It's more complicated if your application needs to do all the above automatically, without the dev manually restarting request or so.

Here's my solution, we only need 3 functions (4th one is optional):

1. Main function, where we call Promise All function with an array of URLs as a parameter.
2. Promise All function, where we put Promise.all() and what we want after all promise has been executed.
3. Axios function, where we put axios.get() function, to get request from third party, and return either JSON if the request is success or URL (string) if the request is failed.
4. Delay (request again) function, where we put setTimeout() to call Promise All function after n seconds delay. (this function is optional if you want to delay the next request).

Axios Function

For example, we will use this simple api. The expected result as JSON:

{
  "test": "one",
  "hello": "world"
}

Assume that this API results in an error (internal server), instead of throw, we will return the URL.

const axiosFunc: any = (async (url: string) => {
  return await axios.get(url)
    .then(response => response.data)
    .catch(e => url);
});

Promise All Function

Promise.all() accepts an Array of Promises. Instead of writing like this:

Promise.all([axiosFunc(url1),axiosFunc(url2),...]);

We could write it like this:

const urls = [url1,url2,...];
Promise.all(urls.map(url => this.axiosFunc(url));

The expected result is an Array of results from Axios Function, either json object (result from api) or string (url). With simple if..else and .some(), we will check whether the result have some string or not. If there are some string in it, we will split the result by its type with .filter().

if(data.some(val => typeof val == 'string')) {
  /** filter failed one by checking its type and execute again */
  const failedReq = data.filter(val => typeof val == 'string');
  this.requestAgain(failedReq);

  /** filter success one by checking its type and save its result to db */
  const successReq = data.filter(val => typeof val !== 'string');
  this.saveResult(successReq);
} else {
  this.saveResult(data);
}

The string one (failed one), goes to this.requestAgain(). We will get to this function later in this post. And the not string one (success one), goes to this.saveResult(). In this function, data from successful results will be saved in database or cache or other data saving methods.

Below is the full code of this function:

const promiseAllFunc: void = ((urls: string[]) => {
  return await Promise.all(urls.map(url => this.axiosFunc(url))
    .then(data => {
      if(data.some(val => typeof val == 'string')) {
        /** filter failed one by checking its type and execute again */
        const failedReq = data.filter(val => typeof val == 'string');
        this.requestAgain(failedReq);

        /** filter success one by checking its type and save its result to db */
        const successReq = data.filter(val => typeof val !== 'string');
        this.saveResult(successReq);
      } else {
        this.saveResult(data);
      }
    })
});

Main Function

Call Promise All function from above with array of urls (third party's api) as parameter.

const main: any = (() => {
  let dummyUrls = [
    'http://echo.jsontest.com/hello/world/test/one',
    'http://echo.jsontest.com/hello/world/test/two',
    'http://echo.jsontest.com/hello/world/test/three'
  ];
  return this.promiseAllFunc(dummyUrls);
});

Delay Function

This function is optional. If you want to delay the next request, you could write additional function with setTimeout(). Below is example with 10 seconds delay for the next request.

const requestAgain: any = ((urls: string[]) => {
  setTimeout(() => {
    this.promiseAllFunc(urls);
  }, 10000);
});

This method solves all of my problems. It's automatically executed (with Cronjob). Retrying failed requests (you could modify Delay Function by using Cronjob too).

Well, that's it, Thank you :)
Feel free to add some feedback in comment section. This is my first post here. Maybe some grammar mistakes here and there (or more :D ), but I think it's still understandable.