Making concurrent API calls in Node

#node #javascript

The problem

When building up a backend API, it is common that we need to fetch data from a third-party API, clean, format and merge them, and then forward it to the front-end.

For instance, NASA's public could be used to fetch
APOD (Astronomy Photo of the Day) with any given date. However, it doesn't support fetching multiple photos with a range of dates. Now suppose we were asked to build a backend API that can return a list of APOD with a given number of days, what should we do?

The first thought I came up with is to generate an array that contains a range of dates. Then I can do a forEach method or a for loop to iterate through the array and make API calls one by one, get the data, push it into a result array, and finally return the result to the front-end. However, even this would work, it doesn't align with the goal, which requires to do the calls concurrently. Using forEach or a for loop still would do the job in order, not simultaneously. It's slow and not efficient.

After a little bit of research, I came across a library called async that perfectly fulfills the requirement of the task. The async library provides various types of functions for working with asynchronous JavaScript.

In this example, the method will be using is parallel, and it's mainly for flow control:

parallel(tasks, callback)

It allows us to run a number of tasks in parallel, without waiting until the previous function has completed. The results are passed to the callback as an array.

Let's get started.

The solution

First, we need to make a helper function, it takes the number of days as a parameter, and returns an array of dates. NASA's API can only take the date format as YYYY-MM-DD, so for example, if today's date is 2020-12-23, and the number of days is equal to 6, the returned array will be:

[
  '2020-12-18',
  '2020-12-19',
  '2020-12-20',
  '2020-12-21',
  '2020-12-22',
  '2020-12-23'
]

Here is what the function looks like:

function generatedates(numberOfDays) {
  const result = []
  const today = new Date()

  for (let i = 0; i < numberOfDays; i++) {
    let date = new Date(today)
    date.setDate(today.getDate() - i)
    let dd = date.getDate()
    let mm = date.getMonth() + 1
    let yyyy = date.getFullYear()

    if (dd < 10) {
      dd = "0" + dd
    }
    if (mm < 10) {
      mm = "0" + mm
    }
    date = yyyy + "-" + mm + "-" + dd
    result.unshift(date)
  }

  return result
}

Then we need to add an endpoint to the node server.

/api/photos

The parallel function takes an array of function as the first argument, so we could use the map method to iterate through the dates array and returns the function array. Each function in the array fires an Axios call to the NASA API and get the picture of that date.

The second argument of the parallel function is a callback function. In this case, since the API calls return promises, the callback function will return two items. The first one is the possible error, and the second one is the array of the result.

If we don't need to further process the data, we can simply pass them to the front-end. We can also use the forEach method to clean the data and only extract the information we need.

Here is the logic of the endpoint:

const URL = "https://api.nasa.gov/planetary/apod"

server.get("/api/photos", (req, res) => {
  const days = req.query.days
  const dates = generateDates(days)

  const functionArray = dates.map((date) => {
    return async function () {
      const data = await axios.get(`${URL}?api_key=${api_key}&date=${date}`)
      return data.data
    }
  })

  async.parallel(functionArray, (err, result) => {
    res.status(200).json({ items: result.length, photos: result })
  })
})

Now the user can make an API request to fetch any number of photos, such as:

//fetch photos of the past week
api/photos?days=7

//fetch photos of the past month
api/photos?days=30

And the result will be shown as:

{
    "items": 6,
    "photos": [...]
}

Code

Check the GitHub repo for this example
Repo

Top comments (5)

Nahuel Bustamante Murua • Dec 24 '20

Hi!!
Maybe you can use Promise.all()
Happy end year!!

ironydelerium • Dec 25 '20

This, pretty much:

server.get("/api/photos", async (req, res) => {
  const days = req.query.days
  const dates = generateDates(days)

  const requests = dates.map((date) => axios.get(`${URL}?api_key=${api_key}&date=${date}`));

  try {
    const result = await Promise.all(requests);
    res.status(200).json({ items: requests.length, photos: result.map(item => item.data) });
  } catch (err) {
    res.status(500).json({ error: String(err) });
  }
})

There's also Promise.allSettled if you want all of the results, error or not.

Gabriele Romanato • Dec 25 '20

This is a practical use case of parallelism in Node. However, we should always make sure that our routes are actually protected against any possible DOS attack by limiting the number of concurrent connections or simply by adding an authentication system on such routes with a middleware. Otherwise we'll sooner or later get into troubles.