DEV Community

Yue Su
Yue Su

Posted on

Making concurrent API calls in Node

The problem

When building up a backend API, it is common that we need to fetch data from a third-party API, clean, format and merge them, and then forward it to the front-end.

For instance, NASA's public could be used to fetch
APOD (Astronomy Photo of the Day) with any given date. However, it doesn't support fetching multiple photos with a range of dates. Now suppose we were asked to build a backend API that can return a list of APOD with a given number of days, what should we do?

API map

The first thought I came up with is to generate an array that contains a range of dates. Then I can do a forEach method or a for loop to iterate through the array and make API calls one by one, get the data, push it into a result array, and finally return the result to the front-end. However, even this would work, it doesn't align with the goal, which requires to do the calls concurrently. Using forEach or a for loop still would do the job in order, not simultaneously. It's slow and not efficient.

After a little bit of research, I came across a library called async that perfectly fulfills the requirement of the task. The async library provides various types of functions for working with asynchronous JavaScript.

In this example, the method will be using is parallel, and it's mainly for flow control:

parallel(tasks, callback)
Enter fullscreen mode Exit fullscreen mode

It allows us to run a number of tasks in parallel, without waiting until the previous function has completed. The results are passed to the callback as an array.

Let's get started.

The solution

First, we need to make a helper function, it takes the number of days as a parameter, and returns an array of dates. NASA's API can only take the date format as YYYY-MM-DD, so for example, if today's date is 2020-12-23, and the number of days is equal to 6, the returned array will be:

[
  '2020-12-18',
  '2020-12-19',
  '2020-12-20',
  '2020-12-21',
  '2020-12-22',
  '2020-12-23'
]
Enter fullscreen mode Exit fullscreen mode

Here is what the function looks like:

function generatedates(numberOfDays) {
  const result = []
  const today = new Date()

  for (let i = 0; i < numberOfDays; i++) {
    let date = new Date(today)
    date.setDate(today.getDate() - i)
    let dd = date.getDate()
    let mm = date.getMonth() + 1
    let yyyy = date.getFullYear()

    if (dd < 10) {
      dd = "0" + dd
    }
    if (mm < 10) {
      mm = "0" + mm
    }
    date = yyyy + "-" + mm + "-" + dd
    result.unshift(date)
  }

  return result
}
Enter fullscreen mode Exit fullscreen mode

Then we need to add an endpoint to the node server.

/api/photos
Enter fullscreen mode Exit fullscreen mode

The parallel function takes an array of function as the first argument, so we could use the map method to iterate through the dates array and returns the function array. Each function in the array fires an Axios call to the NASA API and get the picture of that date.

The second argument of the parallel function is a callback function. In this case, since the API calls return promises, the callback function will return two items. The first one is the possible error, and the second one is the array of the result.

If we don't need to further process the data, we can simply pass them to the front-end. We can also use the forEach method to clean the data and only extract the information we need.

Here is the logic of the endpoint:

const URL = "https://api.nasa.gov/planetary/apod"

server.get("/api/photos", (req, res) => {
  const days = req.query.days
  const dates = generateDates(days)

  const functionArray = dates.map((date) => {
    return async function () {
      const data = await axios.get(`${URL}?api_key=${api_key}&date=${date}`)
      return data.data
    }
  })

  async.parallel(functionArray, (err, result) => {
    res.status(200).json({ items: result.length, photos: result })
  })
})
Enter fullscreen mode Exit fullscreen mode

Now the user can make an API request to fetch any number of photos, such as:

//fetch photos of the past week
api/photos?days=7

//fetch photos of the past month
api/photos?days=30
Enter fullscreen mode Exit fullscreen mode

And the result will be shown as:

{
    "items": 6,
    "photos": [...]
}
Enter fullscreen mode Exit fullscreen mode

Code

Check the GitHub repo for this example
Repo

Top comments (5)

Collapse
 
nahuelbm profile image
Nahuel Bustamante Murua

Hi!!
Maybe you can use Promise.all()
Happy end year!!

Collapse
 
ironydelerium profile image
ironydelerium

This, pretty much:

server.get("/api/photos", async (req, res) => {
  const days = req.query.days
  const dates = generateDates(days)

  const requests = dates.map((date) => axios.get(`${URL}?api_key=${api_key}&date=${date}`));

  try {
    const result = await Promise.all(requests);
    res.status(200).json({ items: requests.length, photos: result.map(item => item.data) });
  } catch (err) {
    res.status(500).json({ error: String(err) });
  }
})
Enter fullscreen mode Exit fullscreen mode

There's also Promise.allSettled if you want all of the results, error or not.

Collapse
 
gabrieleromanato profile image
Gabriele Romanato

This is a practical use case of parallelism in Node. However, we should always make sure that our routes are actually protected against any possible DOS attack by limiting the number of concurrent connections or simply by adding an authentication system on such routes with a middleware. Otherwise we'll sooner or later get into troubles.

Collapse
 
matiishyn profile image
Ivan

why?? why would you ever want to use it??

why don't you use just promise and promise.all

Collapse
 
yuesu profile image
Yue Su

well, it's just trying out some new tools. we could do fetch in JS, but it doesn't mean we can't use Axios.