Recursively fetch data from paginated API

Data from APIs are commonly paginated. We also have often use cases when we have to fetch all the records from an API. For example, when we want to make data visualizations. Therefore, we have to iteratively fetch all the pages which contain the data we need.

In this example, I show you a simple recursive way to achieve that goal from the World Banks API when we want to get indicator data by country.

If you're not familiar with recursive function, A recursive function is a function that calls itself.

First, we import Axios and define our baseUrl.

import axios from 'axios'

const baseUrl: string = 'https://api.worldbank.org/v2/country/'

Then we define our function. This function takes a country, indicator and page with default value 1 as it's parameters and it returns a Promise. Inside the function we define the query with our baseUrl, parameters and fetch our data and save it to a constant.

const getIndicatorByCountry = async (country: string, indicator: string, page:number=1): Promise<[]> => {  
  const query = `${baseUrl}/${country}/indicator/${indicator}?page=${page}&format=json`
  const response = await axios.get(query)  
  const data = response.data
}

Our data looks like this:

[
  {
    "page": 1,
    "pages": 2,
    "per_page": 50,
    "total": 60,
    "sourceid": "2",
    "lastupdated": "2020-07-01"
  },
  [
    {...},
    {...},
    /*50 Items in total*/
  ]
]

The response from API gives us information about which page we have fetched and the total number of pages. With this information, we can recursively fetch the rest of the data.

First, we check if the number of pages in our data is higher than our page parameter is. If there are more pages we can fetch, we call the function again with the same parameters except we add up page number by one. We do this until there are no more pages to be fetched and return the data.

if (data[0].pages > page) {
  return data.concat(await getIndicatorByCountry(country, indicator, page+1)) 
} else {
  return data
}

All in all our code looks like this. At the end the function return data from all pages.

import axios from 'axios'

const baseUrl: string = 'https://api.worldbank.org/v2/country/'

const getIndicatorByCountry = async (country: string, indicator: string, page:number=1): Promise<[]> => {  
  const query = `${baseUrl}/${country}/indicator/${indicator}?page=${page}&format=json`
  const response = await axios.get(query)  
  const data = response.data

  if (data[0].pages > page) {
    return data.concat(await getIndicatorByCountry(country, indicator, page+1)) 
  } else {
    return data
  }
}

This is a simple way to fetch data recursively from an API if the data is paginated. You could do this in just a normal for loop also, nothing wrong with that, but it is also nice to try something new every now and then.

You can learn more about the World Banks API from here.

Comments

[Jessica Iacovozzi]

Exactly what I was looking for thank you!

[Brian]

Hi. This is a good approach, nice to see only const being used. How would you handle errors here btw, if the 3rd call out of 5 failed for example?

[Anders D. Johnson]

This is cool! I had a similar need, so I wrote a library called "fetch-paginate" based on the native "fetch" API - see github.com/AndersDJohnson/fetch-pa....